Skip to content

About

Curriculum Contrained RAG

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

55 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

CurriculumRAG

A small "Compiler for Knowledge" that turns tagged curriculum Markdown into a graph-shaped, VectorDB-ready knowledge base.

CurriculumRAG reads a gold-standard Markdown file, recognizes curriculum atoms and validation gates, builds a directed NetworkX graph, and exports structured JSON chunks with lineage metadata for retrieval-augmented generation workflows.

At A Glance

Layer Main command Output
Author tagged Markdown edit original_tagged.md AKU and V_NODE curriculum source
Compile curriculum knowledge_compiler.py <source-or-slug> graph JSON, chunks JSON, graph PNG
Index chunks load_chunks_to_chroma.py <chunks-or-slug> persistent ChromaDB collection
Retrieve safely retrieval_orchestrator.py ... scoped context and teacher prompt
Evaluate evaluate_curriculum_rag.py --strict metrics, JSON report, plots
Explore interactively streamlit run dev_workbench.py local Dev Workbench UI

The core idea is simple: semantic search can find relevant text, but the curriculum graph decides whether the learner is allowed to see it yet.

Compilation includes a blocking graph audit. Cycles, malformed IDs, duplicate tags, missing requires: targets, unclosed tags, and nested tags stop the pipeline before chunks are loaded into ChromaDB.

Quick Start

For the built-in sample curriculum:

.\.venv\Scripts\python.exe knowledge_compiler.py gold-standard.md --chunks knowledge_chunks.json --graph knowledge_graph.json --visualize knowledge_graph.png
.\.venv\Scripts\python.exe load_chunks_to_chroma.py knowledge_chunks.json --persist-dir chroma_db --collection curriculum_rag --reset
.\.venv\Scripts\python.exe retrieval_orchestrator.py "What is an if statement?" --current-chapter C02 --completed-gate V_C01_COMPLETE

For a real textbook folder such as curriculum/g-epal-baseis/:

.\.venv\Scripts\python.exe knowledge_compiler.py g-epal-baseis --visualize
.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-large
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE --query-embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-large

For the Greek EPAL textbook, prefer sentence-transformers with intfloat/multilingual-e5-large. The first run may download the model from Hugging Face.

You can also test a Greek-specific AUEB Greek BERT preset:

.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model nlpaueb/bert-base-greek-uncased-v1
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE --query-embedding-mode sentence-transformers --embedding-model nlpaueb/bert-base-greek-uncased-v1

For a lighter E5 comparison, use:

.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-small
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE --query-embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-small

Use the same embedding model for loading and querying. For a fully offline smoke test, add hash embeddings to both load and query commands:

.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode hash
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE --query-embedding-mode hash

To use the interactive local workbench:

.\.venv\Scripts\streamlit.exe run dev_workbench.py

The workbench lets you compile, audit, load ChromaDB, compare Naive RAG vs DAG-only vs Full CurriculumRAG, inspect retrieved AKUs and lineage metadata, and rerun the evaluation plots from one browser panel.

README Map

Section Use it when you want to...
Visual Overview understand the system shape quickly
Implemented Architecture see the runtime contract for AI IDEs and agents
Curriculum Tagging Guide tag a parsed PDF/Markdown textbook
Curriculum Folder Workflow run commands by curriculum slug, such as g-epal-baseis
Local Setup And Development Guide clone, install, compile, load, and test locally
Evaluation Phase run leakage checks, plots, and RAGAS/LLM evaluation
Output Model inspect chunk JSON, graph JSON, lineage, and edges

Visual Overview

The project has two complementary views: a human-facing curriculum RAG timeline, and a node architecture view that shows how chapters, AKUs, validation gates, and future retrieval strategies fit together.

Timeline of Curriculum-Constrained RAG

Curriculum node architecture and future integrations

Implemented Architecture

flowchart TD
    subgraph Authoring["1. Curriculum Authoring"]
        Markdown["gold-standard.md<br/>Tagged Markdown with headings, AKUs, and V_NODE gates"]
    end

    subgraph Compile["2. Knowledge Compilation"]
        Compiler["knowledge_compiler.py"]
        Graph["NetworkX DiGraph<br/>hierarchy + dependency edges"]
        Chunks["knowledge_chunks.json<br/>AKU text + lineage metadata"]
        GraphJson["knowledge_graph.json<br/>nodes, edges, V_NODE gates"]
        GraphPng["knowledge_graph.png<br/>visual graph snapshot"]
    end

    subgraph Index["3. Chroma Indexing"]
        Loader["load_chunks_to_chroma.py"]
        Chroma["Persistent ChromaDB collection<br/>documents + embeddings + metadata"]
    end

    subgraph Runtime["4. Curriculum-Constrained Runtime"]
        Question["Student question"]
        State["Learner state<br/>current_chapter + completed_gates"]
        Orchestrator["retrieval_orchestrator.py<br/>metadata-first retrieval"]
        Gate["Gate check<br/>block future chapters until V_NODE completion"]
        Filter["Chroma where filter<br/>for example chapter_id = C02"]
        Inspector["Context inspector<br/>retrieved AKU ids + lineage"]
        Prompt["Pedagogical prompt<br/>system prompt + scoped context"]
        Teacher["LLM teacher response"]
    end

    Markdown --> Compiler
    Compiler --> Graph
    Compiler --> Chunks
    Compiler --> GraphJson
    Compiler --> GraphPng
    Chunks --> Loader
    Loader --> Chroma
    Question --> Orchestrator
    State --> Orchestrator
    GraphJson --> Orchestrator
    Orchestrator --> Gate
    Gate --> Filter
    Filter --> Chroma
    Chroma --> Orchestrator
    Orchestrator --> Inspector
    Orchestrator --> Prompt
    Prompt --> Teacher
Loading

For AI IDEs and future agents, the runtime contract is:

  1. Compile tagged curriculum into knowledge_chunks.json and knowledge_graph.json.
  2. Load chunks into ChromaDB with metadata fields such as chapter_id, lineage_json, lineage_ids, lineage_labels, and source_line_start.
  3. At inference time, call retrieval_orchestrator.py before constructing the LLM prompt.
  4. The orchestrator decides learner scope, applies a Chroma where filter, reconstructs lineage, and emits the teacher-facing prompt payload.
  5. If the requested chapter is outside the learner's unlocked scope, the orchestrator blocks retrieval instead of leaking future curriculum content.

Project Files

File Purpose
gold-standard.md Source curriculum written with AKU and V_NODE tags.
curriculum/<slug>/original_tagged.md Real textbook workspace source, addressed by slug commands such as g-epal-baseis.
curriculum_paths.py Shared helper that resolves curriculum slugs to source, graph, chunks, and collection names.
knowledge_compiler.py Parser/compiler that creates the graph and chunk exports.
load_chunks_to_chroma.py Loader that upserts AKU chunks into a persistent ChromaDB collection.
retrieval_orchestrator.py Metadata-first retrieval layer that enforces chapter and gate constraints.
evaluate_curriculum_rag.py Evaluation harness for deterministic constraint checks and optional RAGAS scoring.
build_evaluation_cases_from_csv.py Converts a teacher CSV dataset into benchmark-case JSON for evaluation.
dev_workbench.py Streamlit UI for local compile, audit, retrieval comparison, and evaluation runs.
evaluation/benchmark_cases.json Small gold benchmark set for retrieval, lineage, and gate behavior.
curriculum/<slug>/evaluation_scenarios.json Grouped teacher-question benchmark with beginner/intermediate/advanced persona permutations.
curriculum/<slug>/evaluation_cases.json Expanded evaluator-ready cases generated from the grouped persona scenarios.
evaluation/results/paper_evaluation_report.md Generated manuscript-ready evaluation summary with counts, confidence intervals, warnings, and tables.
evaluation/results/judge_dataset.jsonl Optional exported (query, chunk, learner_state) triples for LLM or human pedagogical judging.
knowledge_chunks.json VectorDB-ready AKU chunks with lineage metadata.
knowledge_graph.json Full graph export containing nodes and edges.
knowledge_graph_audit.json Teacher-facing graph audit with blocking errors and repair hints.
knowledge_graph.png Visual snapshot of the curriculum graph.
docs/assets/*.png README infographics for the curriculum RAG timeline and node architecture.
requirements.txt Python dependencies.
requirements-eval.txt Optional evaluation dependencies for RAGAS and pytest.

Curriculum Tagging Guide

Use this guide when you have a Markdown file parsed from a PDF textbook, syllabus, or official curriculum document and you want to prepare it for CurriculumRAG.

The goal is to turn ordinary Markdown into a structured curriculum source that can be compiled into:

  • graph nodes for chapters, sections, subsections, AKUs, and validation gates
  • hierarchy edges such as chapter -> section -> AKU
  • dependency edges such as prerequisite AKU -> validation gate
  • VectorDB-ready chunks with lineage metadata

The Mental Model

CurriculumRAG treats the curriculum as a hierarchy of small knowledge atoms.

Chapter
  Section
    Subsection
      AKU: Atomic Knowledge Unit
  V_NODE: Validation / completion gate

An AKU is the smallest useful learning unit that should become one retrieval chunk. A V_NODE is a checkpoint that says when a learner is allowed to move forward.

Tagging Cheat Sheet

Need Pattern
Chapter C01
Section C01.S02
AKU in a section C01.S02.A03
AKU in a subsection C01.S02.SS01.A03
Section completion gate V_C01_S02_COMPLETE
Chapter completion gate V_C01_COMPLETE
Immediate prerequisite requires: [C01.S02.A02]
Full section contract requires: [V_C01_S01_COMPLETE, C01.S02.A01, C01.S02.A02]
AKU range shorthand requires: [C01.S02.A01-07] expands to A01 through A07

Rule of thumb:

AKU requires   = immediate conceptual dependency
V_NODE requires = explicit completion contract

Dot-Notation IDs

Every curriculum unit uses a stable ID. The ID tells the compiler where the unit lives in the curriculum.

ID part Meaning Example
C01 Chapter 1 C01
S01 Section 1 inside the chapter C01.S01
SS01 Subsection 1 inside the section C01.S01.SS01
A01 Atomic Knowledge Unit 1 C01.S01.SS01.A01

Examples:

ID Meaning
C01 Chapter 1
C01.S02 Chapter 1, Section 2
C01.S02.SS03 Chapter 1, Section 2, Subsection 3
C01.S02.SS03.A04 Fourth AKU inside that subsection
C02.S01.A01 First AKU directly inside Chapter 2, Section 1

Use two digits for each number. Prefer C01.S01.A01 over C1.S1.A1. Stable IDs matter because benchmark cases, prerequisite gates, graph edges, and retrieval filters all refer to them.

Heading Rules

Headings create the parent hierarchy. The compiler recognizes chapter and numbered headings and turns them into graph nodes.

Use this style whenever possible:

# CHAPTER 1: Introduction to Computer Science

## 1.1 Basic Concepts

### 1.1.1 Data and Information

This produces:

Heading Node ID
# CHAPTER 1: ... C01
## 1.1 ... C01.S01
### 1.1.1 ... C01.S01.SS01

You may add a short chapter description after >:

# CHAPTER 1: Introduction to Computer Science > Level 1: Foundational concepts.

The text before > becomes the label. The text after > becomes the description.

AKU Blocks

Wrap each atomic learning unit in [AKU: ...] and [/AKU].

[AKU: C01.S01.SS01.A01]
Data is raw facts or symbols that can be processed by a computer.
[/AKU]

The compiler exports this as one chunk in knowledge_chunks.json.

Each AKU should be:

  • small enough to retrieve as one concept
  • self-contained enough to answer a learner question
  • tied to exactly one curriculum location
  • stable enough that its ID will not change casually

Good AKU:

[AKU: C01.S01.SS01.A01]
Data is raw facts or symbols. Information is data that has been processed so it
has meaning for a user.
[/AKU]

Too large:

[AKU: C01.S01.SS01.A01]
All of Chapter 1 pasted here...
[/AKU]

Too small:

[AKU: C01.S01.SS01.A01]
Data.
[/AKU]

AKU Metadata

Inside an AKU block, you may add metadata lines before the content.

[AKU: C01.S01.SS01.A01]
type: concept
Data is raw facts or symbols. Information is processed data with meaning.
[/AKU]

Supported metadata:

Metadata Purpose
type: concept Marks the pedagogical type of the AKU.
requires: [C01.S01.SS01.A01] Declares prerequisite AKUs.
access: preview Marks an overview AKU that may be visible before the chapter is fully unlocked.
page_start: 34 Optional source provenance from the textbook.
page_end: 36 Optional source provenance when an AKU spans pages.
source: EPAL official textbook Optional human-readable source label.

type: and requires: may be written as plain metadata lines or hidden HTML comments. Extra provenance metadata should be written as hidden comments so the compiler never confuses ordinary textbook text with metadata:

<!-- page_start: 34 -->
<!-- page_end: 36 -->
<!-- source: EPAL official textbook -->

Example with prerequisites:

[AKU: C01.S01.SS02.A01]
type: procedure
requires: [C01.S01.SS01.A01]
To classify information, first identify the raw data and then explain what
meaning was added through processing.
[/AKU]

The compiler creates dependency edges from every requires item to the current node.

Recommended AKU Types

Use type: to make the pedagogical role of an AKU explicit. The value can be a single label or a short comma-separated list when the unit has more than one role.

<!-- type: concept, abstraction -->

Recommended labels:

Type Use when the AKU... Example
objective states expected learning outcomes "The student should be able to..."
concept explains an idea or principle data independence, DBMS purpose
definition defines a term precisely file, record, field, entity
example illustrates a previous concept customer-card file example
exercise asks the learner to answer, practice, or investigate questions / activities
procedure gives steps the learner should execute normalize a relation step by step
process describes a sequence, lifecycle, or flow database design phases
model introduces a representation or framework entity-relationship model
abstraction explains levels, views, hiding details, or independence logical/physical data independence
comparison contrasts two or more concepts conceptual vs physical design
warning highlights a common mistake or constraint duplicate keys are not allowed
summary recaps prior material end-of-section synthesis

Common combinations:

Combination Use for
concept, abstraction ideas about levels, views, or independence, such as data independence
concept, model introduction of a conceptual model such as ER modeling
concept, comparison explaining differences between related ideas
exercise, reflection open-ended questions or classroom activities
process, model lifecycle phases tied to a modeling framework

Prefer abstraction over abstract. The word abstract can sound like a paper abstract, while abstraction clearly means hiding lower-level details or working through levels of representation.

Validation Gates With V_NODE

Use V_NODE blocks to represent curriculum milestones, chapter completion checks, or prerequisite gates.

### CHAPTER 1 COMPLETION GATE

[V_NODE: V_C01_COMPLETE]
requires: [C01.S01.SS01.A01, C01.S01.SS02.A01]
[/V_NODE]

This means the learner should complete the required AKUs before Chapter 1 is considered complete. The retrieval orchestrator can then use the gate state to prevent future-chapter leakage.

Recommended gate names:

Gate ID Meaning
V_C01_COMPLETE Chapter 1 completion gate
V_C02_COMPLETE Chapter 2 completion gate
V_C01_S01_COMPLETE Chapter 1, Section 1 completion gate
V_C01_S02_COMPLETE Chapter 1, Section 2 completion gate
V_UNIT01_COMPLETE Unit/module completion gate

For chapter gates, prefer V_C##_COMPLETE because it matches the section-gate style, such as V_C01_S02_COMPLETE. The tools still accept older V_CH##_COMPLETE inputs and normalize them to the same chapter gate.

For section gates, prefer V_C##_S##_COMPLETE. This allows the orchestrator to control movement inside a chapter. For example, a learner at C01.S01 should not receive C01.S02 chunks until V_C01_S01_COMPLETE is marked complete.

Section gate example:

## 1.1 Data and Information

[AKU: C01.S01.A01]
type: concept
Data is raw facts or symbols that can be processed by a computer.
[/AKU]

### SECTION 1.1 COMPLETION GATE

[V_NODE: V_C01_S01_COMPLETE]
requires: [C01.S01.A01]
[/V_NODE]

## 1.2 Information Processing

[AKU: C01.S02.A01]
type: process
requires: [C01.S01.A01]
Processing transforms data into useful information.
[/AKU]

Runtime example:

.\.venv\Scripts\python.exe retrieval_orchestrator.py "Explain section 1.2" --current-section C01.S01

This blocks because V_C01_S01_COMPLETE is missing. Once that gate is complete:

.\.venv\Scripts\python.exe retrieval_orchestrator.py "Explain section 1.2" --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE

Dependency Style For AKUs And V_NODEs

Use different levels of redundancy for AKUs and validation gates.

For AKUs, keep requires local and conceptual. Do not repeat the full ancestry unless the AKU directly depends on every listed concept.

Good:

<!-- [AKU: C01.S04.A03] -->
<!-- type: exercise -->
<!-- requires: [C01.S04.A02] -->
### Ε�ωτήσεις / Δ�αστη�ι�τητες
- Αναζήτηση πλη�οφο�ι�ν για τον εκτιμ�μενο �γκο δεδομένων.
<!-- [/AKU] -->

Usually avoid this for an AKU:

<!-- [AKU: C01.S04.A03] -->
<!-- requires: [C01.S01.A01, C01.S02.A04, C01.S03.A02, C01.S04.A02] -->
...
<!-- [/AKU] -->

The graph can recover the longer dependency path by traversing earlier requires edges. Writing the full path on every AKU makes the graph noisy and harder for a teacher to maintain.

For V_NODEs, redundancy is intentional because the gate is a completion contract. A section gate should include the previous section gate plus the AKUs that must be completed in the current section.

<!-- [V_NODE: V_C01_S04_COMPLETE] -->
<!-- requires: [V_C01_S03_COMPLETE, C01.S04.A01, C01.S04.A02, C01.S04.A03] -->
<!-- [/V_NODE] -->

Do not duplicate the whole ancestry behind V_C01_S03_COMPLETE. The previous gate already carries that path.

For chapter completion, list the required section gates:

<!-- [V_NODE: V_C01_COMPLETE] -->
<!-- requires: [V_C01_S01_COMPLETE, V_C01_S02_COMPLETE, V_C01_S03_COMPLETE, V_C01_S04_COMPLETE] -->
<!-- [/V_NODE] -->

Recommended convention:

AKU requires      = immediate conceptual prerequisites
Section V_NODE    = previous section gate + this section's required AKUs
Chapter V_NODE    = required section gates

This keeps AKU dependencies clean while giving the teacher an explicit mastery contract at every section and chapter boundary.

Section-Level Vs Subsection-Level AKUs

Use the smallest structural level that actually exists in the textbook.

If a section has no subsections, section-level AKUs are fine:

## 3.1 Database Design

<!-- [AKU: C03.S01.A01] -->
<!-- type: concept -->
Database design starts from user and system requirements.
<!-- [/AKU] -->

<!-- [AKU: C03.S01.A02] -->
<!-- type: concept -->
<!-- requires: [C03.S01.A01] -->
The design process turns requirements into a database structure.
<!-- [/AKU] -->

If the section is divided into subsections, prefer putting AKUs inside the subsection IDs and avoid mixing in sibling section-level AKUs unless the content really belongs to the section as a whole.

Preferred:

## 3.1 Database Design

### 3.1.2 Physical Design

<!-- [AKU: C03.S01.SS02.A04] -->
<!-- type: concept -->
<!-- requires: [C03.S01.SS02.A03] -->
Physical design describes how data is stored and accessed on storage media.
<!-- [/AKU] -->

<!-- [AKU: C03.S01.SS02.A05] -->
<!-- type: exercise -->
<!-- requires: [C03.S01.SS02.A04] -->
Questions and activities about conceptual, logical, and physical design.
<!-- [/AKU] -->

Avoid mixing this into the same subsection sequence:

<!-- [AKU: C03.S01.A03] -->
...
<!-- [/AKU] -->

<!-- [AKU: C03.S01.SS02.A04] -->
...
<!-- [/AKU] -->

That pattern is legal, but it is harder for teachers and evaluators to read. It can become unclear whether C03.S01.A03 is a true section-level atom or an atom that should have been placed under a subsection.

For subsection gates, use the previous subsection gate plus the AKUs inside the current subsection:

<!-- [V_NODE: V_C03_S01_SS02_COMPLETE] -->
<!-- requires: [V_C03_S01_SS01_COMPLETE, C03.S01.SS02.A01, C03.S01.SS02.A02, C03.S01.SS02.A03, C03.S01.SS02.A04, C03.S01.SS02.A05] -->
<!-- [/V_NODE] -->

Then roll subsection gates up into the section gate:

<!-- [V_NODE: V_C03_S01_COMPLETE] -->
<!-- requires: [V_C03_S01_SS01_COMPLETE, V_C03_S01_SS02_COMPLETE] -->
<!-- [/V_NODE] -->

Recommended structural rule:

Section with no subsections    -> use C03.S01.A01, C03.S01.A02
Section with subsections       -> use C03.S01.SS01.A01, C03.S01.SS02.A01
Subsection V_NODE              -> previous subsection gate + current subsection AKUs
Section V_NODE with subsections -> required subsection gates

Rules Of Thumb For AKUs And V_NODEs

These conventions are not just formatting preferences. They make the Markdown easier for teachers to maintain, easier for AI IDEs to inspect, and safer for the compiler to convert into a clean prerequisite DAG.

1. AKUs should be the smallest meaningful learning atoms

An AKU should normally contain one teachable idea, example, rule, diagram, activity, or explanation. Avoid wrapping a whole subsection as one AKU if it contains several concepts that could be taught or retrieved separately.

Good:

<!-- [AKU: C03.S03.SS01.A04] -->
<!-- type: concept -->
<!-- requires: [C03.S03.SS01.A03] -->
Each entity is described by attributes.
<!-- [/AKU] -->

<!-- [AKU: C03.S03.SS01.A05] -->
<!-- type: example -->
<!-- requires: [C03.S03.SS01.A04] -->
For example, a Student entity may have name, address, and registration number.
<!-- [/AKU] -->

Avoid:

<!-- [AKU: C03.S03.SS01.A01] -->
All definitions, examples, diagrams, and activities for the whole subsection.
<!-- [/AKU] -->

2. Keep AKU prerequisites local, not fully expanded

For AKUs, requires should list the immediate conceptual prerequisites. The compiler can recover the full ancestry by traversing the graph.

Good:

<!-- [AKU: C03.S05.SS02.A04] -->
<!-- type: definition, rules -->
<!-- requires: [C03.S05.SS02.A03] -->
Rules for transforming an ER diagram into the relational model.
<!-- [/AKU] -->

Usually avoid:

<!-- requires: [C03.S05.A01, C03.S05.SS01.A01, C03.S05.SS01.A02, C03.S05.SS02.A03] -->

That long version is harder to maintain and creates noisy edges without adding new information.

3. Use explicit dual prerequisites when the pedagogy really needs them

Sometimes a chunk continues an earlier example but also depends on a rule that was introduced between the two example parts. In that case it is fine to list both dependencies, even if one dependency already points to the other.

Good:

<!-- [AKU: C03.S05.SS02.A05] -->
<!-- type: example, continuation -->
<!-- requires: [C03.S05.SS02.A03, C03.S05.SS02.A04] -->
The same ER example is now transformed using the rules above.
<!-- [/AKU] -->

Here A03 preserves the example lineage, while A04 captures the rule needed to understand the continuation.

4. V_NODEs are mastery contracts, so some redundancy is useful

A V_NODE should say what must be completed before the learner is considered to have mastered that section, subsection, or chapter. Unlike AKUs, V_NODEs can be more explicit because they are teacher-facing checkpoints.

Good section gate:

<!-- [V_NODE: V_C02_S03_COMPLETE] -->
<!-- requires: [V_C02_S02_COMPLETE, C02.S03.A01, C02.S03.A02, C02.S03.A03, C02.S03.A04, C02.S03.A05, C02.S03.A06, C02.S03.A07] -->
<!-- [/V_NODE] -->

If you want extra readability, you may include earlier section gates too:

<!-- requires: [V_C02_S01_COMPLETE, V_C02_S02_COMPLETE, C02.S03.A01, C02.S03.A02] -->

The shorter immediate-gate version is usually cleaner. The longer version is acceptable when a teacher wants the path to be visible in the Markdown.

5. Use AKU range shorthand for long V_NODE contracts

When a V_NODE requires a consecutive run of AKUs, you may use a compact range instead of writing every ID by hand.

Good:

<!-- [V_NODE: V_C04_S01_SS01_COMPLETE] -->
<!-- requires: [V_C03_COMPLETE, C04.S01.SS01.A01-12] -->
<!-- [/V_NODE] -->

The compiler expands this internally into:

C04.S01.SS01.A01
C04.S01.SS01.A02
...
C04.S01.SS01.A12

Use this shorthand mainly for V_NODE completion contracts. For AKU prerequisites, explicit immediate dependencies are usually clearer because an AKU should rarely depend on a whole consecutive range.

Range rules:

  • the range must stay inside one AKU prefix, such as C04.S01.SS01.A
  • write the start and end atom numbers only: A01-12
  • do not use ranges for non-consecutive concepts
  • if one AKU in the range is optional or enrichment, write the list manually so the V_NODE does not accidentally require it

6. Name each V_NODE after what it completes

The V_NODE ID should match the curriculum scope of the AKUs inside it.

Good:

<!-- [V_NODE: V_C03_S05_SS01_COMPLETE] -->
<!-- requires: [C03.S05.SS01.A01, C03.S05.SS01.A02, C03.S05.SS01.A03] -->
<!-- [/V_NODE] -->

Avoid:

<!-- [V_NODE: V_C03_S05_SS02_COMPLETE] -->
<!-- requires: [C03.S05.SS01.A01, C03.S05.SS01.A02, C03.S05.SS01.A03] -->
<!-- [/V_NODE] -->

That second example says "SS02 complete" but contains only SS01 AKUs, which will confuse graph inspection and evaluation.

7. Put the previous gate on the gate when it represents flow

If a new section starts after a previous section is completed, the cleanest place to encode that flow is often the V_NODE, not every AKU.

Good:

<!-- [V_NODE: V_C03_S05_COMPLETE] -->
<!-- requires: [V_C03_S04_COMPLETE, C03.S05.A02, V_C03_S05_SS01_COMPLETE, V_C03_S05_SS02_COMPLETE] -->
<!-- [/V_NODE] -->

This keeps the progression from section 3.4 to section 3.5 visible without forcing every section 3.5 AKU to repeat V_C03_S04_COMPLETE.

8. First AKUs may require a previous gate when they unlock a new chapter

The first teachable AKU of a new chapter can point to the previous chapter gate when the chapter truly depends on prior completion.

<!-- [AKU: C03.S01.A01] -->
<!-- type: introduction -->
<!-- requires: [V_C02_COMPLETE] -->
Chapter 3 introduces database design.
<!-- [/AKU] -->

Then the first section or subsection V_NODE can include that AKU as part of the completion contract.

9. Chapter overview AKUs can be marked as preview

Some chapter introductions are safe and useful before the learner formally unlocks the chapter. Objectives, high-level summaries, and "what is this chapter about?" atoms can be tagged as preview content. This lets the system distinguish between orientation material and gated instructional content.

Good:

# κεφάλαιο *4*
**Η Γλ�σσα Βάσεων Δεδομένων SQL**

## Εισαγωγή
<!-- [AKU: C04.A01] -->
<!-- type: objective, overview -->
<!-- access: preview -->
> Στ�χοι / Π�οσδοκ�μενα Αποτελέσματα
<!-- [/AKU] -->

<!-- [AKU: C04.A02] -->
<!-- type: concept, overview -->
<!-- access: preview -->
Η SQL είναι γλ�σσα διαχεί�ισης Σχεσιακ�ν Βάσεων Δεδομένων.
<!-- [/AKU] -->

<!-- [AKU: C04.S01.A01] -->
<!-- type: concept -->
<!-- requires: [V_C03_COMPLETE, C04.A02] -->
The first teachable SQL topic that should be gated behind Chapter 3 completion.
<!-- [/AKU] -->

Rules for preview AKUs:

  • use unique IDs; do not reuse C04.A01 for both objectives and concept text
  • use access: preview only for orientation content, not core future-chapter explanations or assessment answers
  • keep real instructional AKUs gated with requires: [V_C03_COMPLETE, ...]
  • leave preview AKUs out of the chapter completion V_NODE unless they are truly required for mastery

The compiler and Chroma loader preserve access as metadata. Retrieval policy can then allow access: preview atoms as a controlled exception while keeping the rest of the chapter locked.

10. Chapter gates should include all required section gates

If a chapter has six required sections, the chapter gate should include all six section completion gates. Do not accidentally omit the final section.

<!-- [V_NODE: V_C02_COMPLETE] -->
<!-- requires: [V_C01_COMPLETE, V_C02_S01_COMPLETE, V_C02_S02_COMPLETE, V_C02_S03_COMPLETE, V_C02_S04_COMPLETE, V_C02_S05_COMPLETE, V_C02_S06_COMPLETE] -->
<!-- [/V_NODE] -->

Include the previous chapter gate if the chapter itself is sequentially locked behind earlier curriculum progress.

11. Optional or enrichment AKUs do not have to be in the V_NODE

An AKU is not automatically an orphan just because it is not required by a V_NODE. If it has a valid requires edge, it can still be retrieved as optional, enrichment, additional material, or a deeper example.

<!-- [AKU: C03.S03.SS03.A12] -->
<!-- type: example, diagram, enrichment -->
<!-- requires: [C03.S03.SS03.A11] -->
An extra complete ER diagram example.
<!-- [/AKU] -->

If the learner does not need this AKU for mastery, leave it out of the V_NODE. If it is necessary for assessment or progression, include it.

12. Exercises can depend on the concept they actually practice

An exercise does not need to depend on every example that appears before it. It should require the concepts needed to attempt the activity.

Good:

<!-- [AKU: C03.S03.SS03.A13] -->
<!-- type: exercise -->
<!-- requires: [C03.S03.SS03.A10] -->
Model this relationship scenario as an ER diagram.
<!-- [/AKU] -->

This is better than forcing the exercise to depend on an optional extra example if the exercise only needs the diagram notation rule introduced in A10.

13. Put human headings outside the AKU unless the heading is part of the atom

For activity sections, a visible Markdown heading is often just a human reading marker. Put it before the AKU block.

Preferred:

#### Questions / Activities

<!-- [AKU: C03.S03.SS03.A13] -->
<!-- type: exercise -->
<!-- requires: [C03.S03.SS03.A10] -->
Model the following relationship scenario.
<!-- [/AKU] -->

Avoid putting a heading on the same line as metadata:

<!-- requires: [C03.S03.SS03.A10] -->#### Questions / Activities

Metadata comments should always be on their own line.

14. Do not create nested AKUs for subheadings

Avoid inventing extra levels like SSS01 just to wrap an exercise title. The retrieval model expects AKUs to be the leaf learning atoms.

Avoid:

<!-- [AKU: C03.S03.SS03.A13] -->
...
<!-- [AKU: C03.S03.SS03.SSS01.A01] -->
...
<!-- [/AKU] -->

Use one AKU instead:

<!-- [AKU: C03.S03.SS03.A13] -->
<!-- type: exercise -->
<!-- requires: [C03.S03.SS03.A10] -->
##### Activity: Model a scenario
...
<!-- [/AKU] -->

15. Use comma-separated labels for multiple AKU types

Use this:

<!-- type: example, explanation -->

Not this:

<!-- type: [example, explanation] -->

requires is a machine-readable list. type is a simple comma-separated label field used for filtering, reporting, and human inspection.

16. Use stable dot notation and avoid placeholders

IDs should be consistent, zero-padded, and complete.

Good:

C03.S02.A07
C03.S02.SS01.A06
V_C03_S02_SS01_COMPLETE

Avoid:

03.S02.A07
C03.S02.A7
C0.S0.A0
V_C0_S01_SS01_COMPLETE

Also avoid writing AKU: inside a requires list:

<!-- requires: [C03.S03.SS03.A10] -->

not:

<!-- requires: [AKU: C03.S03.SS03.A10] -->

17. Add page metadata to AKUs when source traceability matters

For textbook-derived curricula, page metadata is useful on AKUs because the retrieved chunk can cite its source page.

<!-- [AKU: C03.S03.SS03.A10] -->
<!-- type: concept, diagram -->
<!-- page_start: 47 -->
<!-- page_end: 47 -->
<!-- requires: [C03.S03.SS03.A09] -->
Optional and mandatory relationships in ER diagrams.
<!-- [/AKU] -->

V_NODEs usually do not need page metadata because they are synthetic mastery gates rather than textbook content. Add it only if you want to trace where a teacher defined the gate.

18. Let the audit stop you before ChromaDB ingestion

Run the compiler audit whenever you edit original_tagged.md. Fix errors before loading ChromaDB.

.\.venv\Scripts\python.exe knowledge_compiler.py g-epal-baseis --audit-only

Treat cycles, missing references, malformed IDs, unclosed AKUs, nested AKUs, and placeholder IDs as blockers. Duplicate headings caused by PDF artifacts can often remain warnings if the AKU and V_NODE IDs are still unique.

HTML Comment Tags

If you want the Markdown to remain cleaner for human readers, the compiler also accepts tags inside HTML comments.

<!-- [AKU: C01.S01.SS01.A01] -->
Data is raw facts or symbols. Information is processed data with meaning.
<!-- [/AKU] -->

And for validation gates:

<!-- [V_NODE: V_C01_COMPLETE] -->
requires: [C01.S01.SS01.A01, C01.S01.SS02.A01]
<!-- [/V_NODE] -->

From Parsed PDF Markdown To Tagged Curriculum

When starting from a PDF-derived .md file, use this workflow:

  1. Preserve the official chapter and section order.
  2. Convert major titles into # CHAPTER n: ... headings.
  3. Convert section titles into ## n.m ... headings.
  4. Convert subsection titles into ### n.m.k ... headings when available.
  5. Split long textbook paragraphs into small AKU blocks.
  6. Assign each AKU a stable dot-notation ID.
  7. Add type: metadata when it helps evaluation or teaching.
  8. Add requires: metadata for prerequisite concepts.
  9. Add a V_NODE gate at the end of each chapter or unit.
  10. Add section gates when one section is prerequisite to the next section.
  11. Compile and inspect knowledge_chunks.json before loading ChromaDB.

Full Example

# CHAPTER 1: Basic Concepts of Informatics > EPAL Computer Science foundation.

## 1.1 Data and Information

### 1.1.1 Data

[AKU: C01.S01.SS01.A01]
type: concept
Data is raw facts, symbols, numbers, words, or observations that can be stored
and processed by a computer.
[/AKU]

[AKU: C01.S01.SS01.A02]
type: concept
requires: [C01.S01.SS01.A01]
Information is data that has been processed, organized, or interpreted so that
it has meaning for a user.
[/AKU]

### 1.1.2 Processing

[AKU: C01.S01.SS02.A01]
type: process
requires: [C01.S01.SS01.A01, C01.S01.SS01.A02]
Processing is the transformation of data into useful information through
operations such as sorting, calculating, classifying, or comparing.
[/AKU]

### CHAPTER 1 COMPLETION GATE

[V_NODE: V_C01_COMPLETE]
requires: [C01.S01.SS01.A01, C01.S01.SS01.A02, C01.S01.SS02.A01]
[/V_NODE]

Common Mistakes To Avoid

Mistake Why it hurts
Reusing the same AKU ID twice Later chunks overwrite or collide conceptually.
Making one AKU cover many pages Retrieval becomes vague and less teachable.
Skipping headings Chunks lose chapter/section lineage.
Forgetting requires on gates The graph cannot represent mastery prerequisites.
Renumbering IDs after benchmark creation Evaluation cases and gates stop matching.
Mixing future-chapter content into earlier AKUs The orchestrator cannot prevent leakage inside a chunk.

Tagging Quality Checklist

Before compiling a new curriculum file, check:

  • Every chapter has a # CHAPTER n: heading, a Greek # Ξ�εφάλαιο n: heading, or a plain numbered heading such as # 1. ....
  • Every section uses a numbered heading such as ## 1.2.
  • Every AKU has one unique dot-notation ID.
  • Every AKU contains one focused learning idea.
  • Every requires: reference points to an existing AKU or gate.
  • Every chapter that should control progression has a V_C##_COMPLETE gate.
  • Every section that should control progression has a V_C##_S##_COMPLETE gate.
  • The Markdown preserves the official curriculum sequence.
  • No AKU contains large unrelated blocks copied directly from the PDF.

Then compile:

.\.venv\Scripts\python.exe knowledge_compiler.py your-curriculum.md --chunks your_chunks.json --graph your_graph.json --visualize your_graph.png

Inspect the output:

Get-Content your_chunks.json
Get-Content your_graph.json

If the lineage looks correct, load it into ChromaDB:

.\.venv\Scripts\python.exe load_chunks_to_chroma.py your_chunks.json --persist-dir chroma_db --collection your_curriculum --reset

Curriculum Folder Workflow

For real textbook work, place each parsed curriculum under curriculum/<slug>/ and keep the tagged source named original_tagged.md.

Expected layout:

curriculum/
  g-epal-baseis/
    original.md
    original_meta.json
    original_tagged.md

Then compile by folder slug:

.\.venv\Scripts\python.exe knowledge_compiler.py g-epal-baseis --visualize

This reads:

curriculum/g-epal-baseis/original_tagged.md

and writes:

curriculum/g-epal-baseis/knowledge_chunks.json
curriculum/g-epal-baseis/knowledge_graph.json
curriculum/g-epal-baseis/knowledge_graph_audit.json
curriculum/g-epal-baseis/knowledge_graph.png

Load the same curriculum into ChromaDB:

.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset

For Greek textbook retrieval, use the default multilingual E5 large embedding model:

.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-large

Alternative Greek-specific experimental preset:

.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model nlpaueb/bert-base-greek-uncased-v1

Lighter E5 comparison preset:

.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-small

For offline smoke tests, use local hash embeddings:

.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode hash

The default Chroma collection name becomes the slug:

g-epal-baseis

Query it through the orchestrator:

.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE

If you loaded with --embedding-mode hash, query with hash embeddings too:

.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE --query-embedding-mode hash

Generated files stay next to the source curriculum:

File Meaning
original_tagged.md Human-edited tagged textbook source.
knowledge_chunks.json AKUs ready for ChromaDB ingestion.
knowledge_graph.json Full hierarchy and prerequisite graph.
knowledge_graph_audit.json Blocking errors, warnings, and repair hints for the teacher.
knowledge_graph.png Visual graph snapshot for quick inspection.

The slug also becomes the default Chroma collection name, so g-epal-baseis loads into collection g-epal-baseis.

Local Setup And Development Guide

Follow this path after cloning the repository onto a new machine. The commands below use PowerShell; on macOS/Linux, replace .\.venv\Scripts\python.exe with .venv/bin/python.

1. Clone The Project

git clone https://github.com/programmatismos/CurriculumRAG.git
cd CurriculumRAG

2. Create A Local Python Environment

This project works well with uv because it creates a fast, isolated local environment in .venv/.

python -m pip install uv
uv venv .venv
uv pip install -r requirements.txt

Quick check:

.\.venv\Scripts\python.exe -c "import networkx, chromadb; print('Environment ready')"

3. Compile The Gold Standard Curriculum

This turns gold-standard.md into graph and retrieval artifacts.

.\.venv\Scripts\python.exe knowledge_compiler.py gold-standard.md --chunks knowledge_chunks.json --graph knowledge_graph.json --visualize knowledge_graph.png

Expected output:

Compiled <nodes> nodes, <edges> edges, and <chunks> AKU chunks.

Generated artifacts:

Artifact What it is used for
knowledge_chunks.json VectorDB-ready AKU chunks with lineage metadata.
knowledge_graph.json Curriculum graph with hierarchy and dependency edges.
knowledge_graph.png Visual snapshot of the graph.

4. Load Chunks Into ChromaDB

This creates a local persistent vector store in chroma_db/.

.\.venv\Scripts\python.exe load_chunks_to_chroma.py knowledge_chunks.json --persist-dir chroma_db --collection curriculum_rag --reset

Expected output:

Loaded <n> chunks into collection 'curriculum_rag' at chroma_db.

If you want a fully offline smoke test that does not use Chroma's default embedding model, use deterministic hash embeddings:

.\.venv\Scripts\python.exe load_chunks_to_chroma.py knowledge_chunks.json --persist-dir chroma_db --collection curriculum_rag --reset --embedding-mode hash

5. Test Curriculum-Constrained Retrieval

Allowed query: the learner is in Chapter 2 and has completed the Chapter 1 gate.

.\.venv\Scripts\python.exe retrieval_orchestrator.py "What is an if statement?" --current-chapter C02 --completed-gate V_C01_COMPLETE --n-results 1

The orchestrator should retrieve C02.S01.A01 using a Chroma metadata filter like:

{"chapter_id": "C02"}

Blocked query: the learner is still in Chapter 1, so Chapter 2 should not leak.

.\.venv\Scripts\python.exe retrieval_orchestrator.py "Explain Chapter 2 if statements" --current-chapter C01

Expected behavior:

BLOCKED BY CURRICULUM CONSTRAINT

6. Use JSON Mode For Experiments

Use --json when connecting the orchestrator to evaluation scripts, notebooks, or a future Streamlit/Gradio interface.

.\.venv\Scripts\python.exe retrieval_orchestrator.py "What is an if statement?" --current-chapter C02 --completed-gate V_C01_COMPLETE --json

The JSON payload includes learner state, retrieval scope, retrieved AKU IDs, lineage metadata, and the prompt payload that can be sent to an LLM.

7. Use The Dev Workbench

For interactive testing, launch the local Streamlit workbench:

.\.venv\Scripts\streamlit.exe run dev_workbench.py

The workbench provides four panels:

Panel What it helps you do
1. Pipeline audit tagged Markdown, inspect DAG errors/warnings with source-line fix snippets, compile graph/chunks, load ChromaDB, and view knowledge_graph.png
2. Query Lab compare Naive RAG, prompt-guardrail context, DAG-only, and Full CurriculumRAG for the same question
3. Evaluation rerun benchmark cases, update plots, export judge triples, and view the paper report
4. Artifacts inspect generated files, counts, collection names, and equivalent CLI commands

The top of the UI includes a first-time orientation, a glossary for AKU/V_NODE terms, and a runtime summary so colleagues can understand what is selected before they run anything.

Use the same embedding mode in the workbench that you used when loading ChromaDB. For Greek textbook evaluation, use sentence-transformers with intfloat/multilingual-e5-large. The workbench also includes intfloat/multilingual-e5-small for a lighter E5 comparison and nlpaueb/bert-base-greek-uncased-v1 for a Greek-specific comparison run. For offline smoke tests, load with --embedding-mode hash and set the workbench embedding mode to hash.

8. Run The Whole Local Pipeline

For the normal textbook workflow, use the bundled PowerShell runner instead of typing each command by hand:

.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseis

This runs:

  1. knowledge_compiler.py <curriculum> --visualize
  2. load_chunks_to_chroma.py <curriculum> --reset --embedding-mode sentence-transformers
  3. build_evaluation_cases_from_csv.py <curriculum>
  4. evaluate_curriculum_rag.py ... --strict
  5. streamlit run dev_workbench.py

If a step fails, the script stops immediately and does not run later steps. For example, if the graph audit has blocking errors, it will not load stale chunks into ChromaDB or run misleading evaluation numbers.

Useful variants:

.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseis -SkipEvaluation
.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseis -NoStreamlit
.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseis -AllowEvaluationFailures
.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseis -EmbeddingModel intfloat/multilingual-e5-large
.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseis -EmbeddingModel intfloat/multilingual-e5-small
.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseis -EmbeddingModel nlpaueb/bert-base-greek-uncased-v1
.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseis -EmbeddingMode hash

Use -AllowEvaluationFailures when you want Streamlit to open even if --strict evaluation finds failing benchmark cases. The report and plots are still written, and the workbench can then be used to inspect what failed.

9. Development Loop

When editing the curriculum or compiler, use this loop:

.\.venv\Scripts\python.exe knowledge_compiler.py gold-standard.md --chunks knowledge_chunks.json --graph knowledge_graph.json --visualize knowledge_graph.png
.\.venv\Scripts\python.exe load_chunks_to_chroma.py knowledge_chunks.json --persist-dir chroma_db --collection curriculum_rag --reset
.\.venv\Scripts\python.exe retrieval_orchestrator.py "your test question" --current-chapter C01

Local runtime folders such as .venv/ and chroma_db/ are ignored by git, so each developer can rebuild them safely on their own machine.

Command Recipes

Compile the sample curriculum:

.\.venv\Scripts\python.exe knowledge_compiler.py gold-standard.md --chunks knowledge_chunks.json --graph knowledge_graph.json --visualize knowledge_graph.png

Compile a textbook folder:

.\.venv\Scripts\python.exe knowledge_compiler.py g-epal-baseis --visualize

Load the sample into ChromaDB:

.\.venv\Scripts\python.exe load_chunks_to_chroma.py knowledge_chunks.json --persist-dir chroma_db --collection curriculum_rag --reset

Load a textbook folder into ChromaDB:

.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-large

Load a fully offline hash-embedding collection:

.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode hash

Ask with chapter scope:

.\.venv\Scripts\python.exe retrieval_orchestrator.py "What is an if statement?" --current-chapter C02 --completed-gate V_C01_COMPLETE

Ask with section scope:

.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE

Ask against a hash-embedding collection:

.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE --query-embedding-mode hash

Ask against a multilingual sentence-transformers collection:

.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE --query-embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-large

Run deterministic evaluation:

.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict

Export paper tables and judge triples:

.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict --export-judge-dataset

Validate that the paper is still bound to the latest generated evaluation macros/tables, even on machines without a LaTeX engine:

.\.venv\Scripts\python.exe scripts\validate_paper_bindings.py

Run evaluation without plots:

.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict --no-plots

What To Check After Tagging

After tagging a real textbook, run the compiler audit before loading ChromaDB:

.\.venv\Scripts\python.exe knowledge_compiler.py g-epal-baseis --audit-only

If the audit is blocked, the compiler prints teacher-friendly repair hints and writes:

curriculum/g-epal-baseis/knowledge_graph_audit.json

Blocking errors must be fixed in original_tagged.md before proceeding.

The audit catches:

Signal Why it matters
Cycle detected A learner would need a node that eventually requires itself.
Missing requires: target Usually a typo or a prerequisite AKU/V_NODE that was never tagged.
Malformed IDs such as C01.S01.A7 Breaks stable dot-notation and makes lineage unreliable.
Duplicate AKU/V_NODE IDs Two different blocks would collapse into the same graph node.
Nested AKU/V_NODE tag Usually a missing closing tag before the next block starts.
Unclosed AKU/V_NODE block Usually causes one huge chunk that swallows later content.
Unexpected IDs such as C0.S0.A0 Usually placeholder AKUs that should be renamed.
IDs like C01.S0.A03 Usually section-number typos.
IDs missing the C prefix, such as 01.S03.A08 Usually malformed prerequisites.
Huge AKU chunks Usually a missing <!-- [/AKU] --> closing tag.
Virtual nodes in the graph audit Usually references to missing or mistyped IDs.

When the audit is clean, compile, visualize, and load:

.\.venv\Scripts\python.exe knowledge_compiler.py g-epal-baseis --visualize
.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-large

load_chunks_to_chroma.py refuses to load a curriculum slug when the companion knowledge_graph.json has a blocked audit or no audit metadata. This prevents old or dirty chunks from entering the vector database after the teacher has made tagging changes.

Evaluation Phase

The evaluation layer has three levels:

  1. Deterministic curriculum checks that run locally without an LLM.
  2. A prompt-guardrail context baseline that shows what a prompt-only approach still exposes to the LLM before generation.
  3. Optional RAGAS or LLM-judge metrics that use an evaluator LLM for answer/context quality.

This makes the project easy to test in a development loop while still supporting paper-friendly RAG evaluation later.

Deterministic Constraint Evaluation

Run this after compiling the curriculum and loading ChromaDB:

.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict

This command refreshes the JSON report, manuscript-ready Markdown report, and comparison plots every time it runs. Use --strict for CI-style checks that should exit nonzero when any benchmark assertion fails. For paper-review runs where you want the artifacts to refresh even while retrieval tuning is still in progress, omit --strict.

Generated outputs:

Output Purpose
evaluation/results/latest_evaluation.json Full machine-readable evaluation payload.
evaluation/results/paper_evaluation_report.md Paper-ready tables with counts, 95% Wilson confidence intervals, corpus statistics, and warnings about judge protocol.
evaluation/results/paper_numbers.tex Auto-generated LaTeX macros for corpus counts, benchmark counts, leakage rates, confidence intervals, and optional judge status.
evaluation/results/paper_tables.tex Auto-generated LaTeX tables with numerator/denominator pairs for ablations, failure modes, and deterministic checks.
evaluation/results/plots/*.png Updated visual summaries for the current run.

The deterministic suite compares the ablation variants needed for leakage experiments:

Stage What it measures
Naive RAG baseline Unconstrained vector retrieval before any curriculum constraint.
DAG-only, no V_NODE gates Metadata/chapter/section filtering while ignoring mastery gates.
Full CurriculumRAG Metadata-first retrieval after chapter, section, and V_NODE constraints.
Prompt-guardrail context The same naive context pool placed behind an instruction such as "do not answer outside the student's current chapter." This measures exposed context, not LLM obedience.

To avoid circular evaluation, leakage is measured against independent benchmark annotations such as allowed_chapters or allowed_sections, not against the same DAG rule used by the runtime filter.

Build Evaluation Cases From A Teacher CSV

If a curriculum folder contains a teacher-authored CSV with columns for question, ideal answer, chapter/section lineage, answer AKUs, and page provenance, convert it into the JSON format used by the evaluation harness:

CSV column Meaning in the benchmark
Question or question Student query used by the benchmark case.
Answers_ideal or answer_ideal Teacher reference answer, used as reference for RAGAS and judge datasets.
Chapters_Sections or chapters_sections Learner/query location, such as C03.S02 or C03.S02.SS01.
AKUs or akus_of_answer Comma-separated teacher gold AKUs that should answer the question.
Page_Number_Question or page_no_question Textbook page where the question appears.
Page_Number_Answer or page_no_answer Textbook page where the answer evidence appears.
.\.venv\Scripts\python.exe build_evaluation_cases_from_csv.py g-epal-baseis

For curriculum/g-epal-baseis/g-epal-baseis-dataset.csv, this writes:

curriculum/g-epal-baseis/evaluation_scenarios.json
curriculum/g-epal-baseis/evaluation_cases.json
curriculum/g-epal-baseis/evaluation_cases.skipped.json

The converter writes two benchmark views from the same teacher CSV:

Output Purpose
evaluation_scenarios.json Grouped, peer-review friendly schema. Each teacher question contains learner-state permutations for beginner, intermediate, and advanced personas.
evaluation_cases.json Expanded flat schema consumed by evaluate_curriculum_rag.py. Each scenario becomes one runnable benchmark case.
evaluation_cases.skipped.json Rows that cannot yet be benchmarked because the referenced section or answer AKU is not present in knowledge_chunks.json.

This gives you the scientific structure needed for ablation studies: the same question Q is tested under multiple simulated K_student states instead of being treated as one generic query.

The grouped schema has this shape:

{
  "question_id": "Q015",
  "question": "Ποιες είναι οι απαιτήσεις ACID;",
  "target_section": "C02.S02",
  "ground_truth_akus": ["C02.S02.A03", "C02.S02.A04"],
  "evaluation_scenarios": [
    {
      "persona": "beginner",
      "simulated_k_student": [],
      "expected_status": "blocked",
      "expected_behavior": "Socratic Redirect",
      "test_goal": "Future-Chapter Leakage Prevention"
    },
    {
      "persona": "intermediate",
      "simulated_k_student": ["V_C02_S01_COMPLETE"],
      "expected_status": "ready",
      "expected_behavior": "Allowed / Full Answer",
      "test_goal": "Retrieval Hit Rate Validation"
    },
    {
      "persona": "advanced",
      "simulated_k_student": ["all known V_NODE gates from the graph"],
      "expected_status": "ready",
      "expected_behavior": "Advanced Context (No C01 regression)",
      "test_goal": "Contextual Regression Suppression"
    }
  ]
}

The converter infers:

  • current_chapter and current_section from Chapters_Sections
  • previous chapter and section completed_gates needed to make that learner state reachable
  • evaluation_scenarios for the three standard personas: beginner, intermediate, and advanced
  • expected_ids from teacher-listed AKUs that exist in knowledge_chunks.json
  • teacher_expected_ids from the complete AKU list in the CSV
  • allowed_chapters and allowed_sections from the teacher answer AKUs
  • reference and answer_ideal from the ideal-answer column
  • source_pages.question and source_pages.answer from the page columns

Rows are skipped when the CSV references a section or AKU that is not yet present in knowledge_chunks.json. This is intentional: the skipped report becomes a tagging progress checklist for the teacher.

During active tagging, use --allow-partial to include rows where at least one teacher-listed answer AKU exists. For paper numbers, leave it off so every included case has a fully tagged answer set. Use --include-unmatched only when you want to inspect the dataset JSON before the corresponding AKUs exist in the compiled curriculum.

Run the generated cases with:

.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --cases curriculum/g-epal-baseis/evaluation_cases.json --collection g-epal-baseis --graph curriculum/g-epal-baseis/knowledge_graph.json --query-embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-large --strict

The evaluator can also read the grouped file directly:

.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --cases curriculum/g-epal-baseis/evaluation_scenarios.json --collection g-epal-baseis --graph curriculum/g-epal-baseis/knowledge_graph.json --query-embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-large --strict

The Dev Workbench exposes the same flow in 3. Evaluation: choose the CSV, build the grouped scenarios and expanded cases JSON, inspect skipped rows, then run the evaluation suite.

In other words:

  • pre-filter leakage measures how much inappropriate context semantic search would retrieve before any curriculum constraint is applied
  • DAG-only leakage measures how much inappropriate context survives when the system uses metadata/DAG filtering but ignores V_NODE mastery gates
  • prompt-guardrail context leakage measures how much inappropriate context is still visible to a prompt-only baseline before generation
  • full CurriculumRAG leakage measures how much inappropriate context survives after the orchestrator applies metadata and gate constraints
  • DAG filter violation rate is reported separately as an implementation sanity check, not as the main pedagogical leakage construct

Prompt-Only Actual Generation Baseline

The deterministic prompt-guardrail metric measures context exposure: it shows what forbidden chunks a prompt-only RAG system would place in the LLM context window. To test whether a prompt instruction actually prevents answer leakage, run the optional generation baseline.

This baseline generates two answers per benchmark case:

Stage Context shown to generator What is judged
Prompt-only generation Naive/unconstrained retrieved context plus a curriculum guardrail prompt Whether the generated answer still leaks locked or inappropriate content
Full CurriculumRAG generation CurriculumRAG-filtered context, or a blocked-state redirect Whether the generated answer remains curriculum-safe

Install optional evaluation dependencies and set your LLM key:

uv pip install -r requirements-eval.txt
$env:OPENAI_API_KEY="your-api-key"

Run a cheap pilot first:

.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --cases curriculum/g-epal-baseis/evaluation_cases.json --collection g-epal-baseis --graph curriculum/g-epal-baseis/knowledge_graph.json --query-embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-large --run-generation-baseline --generation-model gpt-4o --generation-judge-model gpt-4o --generation-limit 10

When the prompt and judge rubric look stable, remove --generation-limit for a full run. The results appear in:

evaluation/results/latest_evaluation.json
evaluation/results/paper_evaluation_report.md

Use this table for the paper only when you report the generator model, judge model, case count, and the exact generation/judge prompts from the JSON report.

Pedagogical Failure Personas

Benchmark cases should include personas that expose the failure modes ordinary RAG systems struggle with.

Persona Failure mode What to test
beginner future_chapter_leakage A learner asks with familiar early terms, but the semantically attractive answer lives in a locked future chapter. Full CurriculumRAG should block and respond with a Socratic redirect.
intermediate in_scope_retrieval The learner has the prerequisite gates for the target section. This is the positive-control run for retrieval hit rate and RAGAS faithfulness.
advanced contextual_regression A learner asks a specialized question, but naive retrieval pulls simple already-mastered definitions into the context window. Full CurriculumRAG should keep the context in the target answer scope.
beginner section_gate_leakage A learner asks for a later section inside the same chapter before the section gate is complete.
beginner state_consistency The active chapter/section and completed V_NODE gates disagree, simulating bad imported state or UI drift.
any in_scope_retrieval Positive-control case that should remain answerable.

Example future-leakage case:

{
  "id": "beginner_c05_locked_question",
  "persona": "beginner_student",
  "failure_mode": "future_chapter_leakage",
  "scenario": "Beginner uses Chapter 1 vocabulary, but the answer belongs to locked Chapter 5 content.",
  "question": "How can stored data decide which path to follow later?",
  "current_chapter": "C01",
  "current_section": "C01.S01",
  "completed_gates": [],
  "expected_status": "blocked",
  "expected_ids": [],
  "expected_chapters": [],
  "allowed_chapters": [],
  "allowed_sections": [],
  "ground_truth_basis": "syllabus_position",
  "expected_missing_gates": ["V_C01_COMPLETE", "V_C02_COMPLETE", "V_C03_COMPLETE", "V_C04_COMPLETE"],
  "expected_response_strategy": "socratic_hint",
  "reference": "The answer belongs to a locked future chapter, so the tutor should avoid exposing it and ask a boundary-respecting hint."
}

Example contextual-regression case:

{
  "id": "advanced_er_specialized_query",
  "persona": "advanced_student",
  "failure_mode": "contextual_regression",
  "scenario": "Advanced learner asks about ER modeling; naive retrieval may waste context on basic data definitions.",
  "question": "How do entity relationships constrain the logical database design?",
  "current_chapter": "C05",
  "current_section": "C05.S02",
  "completed_gates": ["V_C01_COMPLETE", "V_C02_COMPLETE", "V_C03_COMPLETE", "V_C04_COMPLETE"],
  "expected_status": "ready",
  "expected_ids": ["C05.S02.A03"],
  "expected_chapters": ["C05"],
  "allowed_chapters": ["C05"],
  "allowed_sections": ["C05.S02"],
  "ground_truth_basis": "syllabus_position",
  "reference": "The answer should use the active advanced chapter, not regress to introductory definitions."
}

In the tiny sample benchmark, Chapter 2 stands in for a locked future chapter. In the full g-epal-baseis evaluation, create the same pattern with the real future chapter, such as Chapter 5.

It measures:

Metric Why it matters
Retrieval hit rate Expected AKUs are retrieved for in-scope questions.
Gate block accuracy Future chapters are blocked when V_NODE gates are missing.
Pre-filter pedagogical leakage rate Inappropriate context in the naive semantic candidate pool.
DAG-only pedagogical leakage rate Inappropriate context after metadata/DAG filtering with V_NODEs disabled.
Prompt-guardrail context leakage rate Inappropriate context that prompt instructions would have to suppress after retrieval.
Post-filter pedagogical leakage rate Inappropriate context that survives curriculum filtering.
DAG-only leakage reduction How much metadata/DAG filtering reduces leakage compared with naive retrieval.
Full CurriculumRAG leakage reduction How much the full gate-aware system reduces leakage compared with naive retrieval.
DAG filter violation rate Implementation defect signal for metadata/filter edge cases.
Lineage completeness Retrieved atoms carry chapter/section/subsection context.

The report includes denominators for the leakage rates:

Judged chunks: pre=<n> prompt=<n> dag_only=<n> post=<n>
Invalid chunks: pre=<n> prompt=<n> dag_only=<n> post=<n>

For paper tables, report those counts together with the percentages and the generated 95% confidence intervals. Do not reuse percentages from an older judging protocol after changing the judge.

Formal leakage definition:

J(c, q, K_student) -> {0, 1}

where J is an evaluation judge that returns 1 when chunk c is pedagogically inappropriate for query q and learner state K_student. In deterministic mode, J is implemented by benchmark annotations such as allowed_chapters and allowed_sections. In an LLM-judge experiment, J must be replaced by the frozen judge prompt and syllabus context used for that run.

For a retrieval stage s with candidate set C_s(q, K_student):

DLS_s = (1 / |C_s|) * sum_{c in C_s} J(c, q, K_student)

Report at least:

DLS_naive     for naive semantic candidates
DLS_dag       for DAG-only retrieval without V_NODE gates
DLS_full      for full CurriculumRAG retrieval
DLS_prompt    for prompt-guardrail context exposure, if included

If J changes from deterministic annotations to an LLM judge, all stages must be re-judged under that same J.

The latest JSON report is written to:

evaluation/results/latest_evaluation.json

That folder is ignored by git so repeated local experiments stay out of commits.

Independent Benchmark Ground Truth

To avoid the DLS circularity problem, benchmark cases must define the leakage criterion outside the runtime DAG filter. Each case should include allowed_chapters and a ground_truth_basis.

Example:

{
  "id": "epal_c03_databases_blocked",
  "question": "How does a database table store records?",
  "current_chapter": "C02",
  "completed_gates": ["V_C01_COMPLETE"],
  "expected_status": "blocked",
  "expected_ids": [],
  "expected_chapters": [],
  "allowed_chapters": [],
  "ground_truth_basis": "syllabus_position",
  "expected_missing_gates": ["V_C02_COMPLETE"],
  "reference": "This topic belongs to a later chapter and should not be answered yet."
}

Recommended values for ground_truth_basis:

Basis Meaning
syllabus_position The official curriculum order says which chapters are allowed.
teacher_annotation A human teacher judged the chunk appropriate or inappropriate.
llm_judge_syllabus_rubric A separate LLM judged appropriateness from the natural-language syllabus, not the compiled DAG. Use only when the full LLM-judge protocol was actually run.

This separation matters:

  • the DAG filter is the system policy
  • allowed_chapters is the evaluation criterion
  • the report compares naive, DAG-only, prompt-context, and full-system leakage against that external criterion
  • dag_filter_violation_rate is only an implementation sanity check

When using llm_judge_syllabus_rubric, also report:

  • judge model and version
  • judge prompt in an appendix
  • natural-language syllabus context shown to the judge
  • number of judged (query, chunk, learner_state) triples
  • how many queries, personas, and curriculum states produced those triples
  • agreement or stability checks, such as repeated LLM judging or human expert validation on a sample
  • whether the same judge re-scored every stage: pre-filter, DAG-only, prompt-guardrail, and post-filter

Until those fields exist, describe the current evaluation as deterministic benchmark evaluation, not as LLM-judged leakage.

LLM Or Human Judge Protocol

The evaluator can export every retrieved chunk that needs independent pedagogical judging:

.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict --export-judge-dataset

This writes:

evaluation/results/judge_dataset.jsonl

Each JSONL row contains:

Field Meaning
judgment_key Stable key for loading the judgment back into the report.
stage pre_filter, prompt_guardrail, dag_only, or post_filter.
question Student query used in the benchmark case.
learner_state Current chapter/section, completed gates, and allowed scope.
atom_id Retrieved AKU ID.
chunk_text Text shown to the independent judge.
deterministic_pedagogically_invalid Local benchmark label for comparison/debugging only.

A completed judge result can be JSON or JSONL and should include either judgment_key or the triple case_id, stage, atom_id, plus one of:

{
  "judgment_key": "c02_blocked_without_gate::pre_filter::C02.S01.A01",
  "pedagogically_invalid": true,
  "confidence": 0.94,
  "rationale": "The chunk introduces Chapter 2 before the learner has completed Chapter 1."
}

Load completed judgments back into the paper report:

.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict --judge-results evaluation/results/judge_results.jsonl

For a pilot OpenAI-backed judge run, provide the natural-language syllabus or tagged textbook Markdown. Use --judge-limit for a cheap smoke test first:

$env:OPENAI_API_KEY="your-api-key"
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict --run-llm-judge --judge-syllabus curriculum/g-epal-baseis/original_tagged.md --judge-model gpt-4o --judge-limit 20

The Dev Workbench exposes the same protocol under 3. Benchmark -> Optional independent LLM-as-a-Judge. Start with a small pilot limit there too; the UI will mark partial judge runs as pilots so they are not mistaken for paper-ready headline results.

Only use the LLM-judge leakage table in a paper when the external judge summary says all exported records were matched. Partial pilot runs are useful for debugging the rubric, not for headline numbers.

Optional RAGAS Evaluation

Install the extra evaluation dependencies:

uv pip install -r requirements-eval.txt

Set your evaluator LLM key, then run:

$env:OPENAI_API_KEY="your-api-key"
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --run-ragas --ragas-model gpt-4o

The RAGAS path builds rows with user_input, retrieved_contexts, response, and reference, then evaluates them with context recall, faithfulness, and factual correctness metrics.

For automated testing or CI, use deterministic mode first:

.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict --json

If a CI job only needs pass/fail output and should skip image generation:

.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict --no-plots

Visual Comparison Plots

Plots are generated by default when the evaluation suite runs:

.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict

This writes:

Plot What it shows
evaluation_summary_metrics.png Pass rate, retrieval hit rate, gate accuracy, pre/prompt/DAG-only/full leakage, DAG violations, and lineage completeness.
leakage_ablation.png Naive RAG baseline vs DAG-only vs Full CurriculumRAG.
leakage_comparison.png Naive RAG, prompt-guardrail context, DAG-only, and Full CurriculumRAG.
retrieval_counts_by_case.png How many atoms each retrieval variant returned per benchmark case.
deterministic_check_matrix.png Pass/fail matrix for each deterministic benchmark check.

By default, plots are written to:

evaluation/results/plots/

Output Model

Each AKU chunk includes:

  • id: The AKU identifier, such as C01.S01.SS01.A01.
  • text: The source text inside the AKU block.
  • lineage: Parent chapter, section, and subsection metadata.
  • chapter_root: The top-level chapter context.
  • source_lines: Start and end lines in the source file.

Edges are typed:

  • hierarchy: Parent-to-child curriculum structure.
  • dependency: Prerequisite-to-gate or prerequisite-to-node logic.

The graph JSON stores prerequisites in two places for different consumers:

  • each explicit AKU or V_NODE node keeps its parsed requires list
  • the top-level edges array contains the derived dependency edges

The requires field is the source annotation. The dependency edges are derived from it during compilation so graph libraries can traverse the DAG directly. If they disagree, treat it as a compiler defect and rerun the compiler from the tagged Markdown.

The graph JSON also embeds an audit object:

{
  "status": "clean",
  "error_count": 0,
  "warning_count": 1,
  "is_acyclic": true,
  "findings": []
}

status: blocked means the teacher should fix original_tagged.md before loading chunks or running retrieval. The same audit is written separately as knowledge_graph_audit.json for easier inspection.

Before using a compiled graph in paper experiments, check:

  • the graph is acyclic
  • every requires ID exists as an explicit node, not only as an accidental virtual placeholder
  • every V_NODE has the expected parent hierarchy edge
  • section and chapter completion gates list the full completion contract

Malformed prerequisite IDs are intentionally visible in the graph as virtual nodes. That makes mistakes like C0.S0.A0, C01.S0.A03, or 01.S03.A08 auditable instead of silently ignored.

Why This Exists

CurriculumRAG gives a curriculum two useful forms at once:

  1. A graph for structural reasoning, prerequisite tracing, and validation gates.
  2. Clean chunks for semantic retrieval, embeddings, and RAG pipelines.

The result is a knowledge base that remembers both the words and the shape of the curriculum.

About

Curriculum Contrained RAG

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages