A small "Compiler for Knowledge" that turns tagged curriculum Markdown into a graph-shaped, VectorDB-ready knowledge base.
CurriculumRAG reads a gold-standard Markdown file, recognizes curriculum atoms and validation gates, builds a directed NetworkX graph, and exports structured JSON chunks with lineage metadata for retrieval-augmented generation workflows.
| Layer | Main command | Output |
|---|---|---|
| Author tagged Markdown | edit original_tagged.md |
AKU and V_NODE curriculum source |
| Compile curriculum | knowledge_compiler.py <source-or-slug> |
graph JSON, chunks JSON, graph PNG |
| Index chunks | load_chunks_to_chroma.py <chunks-or-slug> |
persistent ChromaDB collection |
| Retrieve safely | retrieval_orchestrator.py ... |
scoped context and teacher prompt |
| Evaluate | evaluate_curriculum_rag.py --strict |
metrics, JSON report, plots |
| Explore interactively | streamlit run dev_workbench.py |
local Dev Workbench UI |
The core idea is simple: semantic search can find relevant text, but the curriculum graph decides whether the learner is allowed to see it yet.
Compilation includes a blocking graph audit. Cycles, malformed IDs, duplicate
tags, missing requires: targets, unclosed tags, and nested tags stop the
pipeline before chunks are loaded into ChromaDB.
For the built-in sample curriculum:
.\.venv\Scripts\python.exe knowledge_compiler.py gold-standard.md --chunks knowledge_chunks.json --graph knowledge_graph.json --visualize knowledge_graph.png
.\.venv\Scripts\python.exe load_chunks_to_chroma.py knowledge_chunks.json --persist-dir chroma_db --collection curriculum_rag --reset
.\.venv\Scripts\python.exe retrieval_orchestrator.py "What is an if statement?" --current-chapter C02 --completed-gate V_C01_COMPLETEFor a real textbook folder such as curriculum/g-epal-baseis/:
.\.venv\Scripts\python.exe knowledge_compiler.py g-epal-baseis --visualize
.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-large
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE --query-embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-largeFor the Greek EPAL textbook, prefer sentence-transformers with
intfloat/multilingual-e5-large. The first run may download the model from
Hugging Face.
You can also test a Greek-specific AUEB Greek BERT preset:
.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model nlpaueb/bert-base-greek-uncased-v1
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE --query-embedding-mode sentence-transformers --embedding-model nlpaueb/bert-base-greek-uncased-v1For a lighter E5 comparison, use:
.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-small
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE --query-embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-smallUse the same embedding model for loading and querying. For a fully offline smoke test, add hash embeddings to both load and query commands:
.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode hash
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE --query-embedding-mode hashTo use the interactive local workbench:
.\.venv\Scripts\streamlit.exe run dev_workbench.pyThe workbench lets you compile, audit, load ChromaDB, compare Naive RAG vs DAG-only vs Full CurriculumRAG, inspect retrieved AKUs and lineage metadata, and rerun the evaluation plots from one browser panel.
| Section | Use it when you want to... |
|---|---|
| Visual Overview | understand the system shape quickly |
| Implemented Architecture | see the runtime contract for AI IDEs and agents |
| Curriculum Tagging Guide | tag a parsed PDF/Markdown textbook |
| Curriculum Folder Workflow | run commands by curriculum slug, such as g-epal-baseis |
| Local Setup And Development Guide | clone, install, compile, load, and test locally |
| Evaluation Phase | run leakage checks, plots, and RAGAS/LLM evaluation |
| Output Model | inspect chunk JSON, graph JSON, lineage, and edges |
The project has two complementary views: a human-facing curriculum RAG timeline, and a node architecture view that shows how chapters, AKUs, validation gates, and future retrieval strategies fit together.
flowchart TD
subgraph Authoring["1. Curriculum Authoring"]
Markdown["gold-standard.md<br/>Tagged Markdown with headings, AKUs, and V_NODE gates"]
end
subgraph Compile["2. Knowledge Compilation"]
Compiler["knowledge_compiler.py"]
Graph["NetworkX DiGraph<br/>hierarchy + dependency edges"]
Chunks["knowledge_chunks.json<br/>AKU text + lineage metadata"]
GraphJson["knowledge_graph.json<br/>nodes, edges, V_NODE gates"]
GraphPng["knowledge_graph.png<br/>visual graph snapshot"]
end
subgraph Index["3. Chroma Indexing"]
Loader["load_chunks_to_chroma.py"]
Chroma["Persistent ChromaDB collection<br/>documents + embeddings + metadata"]
end
subgraph Runtime["4. Curriculum-Constrained Runtime"]
Question["Student question"]
State["Learner state<br/>current_chapter + completed_gates"]
Orchestrator["retrieval_orchestrator.py<br/>metadata-first retrieval"]
Gate["Gate check<br/>block future chapters until V_NODE completion"]
Filter["Chroma where filter<br/>for example chapter_id = C02"]
Inspector["Context inspector<br/>retrieved AKU ids + lineage"]
Prompt["Pedagogical prompt<br/>system prompt + scoped context"]
Teacher["LLM teacher response"]
end
Markdown --> Compiler
Compiler --> Graph
Compiler --> Chunks
Compiler --> GraphJson
Compiler --> GraphPng
Chunks --> Loader
Loader --> Chroma
Question --> Orchestrator
State --> Orchestrator
GraphJson --> Orchestrator
Orchestrator --> Gate
Gate --> Filter
Filter --> Chroma
Chroma --> Orchestrator
Orchestrator --> Inspector
Orchestrator --> Prompt
Prompt --> Teacher
For AI IDEs and future agents, the runtime contract is:
- Compile tagged curriculum into
knowledge_chunks.jsonandknowledge_graph.json. - Load chunks into ChromaDB with metadata fields such as
chapter_id,lineage_json,lineage_ids,lineage_labels, andsource_line_start. - At inference time, call
retrieval_orchestrator.pybefore constructing the LLM prompt. - The orchestrator decides learner scope, applies a Chroma
wherefilter, reconstructs lineage, and emits the teacher-facing prompt payload. - If the requested chapter is outside the learner's unlocked scope, the orchestrator blocks retrieval instead of leaking future curriculum content.
| File | Purpose |
|---|---|
gold-standard.md |
Source curriculum written with AKU and V_NODE tags. |
curriculum/<slug>/original_tagged.md |
Real textbook workspace source, addressed by slug commands such as g-epal-baseis. |
curriculum_paths.py |
Shared helper that resolves curriculum slugs to source, graph, chunks, and collection names. |
knowledge_compiler.py |
Parser/compiler that creates the graph and chunk exports. |
load_chunks_to_chroma.py |
Loader that upserts AKU chunks into a persistent ChromaDB collection. |
retrieval_orchestrator.py |
Metadata-first retrieval layer that enforces chapter and gate constraints. |
evaluate_curriculum_rag.py |
Evaluation harness for deterministic constraint checks and optional RAGAS scoring. |
build_evaluation_cases_from_csv.py |
Converts a teacher CSV dataset into benchmark-case JSON for evaluation. |
dev_workbench.py |
Streamlit UI for local compile, audit, retrieval comparison, and evaluation runs. |
evaluation/benchmark_cases.json |
Small gold benchmark set for retrieval, lineage, and gate behavior. |
curriculum/<slug>/evaluation_scenarios.json |
Grouped teacher-question benchmark with beginner/intermediate/advanced persona permutations. |
curriculum/<slug>/evaluation_cases.json |
Expanded evaluator-ready cases generated from the grouped persona scenarios. |
evaluation/results/paper_evaluation_report.md |
Generated manuscript-ready evaluation summary with counts, confidence intervals, warnings, and tables. |
evaluation/results/judge_dataset.jsonl |
Optional exported (query, chunk, learner_state) triples for LLM or human pedagogical judging. |
knowledge_chunks.json |
VectorDB-ready AKU chunks with lineage metadata. |
knowledge_graph.json |
Full graph export containing nodes and edges. |
knowledge_graph_audit.json |
Teacher-facing graph audit with blocking errors and repair hints. |
knowledge_graph.png |
Visual snapshot of the curriculum graph. |
docs/assets/*.png |
README infographics for the curriculum RAG timeline and node architecture. |
requirements.txt |
Python dependencies. |
requirements-eval.txt |
Optional evaluation dependencies for RAGAS and pytest. |
Use this guide when you have a Markdown file parsed from a PDF textbook, syllabus, or official curriculum document and you want to prepare it for CurriculumRAG.
The goal is to turn ordinary Markdown into a structured curriculum source that can be compiled into:
- graph nodes for chapters, sections, subsections, AKUs, and validation gates
- hierarchy edges such as chapter -> section -> AKU
- dependency edges such as prerequisite AKU -> validation gate
- VectorDB-ready chunks with lineage metadata
CurriculumRAG treats the curriculum as a hierarchy of small knowledge atoms.
Chapter
Section
Subsection
AKU: Atomic Knowledge Unit
V_NODE: Validation / completion gate
An AKU is the smallest useful learning unit that should become one retrieval
chunk. A V_NODE is a checkpoint that says when a learner is allowed to move
forward.
| Need | Pattern |
|---|---|
| Chapter | C01 |
| Section | C01.S02 |
| AKU in a section | C01.S02.A03 |
| AKU in a subsection | C01.S02.SS01.A03 |
| Section completion gate | V_C01_S02_COMPLETE |
| Chapter completion gate | V_C01_COMPLETE |
| Immediate prerequisite | requires: [C01.S02.A02] |
| Full section contract | requires: [V_C01_S01_COMPLETE, C01.S02.A01, C01.S02.A02] |
| AKU range shorthand | requires: [C01.S02.A01-07] expands to A01 through A07 |
Rule of thumb:
AKU requires = immediate conceptual dependency
V_NODE requires = explicit completion contract
Every curriculum unit uses a stable ID. The ID tells the compiler where the unit lives in the curriculum.
| ID part | Meaning | Example |
|---|---|---|
C01 |
Chapter 1 | C01 |
S01 |
Section 1 inside the chapter | C01.S01 |
SS01 |
Subsection 1 inside the section | C01.S01.SS01 |
A01 |
Atomic Knowledge Unit 1 | C01.S01.SS01.A01 |
Examples:
| ID | Meaning |
|---|---|
C01 |
Chapter 1 |
C01.S02 |
Chapter 1, Section 2 |
C01.S02.SS03 |
Chapter 1, Section 2, Subsection 3 |
C01.S02.SS03.A04 |
Fourth AKU inside that subsection |
C02.S01.A01 |
First AKU directly inside Chapter 2, Section 1 |
Use two digits for each number. Prefer C01.S01.A01 over C1.S1.A1.
Stable IDs matter because benchmark cases, prerequisite gates, graph edges, and
retrieval filters all refer to them.
Headings create the parent hierarchy. The compiler recognizes chapter and numbered headings and turns them into graph nodes.
Use this style whenever possible:
# CHAPTER 1: Introduction to Computer Science
## 1.1 Basic Concepts
### 1.1.1 Data and InformationThis produces:
| Heading | Node ID |
|---|---|
# CHAPTER 1: ... |
C01 |
## 1.1 ... |
C01.S01 |
### 1.1.1 ... |
C01.S01.SS01 |
You may add a short chapter description after >:
# CHAPTER 1: Introduction to Computer Science > Level 1: Foundational concepts.The text before > becomes the label. The text after > becomes the
description.
Wrap each atomic learning unit in [AKU: ...] and [/AKU].
[AKU: C01.S01.SS01.A01]
Data is raw facts or symbols that can be processed by a computer.
[/AKU]The compiler exports this as one chunk in knowledge_chunks.json.
Each AKU should be:
- small enough to retrieve as one concept
- self-contained enough to answer a learner question
- tied to exactly one curriculum location
- stable enough that its ID will not change casually
Good AKU:
[AKU: C01.S01.SS01.A01]
Data is raw facts or symbols. Information is data that has been processed so it
has meaning for a user.
[/AKU]Too large:
[AKU: C01.S01.SS01.A01]
All of Chapter 1 pasted here...
[/AKU]Too small:
[AKU: C01.S01.SS01.A01]
Data.
[/AKU]Inside an AKU block, you may add metadata lines before the content.
[AKU: C01.S01.SS01.A01]
type: concept
Data is raw facts or symbols. Information is processed data with meaning.
[/AKU]Supported metadata:
| Metadata | Purpose |
|---|---|
type: concept |
Marks the pedagogical type of the AKU. |
requires: [C01.S01.SS01.A01] |
Declares prerequisite AKUs. |
access: preview |
Marks an overview AKU that may be visible before the chapter is fully unlocked. |
page_start: 34 |
Optional source provenance from the textbook. |
page_end: 36 |
Optional source provenance when an AKU spans pages. |
source: EPAL official textbook |
Optional human-readable source label. |
type: and requires: may be written as plain metadata lines or hidden HTML
comments. Extra provenance metadata should be written as hidden comments so the
compiler never confuses ordinary textbook text with metadata:
<!-- page_start: 34 -->
<!-- page_end: 36 -->
<!-- source: EPAL official textbook -->Example with prerequisites:
[AKU: C01.S01.SS02.A01]
type: procedure
requires: [C01.S01.SS01.A01]
To classify information, first identify the raw data and then explain what
meaning was added through processing.
[/AKU]The compiler creates dependency edges from every requires item to the current
node.
Use type: to make the pedagogical role of an AKU explicit. The value can be a
single label or a short comma-separated list when the unit has more than one
role.
<!-- type: concept, abstraction -->Recommended labels:
| Type | Use when the AKU... | Example |
|---|---|---|
objective |
states expected learning outcomes | "The student should be able to..." |
concept |
explains an idea or principle | data independence, DBMS purpose |
definition |
defines a term precisely | file, record, field, entity |
example |
illustrates a previous concept | customer-card file example |
exercise |
asks the learner to answer, practice, or investigate | questions / activities |
procedure |
gives steps the learner should execute | normalize a relation step by step |
process |
describes a sequence, lifecycle, or flow | database design phases |
model |
introduces a representation or framework | entity-relationship model |
abstraction |
explains levels, views, hiding details, or independence | logical/physical data independence |
comparison |
contrasts two or more concepts | conceptual vs physical design |
warning |
highlights a common mistake or constraint | duplicate keys are not allowed |
summary |
recaps prior material | end-of-section synthesis |
Common combinations:
| Combination | Use for |
|---|---|
concept, abstraction |
ideas about levels, views, or independence, such as data independence |
concept, model |
introduction of a conceptual model such as ER modeling |
concept, comparison |
explaining differences between related ideas |
exercise, reflection |
open-ended questions or classroom activities |
process, model |
lifecycle phases tied to a modeling framework |
Prefer abstraction over abstract. The word abstract can sound like a
paper abstract, while abstraction clearly means hiding lower-level details or
working through levels of representation.
Use V_NODE blocks to represent curriculum milestones, chapter completion
checks, or prerequisite gates.
### CHAPTER 1 COMPLETION GATE
[V_NODE: V_C01_COMPLETE]
requires: [C01.S01.SS01.A01, C01.S01.SS02.A01]
[/V_NODE]This means the learner should complete the required AKUs before Chapter 1 is considered complete. The retrieval orchestrator can then use the gate state to prevent future-chapter leakage.
Recommended gate names:
| Gate ID | Meaning |
|---|---|
V_C01_COMPLETE |
Chapter 1 completion gate |
V_C02_COMPLETE |
Chapter 2 completion gate |
V_C01_S01_COMPLETE |
Chapter 1, Section 1 completion gate |
V_C01_S02_COMPLETE |
Chapter 1, Section 2 completion gate |
V_UNIT01_COMPLETE |
Unit/module completion gate |
For chapter gates, prefer V_C##_COMPLETE because it matches the section-gate
style, such as V_C01_S02_COMPLETE. The tools still accept older
V_CH##_COMPLETE inputs and normalize them to the same chapter gate.
For section gates, prefer V_C##_S##_COMPLETE. This allows the orchestrator to
control movement inside a chapter. For example, a learner at C01.S01 should
not receive C01.S02 chunks until V_C01_S01_COMPLETE is marked complete.
Section gate example:
## 1.1 Data and Information
[AKU: C01.S01.A01]
type: concept
Data is raw facts or symbols that can be processed by a computer.
[/AKU]
### SECTION 1.1 COMPLETION GATE
[V_NODE: V_C01_S01_COMPLETE]
requires: [C01.S01.A01]
[/V_NODE]
## 1.2 Information Processing
[AKU: C01.S02.A01]
type: process
requires: [C01.S01.A01]
Processing transforms data into useful information.
[/AKU]Runtime example:
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Explain section 1.2" --current-section C01.S01This blocks because V_C01_S01_COMPLETE is missing. Once that gate is complete:
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Explain section 1.2" --current-section C01.S02 --completed-gate V_C01_S01_COMPLETEUse different levels of redundancy for AKUs and validation gates.
For AKUs, keep requires local and conceptual. Do not repeat the full ancestry
unless the AKU directly depends on every listed concept.
Good:
<!-- [AKU: C01.S04.A03] -->
<!-- type: exercise -->
<!-- requires: [C01.S04.A02] -->
### Ε�ωτήσεις / Δ�αστη�ι�τητες
- Αναζήτηση πληΟ�οφοΟ�ΞΉΟ�Ξ½ Ξ³ΞΉΞ± τον εκτιμΟ�ΞΌΞµΞ½ΞΏ Ο�Ξ³ΞΊΞΏ δεδομΞνων.
<!-- [/AKU] -->Usually avoid this for an AKU:
<!-- [AKU: C01.S04.A03] -->
<!-- requires: [C01.S01.A01, C01.S02.A04, C01.S03.A02, C01.S04.A02] -->
...
<!-- [/AKU] -->The graph can recover the longer dependency path by traversing earlier
requires edges. Writing the full path on every AKU makes the graph noisy and
harder for a teacher to maintain.
For V_NODEs, redundancy is intentional because the gate is a completion contract. A section gate should include the previous section gate plus the AKUs that must be completed in the current section.
<!-- [V_NODE: V_C01_S04_COMPLETE] -->
<!-- requires: [V_C01_S03_COMPLETE, C01.S04.A01, C01.S04.A02, C01.S04.A03] -->
<!-- [/V_NODE] -->Do not duplicate the whole ancestry behind V_C01_S03_COMPLETE. The previous
gate already carries that path.
For chapter completion, list the required section gates:
<!-- [V_NODE: V_C01_COMPLETE] -->
<!-- requires: [V_C01_S01_COMPLETE, V_C01_S02_COMPLETE, V_C01_S03_COMPLETE, V_C01_S04_COMPLETE] -->
<!-- [/V_NODE] -->Recommended convention:
AKU requires = immediate conceptual prerequisites
Section V_NODE = previous section gate + this section's required AKUs
Chapter V_NODE = required section gates
This keeps AKU dependencies clean while giving the teacher an explicit mastery contract at every section and chapter boundary.
Use the smallest structural level that actually exists in the textbook.
If a section has no subsections, section-level AKUs are fine:
## 3.1 Database Design
<!-- [AKU: C03.S01.A01] -->
<!-- type: concept -->
Database design starts from user and system requirements.
<!-- [/AKU] -->
<!-- [AKU: C03.S01.A02] -->
<!-- type: concept -->
<!-- requires: [C03.S01.A01] -->
The design process turns requirements into a database structure.
<!-- [/AKU] -->If the section is divided into subsections, prefer putting AKUs inside the subsection IDs and avoid mixing in sibling section-level AKUs unless the content really belongs to the section as a whole.
Preferred:
## 3.1 Database Design
### 3.1.2 Physical Design
<!-- [AKU: C03.S01.SS02.A04] -->
<!-- type: concept -->
<!-- requires: [C03.S01.SS02.A03] -->
Physical design describes how data is stored and accessed on storage media.
<!-- [/AKU] -->
<!-- [AKU: C03.S01.SS02.A05] -->
<!-- type: exercise -->
<!-- requires: [C03.S01.SS02.A04] -->
Questions and activities about conceptual, logical, and physical design.
<!-- [/AKU] -->Avoid mixing this into the same subsection sequence:
<!-- [AKU: C03.S01.A03] -->
...
<!-- [/AKU] -->
<!-- [AKU: C03.S01.SS02.A04] -->
...
<!-- [/AKU] -->That pattern is legal, but it is harder for teachers and evaluators to read.
It can become unclear whether C03.S01.A03 is a true section-level atom or an
atom that should have been placed under a subsection.
For subsection gates, use the previous subsection gate plus the AKUs inside the current subsection:
<!-- [V_NODE: V_C03_S01_SS02_COMPLETE] -->
<!-- requires: [V_C03_S01_SS01_COMPLETE, C03.S01.SS02.A01, C03.S01.SS02.A02, C03.S01.SS02.A03, C03.S01.SS02.A04, C03.S01.SS02.A05] -->
<!-- [/V_NODE] -->Then roll subsection gates up into the section gate:
<!-- [V_NODE: V_C03_S01_COMPLETE] -->
<!-- requires: [V_C03_S01_SS01_COMPLETE, V_C03_S01_SS02_COMPLETE] -->
<!-- [/V_NODE] -->Recommended structural rule:
Section with no subsections -> use C03.S01.A01, C03.S01.A02
Section with subsections -> use C03.S01.SS01.A01, C03.S01.SS02.A01
Subsection V_NODE -> previous subsection gate + current subsection AKUs
Section V_NODE with subsections -> required subsection gates
These conventions are not just formatting preferences. They make the Markdown easier for teachers to maintain, easier for AI IDEs to inspect, and safer for the compiler to convert into a clean prerequisite DAG.
An AKU should normally contain one teachable idea, example, rule, diagram, activity, or explanation. Avoid wrapping a whole subsection as one AKU if it contains several concepts that could be taught or retrieved separately.
Good:
<!-- [AKU: C03.S03.SS01.A04] -->
<!-- type: concept -->
<!-- requires: [C03.S03.SS01.A03] -->
Each entity is described by attributes.
<!-- [/AKU] -->
<!-- [AKU: C03.S03.SS01.A05] -->
<!-- type: example -->
<!-- requires: [C03.S03.SS01.A04] -->
For example, a Student entity may have name, address, and registration number.
<!-- [/AKU] -->Avoid:
<!-- [AKU: C03.S03.SS01.A01] -->
All definitions, examples, diagrams, and activities for the whole subsection.
<!-- [/AKU] -->For AKUs, requires should list the immediate conceptual prerequisites. The
compiler can recover the full ancestry by traversing the graph.
Good:
<!-- [AKU: C03.S05.SS02.A04] -->
<!-- type: definition, rules -->
<!-- requires: [C03.S05.SS02.A03] -->
Rules for transforming an ER diagram into the relational model.
<!-- [/AKU] -->Usually avoid:
<!-- requires: [C03.S05.A01, C03.S05.SS01.A01, C03.S05.SS01.A02, C03.S05.SS02.A03] -->That long version is harder to maintain and creates noisy edges without adding new information.
Sometimes a chunk continues an earlier example but also depends on a rule that was introduced between the two example parts. In that case it is fine to list both dependencies, even if one dependency already points to the other.
Good:
<!-- [AKU: C03.S05.SS02.A05] -->
<!-- type: example, continuation -->
<!-- requires: [C03.S05.SS02.A03, C03.S05.SS02.A04] -->
The same ER example is now transformed using the rules above.
<!-- [/AKU] -->Here A03 preserves the example lineage, while A04 captures the rule needed
to understand the continuation.
A V_NODE should say what must be completed before the learner is considered to have mastered that section, subsection, or chapter. Unlike AKUs, V_NODEs can be more explicit because they are teacher-facing checkpoints.
Good section gate:
<!-- [V_NODE: V_C02_S03_COMPLETE] -->
<!-- requires: [V_C02_S02_COMPLETE, C02.S03.A01, C02.S03.A02, C02.S03.A03, C02.S03.A04, C02.S03.A05, C02.S03.A06, C02.S03.A07] -->
<!-- [/V_NODE] -->If you want extra readability, you may include earlier section gates too:
<!-- requires: [V_C02_S01_COMPLETE, V_C02_S02_COMPLETE, C02.S03.A01, C02.S03.A02] -->The shorter immediate-gate version is usually cleaner. The longer version is acceptable when a teacher wants the path to be visible in the Markdown.
When a V_NODE requires a consecutive run of AKUs, you may use a compact range instead of writing every ID by hand.
Good:
<!-- [V_NODE: V_C04_S01_SS01_COMPLETE] -->
<!-- requires: [V_C03_COMPLETE, C04.S01.SS01.A01-12] -->
<!-- [/V_NODE] -->The compiler expands this internally into:
C04.S01.SS01.A01
C04.S01.SS01.A02
...
C04.S01.SS01.A12
Use this shorthand mainly for V_NODE completion contracts. For AKU prerequisites, explicit immediate dependencies are usually clearer because an AKU should rarely depend on a whole consecutive range.
Range rules:
- the range must stay inside one AKU prefix, such as
C04.S01.SS01.A - write the start and end atom numbers only:
A01-12 - do not use ranges for non-consecutive concepts
- if one AKU in the range is optional or enrichment, write the list manually so the V_NODE does not accidentally require it
The V_NODE ID should match the curriculum scope of the AKUs inside it.
Good:
<!-- [V_NODE: V_C03_S05_SS01_COMPLETE] -->
<!-- requires: [C03.S05.SS01.A01, C03.S05.SS01.A02, C03.S05.SS01.A03] -->
<!-- [/V_NODE] -->Avoid:
<!-- [V_NODE: V_C03_S05_SS02_COMPLETE] -->
<!-- requires: [C03.S05.SS01.A01, C03.S05.SS01.A02, C03.S05.SS01.A03] -->
<!-- [/V_NODE] -->That second example says "SS02 complete" but contains only SS01 AKUs, which will confuse graph inspection and evaluation.
If a new section starts after a previous section is completed, the cleanest place to encode that flow is often the V_NODE, not every AKU.
Good:
<!-- [V_NODE: V_C03_S05_COMPLETE] -->
<!-- requires: [V_C03_S04_COMPLETE, C03.S05.A02, V_C03_S05_SS01_COMPLETE, V_C03_S05_SS02_COMPLETE] -->
<!-- [/V_NODE] -->This keeps the progression from section 3.4 to section 3.5 visible without
forcing every section 3.5 AKU to repeat V_C03_S04_COMPLETE.
The first teachable AKU of a new chapter can point to the previous chapter gate when the chapter truly depends on prior completion.
<!-- [AKU: C03.S01.A01] -->
<!-- type: introduction -->
<!-- requires: [V_C02_COMPLETE] -->
Chapter 3 introduces database design.
<!-- [/AKU] -->Then the first section or subsection V_NODE can include that AKU as part of the completion contract.
Some chapter introductions are safe and useful before the learner formally unlocks the chapter. Objectives, high-level summaries, and "what is this chapter about?" atoms can be tagged as preview content. This lets the system distinguish between orientation material and gated instructional content.
Good:
# κεφάλαιο *4*
**Ξ— ΓλΟ�σσα Βάσεων ΔεδομΞνων SQL**
## Εισαγωγή
<!-- [AKU: C04.A01] -->
<!-- type: objective, overview -->
<!-- access: preview -->
> ΣτΟ�χοι / Ξ Ο�οσδοκΟ�ΞΌΞµΞ½Ξ± ΑποτελΞσματα
<!-- [/AKU] -->
<!-- [AKU: C04.A02] -->
<!-- type: concept, overview -->
<!-- access: preview -->
Ξ— SQL ΞµΞ―Ξ½Ξ±ΞΉ Ξ³Ξ»Ο�σσα διαχείΟ�ισης ΣχεσιακΟ�Ξ½ Βάσεων ΔεδομΞνων.
<!-- [/AKU] -->
<!-- [AKU: C04.S01.A01] -->
<!-- type: concept -->
<!-- requires: [V_C03_COMPLETE, C04.A02] -->
The first teachable SQL topic that should be gated behind Chapter 3 completion.
<!-- [/AKU] -->Rules for preview AKUs:
- use unique IDs; do not reuse
C04.A01for both objectives and concept text - use
access: previewonly for orientation content, not core future-chapter explanations or assessment answers - keep real instructional AKUs gated with
requires: [V_C03_COMPLETE, ...] - leave preview AKUs out of the chapter completion V_NODE unless they are truly required for mastery
The compiler and Chroma loader preserve access as metadata. Retrieval policy
can then allow access: preview atoms as a controlled exception while keeping
the rest of the chapter locked.
If a chapter has six required sections, the chapter gate should include all six section completion gates. Do not accidentally omit the final section.
<!-- [V_NODE: V_C02_COMPLETE] -->
<!-- requires: [V_C01_COMPLETE, V_C02_S01_COMPLETE, V_C02_S02_COMPLETE, V_C02_S03_COMPLETE, V_C02_S04_COMPLETE, V_C02_S05_COMPLETE, V_C02_S06_COMPLETE] -->
<!-- [/V_NODE] -->Include the previous chapter gate if the chapter itself is sequentially locked behind earlier curriculum progress.
An AKU is not automatically an orphan just because it is not required by a
V_NODE. If it has a valid requires edge, it can still be retrieved as
optional, enrichment, additional material, or a deeper example.
<!-- [AKU: C03.S03.SS03.A12] -->
<!-- type: example, diagram, enrichment -->
<!-- requires: [C03.S03.SS03.A11] -->
An extra complete ER diagram example.
<!-- [/AKU] -->If the learner does not need this AKU for mastery, leave it out of the V_NODE. If it is necessary for assessment or progression, include it.
An exercise does not need to depend on every example that appears before it. It should require the concepts needed to attempt the activity.
Good:
<!-- [AKU: C03.S03.SS03.A13] -->
<!-- type: exercise -->
<!-- requires: [C03.S03.SS03.A10] -->
Model this relationship scenario as an ER diagram.
<!-- [/AKU] -->This is better than forcing the exercise to depend on an optional extra example
if the exercise only needs the diagram notation rule introduced in A10.
For activity sections, a visible Markdown heading is often just a human reading marker. Put it before the AKU block.
Preferred:
#### Questions / Activities
<!-- [AKU: C03.S03.SS03.A13] -->
<!-- type: exercise -->
<!-- requires: [C03.S03.SS03.A10] -->
Model the following relationship scenario.
<!-- [/AKU] -->Avoid putting a heading on the same line as metadata:
<!-- requires: [C03.S03.SS03.A10] -->#### Questions / ActivitiesMetadata comments should always be on their own line.
Avoid inventing extra levels like SSS01 just to wrap an exercise title. The
retrieval model expects AKUs to be the leaf learning atoms.
Avoid:
<!-- [AKU: C03.S03.SS03.A13] -->
...
<!-- [AKU: C03.S03.SS03.SSS01.A01] -->
...
<!-- [/AKU] -->Use one AKU instead:
<!-- [AKU: C03.S03.SS03.A13] -->
<!-- type: exercise -->
<!-- requires: [C03.S03.SS03.A10] -->
##### Activity: Model a scenario
...
<!-- [/AKU] -->Use this:
<!-- type: example, explanation -->Not this:
<!-- type: [example, explanation] -->requires is a machine-readable list. type is a simple comma-separated label
field used for filtering, reporting, and human inspection.
IDs should be consistent, zero-padded, and complete.
Good:
C03.S02.A07
C03.S02.SS01.A06
V_C03_S02_SS01_COMPLETEAvoid:
03.S02.A07
C03.S02.A7
C0.S0.A0
V_C0_S01_SS01_COMPLETEAlso avoid writing AKU: inside a requires list:
<!-- requires: [C03.S03.SS03.A10] -->not:
<!-- requires: [AKU: C03.S03.SS03.A10] -->For textbook-derived curricula, page metadata is useful on AKUs because the retrieved chunk can cite its source page.
<!-- [AKU: C03.S03.SS03.A10] -->
<!-- type: concept, diagram -->
<!-- page_start: 47 -->
<!-- page_end: 47 -->
<!-- requires: [C03.S03.SS03.A09] -->
Optional and mandatory relationships in ER diagrams.
<!-- [/AKU] -->V_NODEs usually do not need page metadata because they are synthetic mastery gates rather than textbook content. Add it only if you want to trace where a teacher defined the gate.
Run the compiler audit whenever you edit original_tagged.md. Fix errors
before loading ChromaDB.
.\.venv\Scripts\python.exe knowledge_compiler.py g-epal-baseis --audit-onlyTreat cycles, missing references, malformed IDs, unclosed AKUs, nested AKUs, and placeholder IDs as blockers. Duplicate headings caused by PDF artifacts can often remain warnings if the AKU and V_NODE IDs are still unique.
If you want the Markdown to remain cleaner for human readers, the compiler also accepts tags inside HTML comments.
<!-- [AKU: C01.S01.SS01.A01] -->
Data is raw facts or symbols. Information is processed data with meaning.
<!-- [/AKU] -->And for validation gates:
<!-- [V_NODE: V_C01_COMPLETE] -->
requires: [C01.S01.SS01.A01, C01.S01.SS02.A01]
<!-- [/V_NODE] -->When starting from a PDF-derived .md file, use this workflow:
- Preserve the official chapter and section order.
- Convert major titles into
# CHAPTER n: ...headings. - Convert section titles into
## n.m ...headings. - Convert subsection titles into
### n.m.k ...headings when available. - Split long textbook paragraphs into small AKU blocks.
- Assign each AKU a stable dot-notation ID.
- Add
type:metadata when it helps evaluation or teaching. - Add
requires:metadata for prerequisite concepts. - Add a
V_NODEgate at the end of each chapter or unit. - Add section gates when one section is prerequisite to the next section.
- Compile and inspect
knowledge_chunks.jsonbefore loading ChromaDB.
# CHAPTER 1: Basic Concepts of Informatics > EPAL Computer Science foundation.
## 1.1 Data and Information
### 1.1.1 Data
[AKU: C01.S01.SS01.A01]
type: concept
Data is raw facts, symbols, numbers, words, or observations that can be stored
and processed by a computer.
[/AKU]
[AKU: C01.S01.SS01.A02]
type: concept
requires: [C01.S01.SS01.A01]
Information is data that has been processed, organized, or interpreted so that
it has meaning for a user.
[/AKU]
### 1.1.2 Processing
[AKU: C01.S01.SS02.A01]
type: process
requires: [C01.S01.SS01.A01, C01.S01.SS01.A02]
Processing is the transformation of data into useful information through
operations such as sorting, calculating, classifying, or comparing.
[/AKU]
### CHAPTER 1 COMPLETION GATE
[V_NODE: V_C01_COMPLETE]
requires: [C01.S01.SS01.A01, C01.S01.SS01.A02, C01.S01.SS02.A01]
[/V_NODE]| Mistake | Why it hurts |
|---|---|
| Reusing the same AKU ID twice | Later chunks overwrite or collide conceptually. |
| Making one AKU cover many pages | Retrieval becomes vague and less teachable. |
| Skipping headings | Chunks lose chapter/section lineage. |
Forgetting requires on gates |
The graph cannot represent mastery prerequisites. |
| Renumbering IDs after benchmark creation | Evaluation cases and gates stop matching. |
| Mixing future-chapter content into earlier AKUs | The orchestrator cannot prevent leakage inside a chunk. |
Before compiling a new curriculum file, check:
- Every chapter has a
# CHAPTER n:heading, a Greek# �εφάλαιο n:heading, or a plain numbered heading such as# 1. .... - Every section uses a numbered heading such as
## 1.2. - Every AKU has one unique dot-notation ID.
- Every AKU contains one focused learning idea.
- Every
requires:reference points to an existing AKU or gate. - Every chapter that should control progression has a
V_C##_COMPLETEgate. - Every section that should control progression has a
V_C##_S##_COMPLETEgate. - The Markdown preserves the official curriculum sequence.
- No AKU contains large unrelated blocks copied directly from the PDF.
Then compile:
.\.venv\Scripts\python.exe knowledge_compiler.py your-curriculum.md --chunks your_chunks.json --graph your_graph.json --visualize your_graph.pngInspect the output:
Get-Content your_chunks.json
Get-Content your_graph.jsonIf the lineage looks correct, load it into ChromaDB:
.\.venv\Scripts\python.exe load_chunks_to_chroma.py your_chunks.json --persist-dir chroma_db --collection your_curriculum --resetFor real textbook work, place each parsed curriculum under curriculum/<slug>/
and keep the tagged source named original_tagged.md.
Expected layout:
curriculum/
g-epal-baseis/
original.md
original_meta.json
original_tagged.md
Then compile by folder slug:
.\.venv\Scripts\python.exe knowledge_compiler.py g-epal-baseis --visualizeThis reads:
curriculum/g-epal-baseis/original_tagged.md
and writes:
curriculum/g-epal-baseis/knowledge_chunks.json
curriculum/g-epal-baseis/knowledge_graph.json
curriculum/g-epal-baseis/knowledge_graph_audit.json
curriculum/g-epal-baseis/knowledge_graph.png
Load the same curriculum into ChromaDB:
.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --resetFor Greek textbook retrieval, use the default multilingual E5 large embedding model:
.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-largeAlternative Greek-specific experimental preset:
.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model nlpaueb/bert-base-greek-uncased-v1Lighter E5 comparison preset:
.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-smallFor offline smoke tests, use local hash embeddings:
.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode hashThe default Chroma collection name becomes the slug:
g-epal-baseis
Query it through the orchestrator:
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETEIf you loaded with --embedding-mode hash, query with hash embeddings too:
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE --query-embedding-mode hashGenerated files stay next to the source curriculum:
| File | Meaning |
|---|---|
original_tagged.md |
Human-edited tagged textbook source. |
knowledge_chunks.json |
AKUs ready for ChromaDB ingestion. |
knowledge_graph.json |
Full hierarchy and prerequisite graph. |
knowledge_graph_audit.json |
Blocking errors, warnings, and repair hints for the teacher. |
knowledge_graph.png |
Visual graph snapshot for quick inspection. |
The slug also becomes the default Chroma collection name, so g-epal-baseis
loads into collection g-epal-baseis.
Follow this path after cloning the repository onto a new machine. The commands
below use PowerShell; on macOS/Linux, replace .\.venv\Scripts\python.exe with
.venv/bin/python.
git clone https://github.com/programmatismos/CurriculumRAG.git
cd CurriculumRAGThis project works well with uv because it creates a fast, isolated local
environment in .venv/.
python -m pip install uv
uv venv .venv
uv pip install -r requirements.txtQuick check:
.\.venv\Scripts\python.exe -c "import networkx, chromadb; print('Environment ready')"This turns gold-standard.md into graph and retrieval artifacts.
.\.venv\Scripts\python.exe knowledge_compiler.py gold-standard.md --chunks knowledge_chunks.json --graph knowledge_graph.json --visualize knowledge_graph.pngExpected output:
Compiled <nodes> nodes, <edges> edges, and <chunks> AKU chunks.
Generated artifacts:
| Artifact | What it is used for |
|---|---|
knowledge_chunks.json |
VectorDB-ready AKU chunks with lineage metadata. |
knowledge_graph.json |
Curriculum graph with hierarchy and dependency edges. |
knowledge_graph.png |
Visual snapshot of the graph. |
This creates a local persistent vector store in chroma_db/.
.\.venv\Scripts\python.exe load_chunks_to_chroma.py knowledge_chunks.json --persist-dir chroma_db --collection curriculum_rag --resetExpected output:
Loaded <n> chunks into collection 'curriculum_rag' at chroma_db.
If you want a fully offline smoke test that does not use Chroma's default embedding model, use deterministic hash embeddings:
.\.venv\Scripts\python.exe load_chunks_to_chroma.py knowledge_chunks.json --persist-dir chroma_db --collection curriculum_rag --reset --embedding-mode hashAllowed query: the learner is in Chapter 2 and has completed the Chapter 1 gate.
.\.venv\Scripts\python.exe retrieval_orchestrator.py "What is an if statement?" --current-chapter C02 --completed-gate V_C01_COMPLETE --n-results 1The orchestrator should retrieve C02.S01.A01 using a Chroma metadata filter
like:
{"chapter_id": "C02"}Blocked query: the learner is still in Chapter 1, so Chapter 2 should not leak.
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Explain Chapter 2 if statements" --current-chapter C01Expected behavior:
BLOCKED BY CURRICULUM CONSTRAINT
Use --json when connecting the orchestrator to evaluation scripts, notebooks,
or a future Streamlit/Gradio interface.
.\.venv\Scripts\python.exe retrieval_orchestrator.py "What is an if statement?" --current-chapter C02 --completed-gate V_C01_COMPLETE --jsonThe JSON payload includes learner state, retrieval scope, retrieved AKU IDs, lineage metadata, and the prompt payload that can be sent to an LLM.
For interactive testing, launch the local Streamlit workbench:
.\.venv\Scripts\streamlit.exe run dev_workbench.pyThe workbench provides four panels:
| Panel | What it helps you do |
|---|---|
| 1. Pipeline | audit tagged Markdown, inspect DAG errors/warnings with source-line fix snippets, compile graph/chunks, load ChromaDB, and view knowledge_graph.png |
| 2. Query Lab | compare Naive RAG, prompt-guardrail context, DAG-only, and Full CurriculumRAG for the same question |
| 3. Evaluation | rerun benchmark cases, update plots, export judge triples, and view the paper report |
| 4. Artifacts | inspect generated files, counts, collection names, and equivalent CLI commands |
The top of the UI includes a first-time orientation, a glossary for AKU/V_NODE terms, and a runtime summary so colleagues can understand what is selected before they run anything.
Use the same embedding mode in the workbench that you used when loading
ChromaDB. For Greek textbook evaluation, use sentence-transformers with
intfloat/multilingual-e5-large. The workbench also includes
intfloat/multilingual-e5-small for a lighter E5 comparison and
nlpaueb/bert-base-greek-uncased-v1 for a Greek-specific comparison run. For
offline smoke tests, load with --embedding-mode hash and set the workbench
embedding mode to hash.
For the normal textbook workflow, use the bundled PowerShell runner instead of typing each command by hand:
.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseisThis runs:
knowledge_compiler.py <curriculum> --visualizeload_chunks_to_chroma.py <curriculum> --reset --embedding-mode sentence-transformersbuild_evaluation_cases_from_csv.py <curriculum>evaluate_curriculum_rag.py ... --strictstreamlit run dev_workbench.py
If a step fails, the script stops immediately and does not run later steps. For example, if the graph audit has blocking errors, it will not load stale chunks into ChromaDB or run misleading evaluation numbers.
Useful variants:
.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseis -SkipEvaluation
.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseis -NoStreamlit
.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseis -AllowEvaluationFailures
.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseis -EmbeddingModel intfloat/multilingual-e5-large
.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseis -EmbeddingModel intfloat/multilingual-e5-small
.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseis -EmbeddingModel nlpaueb/bert-base-greek-uncased-v1
.\scripts\run_pipeline.ps1 -Curriculum g-epal-baseis -EmbeddingMode hashUse -AllowEvaluationFailures when you want Streamlit to open even if
--strict evaluation finds failing benchmark cases. The report and plots are
still written, and the workbench can then be used to inspect what failed.
When editing the curriculum or compiler, use this loop:
.\.venv\Scripts\python.exe knowledge_compiler.py gold-standard.md --chunks knowledge_chunks.json --graph knowledge_graph.json --visualize knowledge_graph.png
.\.venv\Scripts\python.exe load_chunks_to_chroma.py knowledge_chunks.json --persist-dir chroma_db --collection curriculum_rag --reset
.\.venv\Scripts\python.exe retrieval_orchestrator.py "your test question" --current-chapter C01Local runtime folders such as .venv/ and chroma_db/ are ignored by git, so
each developer can rebuild them safely on their own machine.
Compile the sample curriculum:
.\.venv\Scripts\python.exe knowledge_compiler.py gold-standard.md --chunks knowledge_chunks.json --graph knowledge_graph.json --visualize knowledge_graph.pngCompile a textbook folder:
.\.venv\Scripts\python.exe knowledge_compiler.py g-epal-baseis --visualizeLoad the sample into ChromaDB:
.\.venv\Scripts\python.exe load_chunks_to_chroma.py knowledge_chunks.json --persist-dir chroma_db --collection curriculum_rag --resetLoad a textbook folder into ChromaDB:
.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-largeLoad a fully offline hash-embedding collection:
.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode hashAsk with chapter scope:
.\.venv\Scripts\python.exe retrieval_orchestrator.py "What is an if statement?" --current-chapter C02 --completed-gate V_C01_COMPLETEAsk with section scope:
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETEAsk against a hash-embedding collection:
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE --query-embedding-mode hashAsk against a multilingual sentence-transformers collection:
.\.venv\Scripts\python.exe retrieval_orchestrator.py "Τι είναι α�χείο;" --curriculum g-epal-baseis --current-section C01.S02 --completed-gate V_C01_S01_COMPLETE --query-embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-largeRun deterministic evaluation:
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strictExport paper tables and judge triples:
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict --export-judge-datasetValidate that the paper is still bound to the latest generated evaluation macros/tables, even on machines without a LaTeX engine:
.\.venv\Scripts\python.exe scripts\validate_paper_bindings.pyRun evaluation without plots:
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict --no-plotsAfter tagging a real textbook, run the compiler audit before loading ChromaDB:
.\.venv\Scripts\python.exe knowledge_compiler.py g-epal-baseis --audit-onlyIf the audit is blocked, the compiler prints teacher-friendly repair hints and writes:
curriculum/g-epal-baseis/knowledge_graph_audit.json
Blocking errors must be fixed in original_tagged.md before proceeding.
The audit catches:
| Signal | Why it matters |
|---|---|
| Cycle detected | A learner would need a node that eventually requires itself. |
Missing requires: target |
Usually a typo or a prerequisite AKU/V_NODE that was never tagged. |
Malformed IDs such as C01.S01.A7 |
Breaks stable dot-notation and makes lineage unreliable. |
| Duplicate AKU/V_NODE IDs | Two different blocks would collapse into the same graph node. |
| Nested AKU/V_NODE tag | Usually a missing closing tag before the next block starts. |
| Unclosed AKU/V_NODE block | Usually causes one huge chunk that swallows later content. |
Unexpected IDs such as C0.S0.A0 |
Usually placeholder AKUs that should be renamed. |
IDs like C01.S0.A03 |
Usually section-number typos. |
IDs missing the C prefix, such as 01.S03.A08 |
Usually malformed prerequisites. |
| Huge AKU chunks | Usually a missing <!-- [/AKU] --> closing tag. |
| Virtual nodes in the graph audit | Usually references to missing or mistyped IDs. |
When the audit is clean, compile, visualize, and load:
.\.venv\Scripts\python.exe knowledge_compiler.py g-epal-baseis --visualize
.\.venv\Scripts\python.exe load_chunks_to_chroma.py g-epal-baseis --reset --embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-largeload_chunks_to_chroma.py refuses to load a curriculum slug when the companion
knowledge_graph.json has a blocked audit or no audit metadata. This prevents
old or dirty chunks from entering the vector database after the teacher has made
tagging changes.
The evaluation layer has three levels:
- Deterministic curriculum checks that run locally without an LLM.
- A prompt-guardrail context baseline that shows what a prompt-only approach still exposes to the LLM before generation.
- Optional RAGAS or LLM-judge metrics that use an evaluator LLM for answer/context quality.
This makes the project easy to test in a development loop while still supporting paper-friendly RAG evaluation later.
Run this after compiling the curriculum and loading ChromaDB:
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strictThis command refreshes the JSON report, manuscript-ready Markdown report, and
comparison plots every time it runs. Use --strict for CI-style checks that
should exit nonzero when any benchmark assertion fails. For paper-review runs
where you want the artifacts to refresh even while retrieval tuning is still in
progress, omit --strict.
Generated outputs:
| Output | Purpose |
|---|---|
evaluation/results/latest_evaluation.json |
Full machine-readable evaluation payload. |
evaluation/results/paper_evaluation_report.md |
Paper-ready tables with counts, 95% Wilson confidence intervals, corpus statistics, and warnings about judge protocol. |
evaluation/results/paper_numbers.tex |
Auto-generated LaTeX macros for corpus counts, benchmark counts, leakage rates, confidence intervals, and optional judge status. |
evaluation/results/paper_tables.tex |
Auto-generated LaTeX tables with numerator/denominator pairs for ablations, failure modes, and deterministic checks. |
evaluation/results/plots/*.png |
Updated visual summaries for the current run. |
The deterministic suite compares the ablation variants needed for leakage experiments:
| Stage | What it measures |
|---|---|
| Naive RAG baseline | Unconstrained vector retrieval before any curriculum constraint. |
| DAG-only, no V_NODE gates | Metadata/chapter/section filtering while ignoring mastery gates. |
| Full CurriculumRAG | Metadata-first retrieval after chapter, section, and V_NODE constraints. |
| Prompt-guardrail context | The same naive context pool placed behind an instruction such as "do not answer outside the student's current chapter." This measures exposed context, not LLM obedience. |
To avoid circular evaluation, leakage is measured against independent benchmark
annotations such as allowed_chapters or allowed_sections, not against the
same DAG rule used by the runtime filter.
If a curriculum folder contains a teacher-authored CSV with columns for question, ideal answer, chapter/section lineage, answer AKUs, and page provenance, convert it into the JSON format used by the evaluation harness:
| CSV column | Meaning in the benchmark |
|---|---|
Question or question |
Student query used by the benchmark case. |
Answers_ideal or answer_ideal |
Teacher reference answer, used as reference for RAGAS and judge datasets. |
Chapters_Sections or chapters_sections |
Learner/query location, such as C03.S02 or C03.S02.SS01. |
AKUs or akus_of_answer |
Comma-separated teacher gold AKUs that should answer the question. |
Page_Number_Question or page_no_question |
Textbook page where the question appears. |
Page_Number_Answer or page_no_answer |
Textbook page where the answer evidence appears. |
.\.venv\Scripts\python.exe build_evaluation_cases_from_csv.py g-epal-baseisFor curriculum/g-epal-baseis/g-epal-baseis-dataset.csv, this writes:
curriculum/g-epal-baseis/evaluation_scenarios.json
curriculum/g-epal-baseis/evaluation_cases.json
curriculum/g-epal-baseis/evaluation_cases.skipped.json
The converter writes two benchmark views from the same teacher CSV:
| Output | Purpose |
|---|---|
evaluation_scenarios.json |
Grouped, peer-review friendly schema. Each teacher question contains learner-state permutations for beginner, intermediate, and advanced personas. |
evaluation_cases.json |
Expanded flat schema consumed by evaluate_curriculum_rag.py. Each scenario becomes one runnable benchmark case. |
evaluation_cases.skipped.json |
Rows that cannot yet be benchmarked because the referenced section or answer AKU is not present in knowledge_chunks.json. |
This gives you the scientific structure needed for ablation studies: the same
question Q is tested under multiple simulated K_student states instead of
being treated as one generic query.
The grouped schema has this shape:
{
"question_id": "Q015",
"question": "Ποιες είναι οι απαιτήσεις ACID;",
"target_section": "C02.S02",
"ground_truth_akus": ["C02.S02.A03", "C02.S02.A04"],
"evaluation_scenarios": [
{
"persona": "beginner",
"simulated_k_student": [],
"expected_status": "blocked",
"expected_behavior": "Socratic Redirect",
"test_goal": "Future-Chapter Leakage Prevention"
},
{
"persona": "intermediate",
"simulated_k_student": ["V_C02_S01_COMPLETE"],
"expected_status": "ready",
"expected_behavior": "Allowed / Full Answer",
"test_goal": "Retrieval Hit Rate Validation"
},
{
"persona": "advanced",
"simulated_k_student": ["all known V_NODE gates from the graph"],
"expected_status": "ready",
"expected_behavior": "Advanced Context (No C01 regression)",
"test_goal": "Contextual Regression Suppression"
}
]
}The converter infers:
current_chapterandcurrent_sectionfromChapters_Sections- previous chapter and section
completed_gatesneeded to make that learner state reachable evaluation_scenariosfor the three standard personas:beginner,intermediate, andadvancedexpected_idsfrom teacher-listedAKUsthat exist inknowledge_chunks.jsonteacher_expected_idsfrom the complete AKU list in the CSVallowed_chaptersandallowed_sectionsfrom the teacher answer AKUsreferenceandanswer_idealfrom the ideal-answer columnsource_pages.questionandsource_pages.answerfrom the page columns
Rows are skipped when the CSV references a section or AKU that is not yet present
in knowledge_chunks.json. This is intentional: the skipped report becomes a
tagging progress checklist for the teacher.
During active tagging, use --allow-partial to include rows where at least one
teacher-listed answer AKU exists. For paper numbers, leave it off so every
included case has a fully tagged answer set. Use --include-unmatched only when
you want to inspect the dataset JSON before the corresponding AKUs exist in the
compiled curriculum.
Run the generated cases with:
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --cases curriculum/g-epal-baseis/evaluation_cases.json --collection g-epal-baseis --graph curriculum/g-epal-baseis/knowledge_graph.json --query-embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-large --strictThe evaluator can also read the grouped file directly:
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --cases curriculum/g-epal-baseis/evaluation_scenarios.json --collection g-epal-baseis --graph curriculum/g-epal-baseis/knowledge_graph.json --query-embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-large --strictThe Dev Workbench exposes the same flow in 3. Evaluation: choose the CSV, build the grouped scenarios and expanded cases JSON, inspect skipped rows, then run the evaluation suite.
In other words:
- pre-filter leakage measures how much inappropriate context semantic search would retrieve before any curriculum constraint is applied
- DAG-only leakage measures how much inappropriate context survives when the system uses metadata/DAG filtering but ignores V_NODE mastery gates
- prompt-guardrail context leakage measures how much inappropriate context is still visible to a prompt-only baseline before generation
- full CurriculumRAG leakage measures how much inappropriate context survives after the orchestrator applies metadata and gate constraints
- DAG filter violation rate is reported separately as an implementation sanity check, not as the main pedagogical leakage construct
The deterministic prompt-guardrail metric measures context exposure: it shows what forbidden chunks a prompt-only RAG system would place in the LLM context window. To test whether a prompt instruction actually prevents answer leakage, run the optional generation baseline.
This baseline generates two answers per benchmark case:
| Stage | Context shown to generator | What is judged |
|---|---|---|
| Prompt-only generation | Naive/unconstrained retrieved context plus a curriculum guardrail prompt | Whether the generated answer still leaks locked or inappropriate content |
| Full CurriculumRAG generation | CurriculumRAG-filtered context, or a blocked-state redirect | Whether the generated answer remains curriculum-safe |
Install optional evaluation dependencies and set your LLM key:
uv pip install -r requirements-eval.txt
$env:OPENAI_API_KEY="your-api-key"Run a cheap pilot first:
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --cases curriculum/g-epal-baseis/evaluation_cases.json --collection g-epal-baseis --graph curriculum/g-epal-baseis/knowledge_graph.json --query-embedding-mode sentence-transformers --embedding-model intfloat/multilingual-e5-large --run-generation-baseline --generation-model gpt-4o --generation-judge-model gpt-4o --generation-limit 10When the prompt and judge rubric look stable, remove --generation-limit for a
full run. The results appear in:
evaluation/results/latest_evaluation.json
evaluation/results/paper_evaluation_report.md
Use this table for the paper only when you report the generator model, judge model, case count, and the exact generation/judge prompts from the JSON report.
Benchmark cases should include personas that expose the failure modes ordinary RAG systems struggle with.
| Persona | Failure mode | What to test |
|---|---|---|
beginner |
future_chapter_leakage |
A learner asks with familiar early terms, but the semantically attractive answer lives in a locked future chapter. Full CurriculumRAG should block and respond with a Socratic redirect. |
intermediate |
in_scope_retrieval |
The learner has the prerequisite gates for the target section. This is the positive-control run for retrieval hit rate and RAGAS faithfulness. |
advanced |
contextual_regression |
A learner asks a specialized question, but naive retrieval pulls simple already-mastered definitions into the context window. Full CurriculumRAG should keep the context in the target answer scope. |
beginner |
section_gate_leakage |
A learner asks for a later section inside the same chapter before the section gate is complete. |
beginner |
state_consistency |
The active chapter/section and completed V_NODE gates disagree, simulating bad imported state or UI drift. |
| any | in_scope_retrieval |
Positive-control case that should remain answerable. |
Example future-leakage case:
{
"id": "beginner_c05_locked_question",
"persona": "beginner_student",
"failure_mode": "future_chapter_leakage",
"scenario": "Beginner uses Chapter 1 vocabulary, but the answer belongs to locked Chapter 5 content.",
"question": "How can stored data decide which path to follow later?",
"current_chapter": "C01",
"current_section": "C01.S01",
"completed_gates": [],
"expected_status": "blocked",
"expected_ids": [],
"expected_chapters": [],
"allowed_chapters": [],
"allowed_sections": [],
"ground_truth_basis": "syllabus_position",
"expected_missing_gates": ["V_C01_COMPLETE", "V_C02_COMPLETE", "V_C03_COMPLETE", "V_C04_COMPLETE"],
"expected_response_strategy": "socratic_hint",
"reference": "The answer belongs to a locked future chapter, so the tutor should avoid exposing it and ask a boundary-respecting hint."
}Example contextual-regression case:
{
"id": "advanced_er_specialized_query",
"persona": "advanced_student",
"failure_mode": "contextual_regression",
"scenario": "Advanced learner asks about ER modeling; naive retrieval may waste context on basic data definitions.",
"question": "How do entity relationships constrain the logical database design?",
"current_chapter": "C05",
"current_section": "C05.S02",
"completed_gates": ["V_C01_COMPLETE", "V_C02_COMPLETE", "V_C03_COMPLETE", "V_C04_COMPLETE"],
"expected_status": "ready",
"expected_ids": ["C05.S02.A03"],
"expected_chapters": ["C05"],
"allowed_chapters": ["C05"],
"allowed_sections": ["C05.S02"],
"ground_truth_basis": "syllabus_position",
"reference": "The answer should use the active advanced chapter, not regress to introductory definitions."
}In the tiny sample benchmark, Chapter 2 stands in for a locked future chapter.
In the full g-epal-baseis evaluation, create the same pattern with the real
future chapter, such as Chapter 5.
It measures:
| Metric | Why it matters |
|---|---|
| Retrieval hit rate | Expected AKUs are retrieved for in-scope questions. |
| Gate block accuracy | Future chapters are blocked when V_NODE gates are missing. |
| Pre-filter pedagogical leakage rate | Inappropriate context in the naive semantic candidate pool. |
| DAG-only pedagogical leakage rate | Inappropriate context after metadata/DAG filtering with V_NODEs disabled. |
| Prompt-guardrail context leakage rate | Inappropriate context that prompt instructions would have to suppress after retrieval. |
| Post-filter pedagogical leakage rate | Inappropriate context that survives curriculum filtering. |
| DAG-only leakage reduction | How much metadata/DAG filtering reduces leakage compared with naive retrieval. |
| Full CurriculumRAG leakage reduction | How much the full gate-aware system reduces leakage compared with naive retrieval. |
| DAG filter violation rate | Implementation defect signal for metadata/filter edge cases. |
| Lineage completeness | Retrieved atoms carry chapter/section/subsection context. |
The report includes denominators for the leakage rates:
Judged chunks: pre=<n> prompt=<n> dag_only=<n> post=<n>
Invalid chunks: pre=<n> prompt=<n> dag_only=<n> post=<n>
For paper tables, report those counts together with the percentages and the generated 95% confidence intervals. Do not reuse percentages from an older judging protocol after changing the judge.
Formal leakage definition:
J(c, q, K_student) -> {0, 1}
where J is an evaluation judge that returns 1 when chunk c is
pedagogically inappropriate for query q and learner state K_student.
In deterministic mode, J is implemented by benchmark annotations such as
allowed_chapters and allowed_sections. In an LLM-judge experiment, J
must be replaced by the frozen judge prompt and syllabus context used for that
run.
For a retrieval stage s with candidate set C_s(q, K_student):
DLS_s = (1 / |C_s|) * sum_{c in C_s} J(c, q, K_student)
Report at least:
DLS_naive for naive semantic candidates
DLS_dag for DAG-only retrieval without V_NODE gates
DLS_full for full CurriculumRAG retrieval
DLS_prompt for prompt-guardrail context exposure, if included
If J changes from deterministic annotations to an LLM judge, all stages must
be re-judged under that same J.
The latest JSON report is written to:
evaluation/results/latest_evaluation.json
That folder is ignored by git so repeated local experiments stay out of commits.
To avoid the DLS circularity problem, benchmark cases must define the leakage
criterion outside the runtime DAG filter. Each case should include
allowed_chapters and a ground_truth_basis.
Example:
{
"id": "epal_c03_databases_blocked",
"question": "How does a database table store records?",
"current_chapter": "C02",
"completed_gates": ["V_C01_COMPLETE"],
"expected_status": "blocked",
"expected_ids": [],
"expected_chapters": [],
"allowed_chapters": [],
"ground_truth_basis": "syllabus_position",
"expected_missing_gates": ["V_C02_COMPLETE"],
"reference": "This topic belongs to a later chapter and should not be answered yet."
}Recommended values for ground_truth_basis:
| Basis | Meaning |
|---|---|
syllabus_position |
The official curriculum order says which chapters are allowed. |
teacher_annotation |
A human teacher judged the chunk appropriate or inappropriate. |
llm_judge_syllabus_rubric |
A separate LLM judged appropriateness from the natural-language syllabus, not the compiled DAG. Use only when the full LLM-judge protocol was actually run. |
This separation matters:
- the DAG filter is the system policy
allowed_chaptersis the evaluation criterion- the report compares naive, DAG-only, prompt-context, and full-system leakage against that external criterion
dag_filter_violation_rateis only an implementation sanity check
When using llm_judge_syllabus_rubric, also report:
- judge model and version
- judge prompt in an appendix
- natural-language syllabus context shown to the judge
- number of judged
(query, chunk, learner_state)triples - how many queries, personas, and curriculum states produced those triples
- agreement or stability checks, such as repeated LLM judging or human expert validation on a sample
- whether the same judge re-scored every stage: pre-filter, DAG-only, prompt-guardrail, and post-filter
Until those fields exist, describe the current evaluation as deterministic benchmark evaluation, not as LLM-judged leakage.
The evaluator can export every retrieved chunk that needs independent pedagogical judging:
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict --export-judge-datasetThis writes:
evaluation/results/judge_dataset.jsonl
Each JSONL row contains:
| Field | Meaning |
|---|---|
judgment_key |
Stable key for loading the judgment back into the report. |
stage |
pre_filter, prompt_guardrail, dag_only, or post_filter. |
question |
Student query used in the benchmark case. |
learner_state |
Current chapter/section, completed gates, and allowed scope. |
atom_id |
Retrieved AKU ID. |
chunk_text |
Text shown to the independent judge. |
deterministic_pedagogically_invalid |
Local benchmark label for comparison/debugging only. |
A completed judge result can be JSON or JSONL and should include either
judgment_key or the triple case_id, stage, atom_id, plus one of:
{
"judgment_key": "c02_blocked_without_gate::pre_filter::C02.S01.A01",
"pedagogically_invalid": true,
"confidence": 0.94,
"rationale": "The chunk introduces Chapter 2 before the learner has completed Chapter 1."
}Load completed judgments back into the paper report:
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict --judge-results evaluation/results/judge_results.jsonlFor a pilot OpenAI-backed judge run, provide the natural-language syllabus or
tagged textbook Markdown. Use --judge-limit for a cheap smoke test first:
$env:OPENAI_API_KEY="your-api-key"
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict --run-llm-judge --judge-syllabus curriculum/g-epal-baseis/original_tagged.md --judge-model gpt-4o --judge-limit 20The Dev Workbench exposes the same protocol under
3. Benchmark -> Optional independent LLM-as-a-Judge. Start with a small pilot
limit there too; the UI will mark partial judge runs as pilots so they are not
mistaken for paper-ready headline results.
Only use the LLM-judge leakage table in a paper when the external judge summary says all exported records were matched. Partial pilot runs are useful for debugging the rubric, not for headline numbers.
Install the extra evaluation dependencies:
uv pip install -r requirements-eval.txtSet your evaluator LLM key, then run:
$env:OPENAI_API_KEY="your-api-key"
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --run-ragas --ragas-model gpt-4oThe RAGAS path builds rows with user_input, retrieved_contexts, response,
and reference, then evaluates them with context recall, faithfulness, and
factual correctness metrics.
For automated testing or CI, use deterministic mode first:
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict --jsonIf a CI job only needs pass/fail output and should skip image generation:
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strict --no-plotsPlots are generated by default when the evaluation suite runs:
.\.venv\Scripts\python.exe evaluate_curriculum_rag.py --strictThis writes:
| Plot | What it shows |
|---|---|
evaluation_summary_metrics.png |
Pass rate, retrieval hit rate, gate accuracy, pre/prompt/DAG-only/full leakage, DAG violations, and lineage completeness. |
leakage_ablation.png |
Naive RAG baseline vs DAG-only vs Full CurriculumRAG. |
leakage_comparison.png |
Naive RAG, prompt-guardrail context, DAG-only, and Full CurriculumRAG. |
retrieval_counts_by_case.png |
How many atoms each retrieval variant returned per benchmark case. |
deterministic_check_matrix.png |
Pass/fail matrix for each deterministic benchmark check. |
By default, plots are written to:
evaluation/results/plots/
Each AKU chunk includes:
id: The AKU identifier, such asC01.S01.SS01.A01.text: The source text inside the AKU block.lineage: Parent chapter, section, and subsection metadata.chapter_root: The top-level chapter context.source_lines: Start and end lines in the source file.
Edges are typed:
hierarchy: Parent-to-child curriculum structure.dependency: Prerequisite-to-gate or prerequisite-to-node logic.
The graph JSON stores prerequisites in two places for different consumers:
- each explicit AKU or V_NODE node keeps its parsed
requireslist - the top-level
edgesarray contains the deriveddependencyedges
The requires field is the source annotation. The dependency edges are
derived from it during compilation so graph libraries can traverse the DAG
directly. If they disagree, treat it as a compiler defect and rerun the compiler
from the tagged Markdown.
The graph JSON also embeds an audit object:
{
"status": "clean",
"error_count": 0,
"warning_count": 1,
"is_acyclic": true,
"findings": []
}status: blocked means the teacher should fix original_tagged.md before
loading chunks or running retrieval. The same audit is written separately as
knowledge_graph_audit.json for easier inspection.
Before using a compiled graph in paper experiments, check:
- the graph is acyclic
- every
requiresID exists as an explicit node, not only as an accidental virtual placeholder - every V_NODE has the expected parent hierarchy edge
- section and chapter completion gates list the full completion contract
Malformed prerequisite IDs are intentionally visible in the graph as virtual
nodes. That makes mistakes like C0.S0.A0, C01.S0.A03, or 01.S03.A08
auditable instead of silently ignored.
CurriculumRAG gives a curriculum two useful forms at once:
- A graph for structural reasoning, prerequisite tracing, and validation gates.
- Clean chunks for semantic retrieval, embeddings, and RAG pipelines.
The result is a knowledge base that remembers both the words and the shape of the curriculum.

