Organized, reproducible workspace for composing sets from authentic domain docs and cheatsheets. This repo emphasizes privacy-preserving local processing, deterministic outputs, and clear provenance.
docs/— Human-facing docs and indexesdocs/index.html— Root docs index (entry point)docs/compositions/— Orchestrated sets, live sets, performance guidesdocs/cheatsheets/— Domain cheatsheets generated from processed functionsdocs/per_domain_faiss_evaluation.md— Dense evaluation notes
data/raw/— Scraped/exported authentic inputs (HTML/TXT/JSON)data/processed/— Pipeline outputschunks/— Content chunksfunctions/— Enhanced function/operator catalogsindexes/— Retrieval artifacts, harnesses, evaluation reports
scripts/— Deterministic pipeline and tooling- See
scripts/README.mdfor usage and common commands
- See
- Create a virtualenv and install deps as needed.
- Run processing and validation:
make processthenmake validate
- Build indexes and evaluate retrieval:
make clean_indexthenmake evaluate_keyword
- Generate compositions:
make suggest_planthenmake orchestrate_set
- Preview docs:
make serveand openhttp://localhost:8000/docs/index.html
- No synthetic data; all inputs are authentic public docs/transcripts.
- Reproducible scripts and evaluation; deterministic artifacts saved under
data/processed. - Secure defaults and offline-friendly harnesses (no external CDNs).
- If markdown links download in your browser, open them in an editor.
- See
docs/compositions/index.htmlfor curated Strudel/Tidal/Hydra sets.