Pinned Loading

1. data-collection-and-sourcing data-collection-and-sourcing Public

   Decide the data for a 40B model that is as good as Gemma 4 31B, strong in code and agentic work, strong in Indian languages, and India-first. Measure each number that a decision depends on. Do not …

   Python

2. reversibletransformer reversibletransformer Public

   Train a ~20M-param GPT for 50M tokens three ways (baseline / reversible same batch / reversible max batch) and measure loss, tokens/s and peak memory. See REPORT.md for results and findings. See ex…

   Jupyter Notebook 1

3. loss_func_op_heads loss_func_op_heads Public

   This repository contains the complete implementation, verification harness, and technical report: From Hidden State to Loss — Logits, Softmax, Cross-Entropy, and Output Heads. The pipeline under st…

   Jupyter Notebook

4. mixture-of-experts-impl mixture-of-experts-impl Public

   train a small dense GPT, convert every FFN into an 8-expert top-2 MoE

   Python

5. optimizer-schedule-diagnostics optimizer-schedule-diagnostics Public

   Adam by hand, bias correction, update ratio, cosine vs WSD, LR vs width

   Python

6. token-vocab-design token-vocab-design Public

   A 10,000-vocabulary byte-level BPE tokenizer optimizing cross-lingual parity across English, Hindi, Telugu, and Marathi, built with Indic-aware pre-tokenization to prevent script fragmentation.

   Python