Open source evaluation framework: accuracy + cost + latency + hallucination

[vignesh2027]

Hey OpenAI Evals community!

OpenAI Evals is fantastic for task-specific evaluation. For teams who also need production metrics alongside task accuracy, I built a complementary open source framework.

What it adds beyond task accuracy:

• Cost per 1K tokens (from real token counts, not estimates)
• Latency p50/p95/p99 (async parallel, realistic concurrency)
• Hallucination Rate (linguistic signal analysis, no judge needed)
• Reasoning Quality (CoT depth score 1-10)
• Accuracy (MMLU + TruthfulQA + custom benchmark support)

Key insight from running this:
GPT-4o-mini vs Gemini Flash: 78.4% vs 76.8% accuracy. But $0.0003 vs $0.0001 per 1K. For production at scale, that 2% accuracy gap rarely justifies the 3x cost difference.

Live demo (no API key needed): https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework

71 tests, 82% coverage, full CI/CD. Open source, free forever.
Task evaluation (OpenAI Evals) + production metrics (this) = complete evaluation stack.