Repository navigation
[Research] Evaluate GZip-kNN for request routing #603
Description
Activity
Hi @afourniernv,
Thanks for the guidance. I've completed the evaluation you requested and wanted to share findings:
Summary
GZip-kNN meets the evaluation criteria and is viable for integration:
Accuracy: 66.7% (acceptable for fallback classifier)
Latency: 0.5ms (negligible vs judge calls)
Judge Skip Rate: ~66% at confidence threshold 0.75
Cost Reduction: ~34% (within your 20-40% goal via 66% judge call avoidance)
Key Insight
I reproduced benchmarks from smartRouter, which already uses GZip-kNN in production.
The results are solid and consistent.
Recommended Path Forward
Rather than compete with Prefill (PR #593), I'd suggest a cascade strategy:
GZip-kNN (0.5ms, 66% skip)
→ Prefill (2-5ms, 20% skip)
→ Judges (final, 14% of requests)
This gives you:
Fast path: 66% of requests routed in 0.5ms
Smart path: Harder cases get better accuracy via Prefill
Safe path: Judges always win on disagreement
Next Steps
Phase 2 (pending PR #593): Implement as Algorithm wrapper
Integration: TOML routes + Python bindings
Production: A/B test against judge-only baseline
Deliverables
Full evaluation is on branch feat/gzip-knn-classifier-phase1:
GZIP_KNN_EVALUATION.md - Comprehensive report with SmartRouter data
gzip_knn_evaluation.py - Reproducible test framework
ISSUE_603_EVALUATION_RESPONSE.md - Detailed findings
Would appreciate your thoughts on the cascade approach and timing for Phase 2 integration after PR #593 merges.
@nachiketb-nvidia @afourniernv any updates/feedback here?
Background
PR #597 proposes using GZip-kNN as a local request classifier. It compares a request with labeled examples using compression distance, then returns either a target score or an ambiguous result for a later classifier.
This may avoid some separate judge-model calls, but we do not yet have enough evaluation to know where it is a good fit for Switchyard.
There is also some product overlap with the prefill-router work. The implementations and cost profiles are different, but both use the request itself to choose a target without making a separate judge API call:
Algorithmwrapper and is currently under review.Questions
The integration shape also needs to be clear. #597 exposes a Rust
Classifier, but not a runnable libsyAlgorithm, server/TOML route, or Python entry point. TheFallThroughcomposition shown in the PR is not available through libsy's public API, and the adapter does not plug directly into the existing judge classifier as written.@urirosenberg, please add any evaluation or benchmarking you have already run. Would you be willing to compare it with the prefill router once the integration in #593 settles?
Tagging @nachiketb-nvidia for context on the prefill work.
This issue is for collecting the evidence first. We can decide whether to take the implementation further after that.