A logistic-regression calibrator, trained on real labeled outcomes, routes each prompt to the cheapest model tier that's still likely to get it right.
Model tiers
93.9% cheaper
No measurable quality loss: 96.0% vs 92.0% accuracy, CIs overlap (n=50, held-out set).
At 1M req/mo: $21,163/yr saved (illustrative, extrapolated)1
¹ Extrapolated from the measured $0.0018774/prompt average rate (results/eval_report.md), not a claim about any specific deployment's real traffic.
Send a prompt to see the calibrator route it live.
Live total from your requests above, vs. every prompt going straight to the capable tier (Claude Sonnet 5) at the real measured average rate ($0.0018774/prompt); not a live re-query.
The calibrator (app/router/calibrator.py) is a logistic regression over four features, not an LLM call. The cascade-on-confidence architecture follows FrugalGPT[1]; the self-consistency feature follows Wang et al.[2]:
Chosen over gradient-boosted trees (scripts/train_calibrator.py) for this dataset size (about 100 rows): fewer parameters to overfit, and it stays monotonic in each feature, matching the actual assumption that higher uncertainty means higher predicted error, instead of letting a tree carve out noise-driven splits on a small sample.
The router accepts the cheap tier's answer only if its predicted error rate stays under a configured budget ε (app/router/decision.py), treating that threshold as a hyperparameter tuned on held-out data rather than a hand-picked value, following the C3PO-style deferral rules of Kang et al.[3] and the decision-theoretic cascade literature[4]:
Calibration quality is checked with Expected Calibration Error, following Guo et al.[5] (scripts/stats_utils.py): does the model's confidence match its actual accuracy?
def decide_tier(p_correct_cheap, error_budget, p_correct_mid=None):
predicted_error_cheap = 1.0 - p_correct_cheap
if predicted_error_cheap <= error_budget + 1e-9:
return RoutingDecision(chosen_tier="cheap", ...)
# escalate one rung, re-check against the same budget
...Held-out evaluation, 50 prompts never seen during calibrator training. Baseline (always_capable, every prompt sent to Claude Sonnet 5 directly): 92.0% accuracy, $0.09387 total cost.
96.0% vs 92.0% accuracy: the confidence intervals overlap (86.5-98.9% vs 83.8-97.9%), so this is parity, not a claimed accuracy gain. Verified across a full error-budget sweep (ε=0.05 to 0.50), not one cherry-picked value; ε=0.10 through 0.20 all beat the baseline on cost with no accuracy loss. Full sweep in results/budget_sweep.md, ablation and Pareto frontier in results/eval_report.md. n=50, a real, non-trivial CI width; treat this as a strong validated signal, not a fully settled number.
Correctness labels come from GSM8K[6] and MMLU[7], established peer-reviewed benchmarks pulled live from their canonical HuggingFace releases, not invented for this project. But both datasets have a documented, non-zero label-error rate in the research literature, and our own correctness check (a substring match against ground truth, not a real answer parser) adds further noise in both directions. The calibrator's training labels carry some irreducible baseline error; this isn't unique to this project, every benchmark-trained system inherits it, but it's stated here rather than left implicit.
The cheap tier also changed mid-build: originally Groq's openai/gpt-oss-120b, which didn't support logprobs and hit an 8000-tokens/minute free-tier rate limit during bulk data collection. Switched to OpenAI's gpt-5.4-nano, which supports real logprobs and has far higher rate limits at this account's tier: a real pivot during the build, not the original plan.