llm-as-judge

The Cheap Model Was Fine. Proving It Was the Hard Part.

I built a gateway that sends each prompt to the cheapest Gemini model that can answer it, then benchmarked it on 300 prompts with a Pro-tier judge. The cheap model held up. My test set, my verifier, and my first table of numbers did not. Six lessons for anyone paying an AI bill.