On October 1, 2026, OpenRouter published a three-step framework on its official blog for picking models for agent tasks. Its starting point is a counterintuitive claim: choosing by leaderboard rank makes you pay frontier prices for rankings you don't need. The right question isn't "which model scores highest," but "which model is the cheapest one that is just good enough."
Why leaderboard rankings mislead agent model selection
A leaderboard averages a model's results across tasks that have nothing to do with yours. Your own agent tasks are usually narrow: triage a ticket, extract one field, escalate to a human when unsure. Whether a cheap model can match a frontier model's accuracy on a narrow task is a measurement question, and no leaderboard makes that measurement for you.
You also can't bill an agent like a single chat turn. One conversation is billed once; every tool call, every intermediate step, and every retry of an agent costs money, so a three-step loop pays the token price at least three times over. Picking by leaderboard rank means paying the frontier price three times for a task a cheaper model could have handled just as well.
How the three steps work in practice
Step one: set a quality bar for the task. Tasks where a wrong answer is a liability — compliance, healthcare, legal review — get a high bar; high-volume support, where the aggregate outcome matters, might be fine with 90% auto-resolved and 10% cleanly escalated. For latency-sensitive work like fraud detection or live chat, speed is the third hard constraint, and a model that is cheap and accurate but too slow is disqualified first.
Step two: measure on your own examples. Pick one representative per tier — cheap, mid-tier, frontier — run 20 to 50 real examples from your business through each, score them with one consistent rubric, and divide cost by score to get a "cost per quality point." Don't estimate cost from the price list; read the usage.cost field in each response instead — that's what the platform actually charged.
Step three: pick the cheapest model that clears the bar with margin. The margin matters because scores drift: model weight updates and shifts in your own traffic move the numbers around. The practice is to run multiple rounds, observe how much the score swings between runs, and require the winner to clear the bar by more than that swing.
OpenRouter gives a worked example: with the bar at 85%, GPT-5.6 Luna scores 82 and is disqualified outright — cheap doesn't matter if it's below the bar; Gemini 3.7 Flash clears at 89 and becomes the pick at $1.09 per 1,000 requests; Claude Opus 5 scores 97 but costs $7.25 per 1,000, a luxury the task didn't ask for. Prices and scores are illustrative; use your own measured numbers in practice.
Where this method's boundaries lie
The framework turns model selection from guesswork into measurement, but it rests on three premises. First, you set the quality bar yourself — if that subjective judgment is wrong, everything downstream is wrong. Second, a measurement is only true on the day it was taken: a new model version or a price change can expire the conclusion, so rerun it. Third, it governs the cost-versus-quality tradeoff, not whether you should use an agent at all — a badly decomposed task wastes money no matter how cheap the model is.
