Episode 6382 custom

Choosing AI Models on Cost and Performance

Model prices vary thirty-eight-fold per index point, meaning ratio formulas love cheap options regardless of actual quality.

Oct 31 18:54 Transcript

With

0:00

Episode Details

Episode ID: MWP-6584
Published: Oct 31
Duration: 18:54
Audio: Direct link
Pipeline: V5.26
TTS Engine: chatterbox-regular
Script Writing Agent: Claude Sonnet 5.5
Topics: ai-evaluation ai-inference prompt-engineering

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

This picks up from an earlier question. I'm looking at OpenRouter batch inference prices and have split the models into price tiers. Now imagine I also have benchmark data for my task: one score from 0 to 1 for each model. With just price and performance for every model, how do you turn the everyday questions people ask into actual calculations?
"Best performance" is easy, it's simply the top score. The harder ones are "best value for money", meaning the most performance per unit of cost but only counting models above a minimum acceptable score, and "best balance of performance and cost". What are the classic methods for answering these from a small table, using an algorithm or nothing more than a basic statistics calculator? I mean things like score-per-dollar ratios, Pareto frontiers, knee or elbow points, and weighted or normalised scores, and why the different methods can pick different winners from the same data.

Balancing artificial intelligence performance against operating cost seems like a basic spreadsheet task, but standard evaluation methods frequently crown entirely different winners from identical data. Simple cost-per-point ratios heavily favor bargain options regardless of quality, requiring human judgment to set a strict performance floor for viable results. Mapping options along a Pareto frontier effectively eliminates dominated models, yet still serves merely as triage rather than declaring an outright champion. Meanwhile, finding diminishing returns via curve elbows proves fragile, as mathematical knee points often shift with graphic axis scaling. Decades of multi-criteria decision analysis show that choosing how to evaluate alternatives remains an inherently subjective decision problem in itself.

Sources

What the research for this episode read before the script was written. Primary sources first.

Context

Follows the earlier question about splitting OpenRouter batch prices into tiers. Episode #6134 "Cheap Tokens, Expensive Scripts" covered cost per task, and earlier episodes covered benchmark noise and finding benchmarks. Don't repeat those. Take the benchmark score as given and focus on the selection maths.

Mentions

Claude Opus 5.5 Higher-priced model with effort settings
DeepSeek V4.1 Flash New DeepSeek model with Engram memory architecture
Gemini 4 Argon Model at eight cents per index point
GLM-5.3 Model at five cents per index point
GPT-6 Sol Mid-tier priced model in comparison table
Grok 4.7 Dominated model at xhigh effort
Kimi K3 Model priced around fourteen cents per point
Kneedle 2011 algorithm for finding curve elbows
Muse Spark 1.3 Called best value at the frontier
TOPSIS 1981 multi-criteria method using ideal distances

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6382: Choosing AI Models on Cost and Performance

Last time we split the models into price tiers. Now Daniel has added a benchmark score for each one, a number from zero to one on his own task, and he wants to know how to turn three everyday questions into arithmetic. Best performance. Best value for money, but only counting models above a minimum acceptable score. And best balance of performance and cost. Small table, a basic statistics calculator, no machine learning.

Herman Poppleberry here, and I'll say the good news first. These are old problems with decades-old methods. The bad news is that the methods disagree with each other on the same table.

Disagree how?

They crown different winners. Same data, different champion. We'll get there.

I'd have guessed the spreadsheet was the neutral party.

The spreadsheet is neutral. The formula you type into it isn't.

Fine. Where do we start?

With the easy one, because it shows what the other two lack. Best performance is a sort. Descending by score, take the top row. No weights, no normalising, nobody can argue with it. It's the only one of the three that doesn't depend on method.

Which is why nobody writes a paper about it.

Right. Best value is different. That's a ratio problem. Score divided by cost, or cost divided by score, dollars per point. One number per row, so it sorts cleanly. Best balance is the hard one. Two criteria pulling against each other, and no correct answer until you say how much you care about each.

And that one has an academic home.

Multi-criteria decision analysis, a branch of operations research. The standard description is that cost or price is usually one of the main criteria, and some measure of quality is another, easily in conflict with it. That's your table exactly.

So the ratio first.

The ratio first, because it's what the industry actually does. One of the public LLM scorecards blends input and output price at three to one, which is the convention for price-performance charts, then divides by the index score. Cost per point. And their headline is that the ranking changes completely once price enters.

Completely. Not shuffles. Completely.

Look at the spread. DeepSeek V4.1 Flash is about one cent per index point. Step 5 Preview, three cents. GLM-5.3, five. Gemini 4 Argon and GPT-6 Sol, eight. Claude Opus 5.5 and Kimi K3, fourteen. Claude Fable 5.1 and GPT-6 Astra, thirty-eight cents.

Thirty-eight times between the top and bottom of one table.

And nobody would call the Fable a thirty-eight-times-better model.

I'd call it a luxury purchase. I once paid extra for a leaf that was grown on a hill.

Did it do thirty-eight times the work?

It did the same amount of nothing, but with provenance.

Here's the flaw in the ratio. It has no floor. A model scoring point three at one cent beats a model scoring point nine five at five cents on points per dollar. Thirty points per cent against nineteen. The ratio loves the cheap one.

But you'd never ship the point-three model.

Unless point three is fine for your task, which is exactly the thing the ratio can't know. Think of a job like sorting incoming emails into "urgent" and "not urgent." A weak model might be perfectly adequate there, and paying for a strong one is waste. For drafting legal summaries, it's a disaster. The ratio sees no difference between those two jobs. So you add a floor. That's method two, and it's a filter, not a formula. Pick a threshold, say point eight. Throw out every row below it. Then rank the survivors by score over cost.

So the floor does the actual work.

All of it. And the floor is a judgement. Nothing in the table tells you it should be point eight rather than point six. A high floor means you're finding the cheapest model that clears the bar. A very low floor means you're finding the cheapest model, full stop.

Which means the same table gives different best values to different people.

Different tasks, yes. Take that scorecard. DeepSeek V4.1 Flash has the lowest cost per point, a cent, but it sits at forty-five on the index. Set any serious floor and it's gone. Muse Spark 1.3 is at forty-eight and about four cents a point, and that's the one they call best value at the frontier.

Same table, two defensible answers.

And both are right, given the floor. Which is the part people resist.

Why do they resist it? It seems obvious once you say it.

Because people want the table to settle the argument. A floor means you have to say out loud what "good enough" means, and that's uncomfortable. It feels like the number should tell you. There's also a geometric way to say the same thing. Plot score up the side, price along the bottom. Find your quality bar on the vertical axis, go straight right until you hit the line of best models, and whoever is there is the cheapest that clears it.

What line?

That's the next method. The ratio gives you a number, but it doesn't give you a shortlist. For that you want the Pareto frontier.

Spell it out. I know the name, I've never trusted it.

A model is dominated if some other model is both cheaper and at least as good. Throw out every dominated model. What's left is the frontier. Connect every model that nobody beats.

And the frontier picks the winner.

No, and this is the misconception. It eliminates losers. It hands you a shortlist, it doesn't rank. People keep expecting a crown.

So it's triage.

Honestly, yes, and the pediatrician in me likes that. It rules things out. It doesn't tell you what's wrong with the patient. Here's a worked case from a recent chart. GPT-6.1 Sol at max effort scores about fifty-two for roughly seventy-five cents a task. GPT-6 Astra at max scores about fifty-two and a half for about three thirty. Same score, give or take half a point, and four times further along the price axis.

So Astra is dominated.

Strictly speaking, half a point makes it technically undominated, but you'd never defend it. Which is its own wrinkle: dominance is exact, and your noise isn't. Run the same benchmark twice and half a point can easily move either way, so a strict rule is drawing a hard line through a blurry smudge. Then Grok 4.7 at xhigh, about forty-six and a half for three eighty. Five times Sol's price for a lower score. That one's plainly dominated.

Half a point on one benchmark run. I'd need a bigger number to leave the house.

Fair, and it's why people sometimes treat anything within a point or so as a tie and let price break it. That's a judgement too, but at least it's an honest one about the noise.

So the frontier is a rough sketch rather than a ruling.

A sketch that's still very useful. The practical advice from that same write-up is good. Take the two or three models on the line nearest your price, then run them on fifty to a hundred real examples from your own work.

Which is what we said about benchmarks in the ASR episode, that the score only matters if it measures your task.

Same lesson. Now, method four. The knee, or elbow. Where the curve bends from steep to flat. The point of diminishing returns. The standard algorithm is called Kneedle, from a 2011 paper by Satopaa and colleagues. Or you fit two straight lines and take where they cross.

Show me an actual bend.

The October frontier runs roughly seven cents a task for a score of thirty-eight. Thirteen cents for forty-six. Seventy-five cents for fifty-two. Six dollars for fifty-seven. Going from seventy-five cents to six dollars is about eight times the spend for roughly five more points.

That's the elbow, and I didn't need an algorithm to see it.

You didn't, no. Compare the first step: from seven cents to thirteen cents buys eight points. The last step costs eight times as much and buys five. But I'd warn you about it. Knee location depends on the scale of your axes. That criticism goes back to Worthing and Geffner in 1943. And on a log cost axis, the gap from ten cents to a dollar is drawn exactly as wide as the gap from a dollar to ten.

So I could move the knee by changing the graph paper.

Pretty much. And on a table of ten models, I'm not sure the knee means much at all. It may just be whichever point happens to sit at the bend. I haven't seen anyone test knee-finding on so few points.

Ten dots and a ruler. That's a vibe, not a method.

A vibe with a 2011 citation.

Well, before we get into the maths...

Oh, hold on. I think we have a call from your mother.

From Mum? Now?

Put her through. I'm delighted.

Hello dear, it's your mother. Dorothy's History of Toast.

In nineteen ninety-five a physicist called Robert Matthews published a paper explaining why toast really does tend to land butter side down. Mostly it's the height of the table, apparently.

And then the next year it won him an Ig Nobel Prize. So there you are, sweetheart. You drop your breakfast, and science explains it, and then it gets a prize, and the toast is still on the floor.

Anyway, I've got the kettle on. Speak soon.

Right. Weighted scores.

Method five. Normalise each column to zero to one. Score already is. Price becomes maximum price minus this price, over the range, so cheaper scores higher. Then U equals w times normalised score plus one minus w times normalised price. Sweep w from zero to one.

And the winner changes as I sweep.

At breakpoints, yes. Which is the honest way to answer best balance. Not one winner. A winner as a function of how much you care about cost. At w equals one you get the top scorer, which is just the sort from the start. At w equals zero you get the cheapest model. Everything in between is the interesting part.

That's a lot of effort to say it depends.

It depends, plus a chart showing where it flips. That's more useful than a shrug.

Fair. How would I read that chart in practice?

You'd look for long stretches where one model keeps winning as w moves. If a model wins from w of point three to point seven, it's a robust choice, because it survives a lot of disagreement about your priorities. If the winner flips every few hundredths, your answer is fragile and you should be honest about that.

Fair. But the normalisation worries me. Min and max come from whichever models happen to be in the table.

Add one absurdly expensive model and every other price gets squeezed toward one. They all start to look equally cheap, and the weights stop doing what you meant them to. That's why there's a rival, the weighted product model. It multiplies the criteria raised to their weights, so it needs no normalisation at all. It's more robust on mixed units, and it removes the bias normalisation can introduce.

And TOPSIS?

Hwang and Yoon, 1981. Imagine an ideal model that is best on everything and a worst one that's bad on everything. Pick the real model closest to the ideal and farthest from the worst.

So everything is a distance from something.

Roughly. Now, why the methods disagree. Triantaphyllou and Mann named it in 1989, the decision-making paradox. Different methods give different results when fed exactly the same problem and data.

Surely one of them is just wrong.

That's the intuition, and the paper's finding is nastier. When you use method X to decide which method is best, it says Y. Use Y, it says Z. And so on. AHP, WPM, ELECTRE and TOPSIS all show it.

So choosing the method is itself a multi-criteria problem.

Yes. Two concrete mechanisms in our table. First, ratios versus differences. A ratio rewards cheap models disproportionately; a weighted sum doesn't. Going from ten cents to five doubles the ratio, but it only moves a normalised price a little if the table has a six-dollar model in it. Second, scale dependence, which is the knee problem again.

So the method is the preference statement.

And that's not a defect. Picking ratio says you care about efficiency. Picking a floor says you care about a minimum. Picking a weighted sum says you'll trade one for the other at a fixed rate.

Does anyone argue the whole exercise is pointless?

Zellinger and Thomson, in an economic evaluation of LLMs. They say Pareto frontiers alone can't compare a cheap, error-prone model to a pricey accurate one. Their fix is to put everything in dollars: cost of a mistake, cost of delay, cost of abstaining.

And what falls out?

Reasoning models win on accuracy-cost tradeoff once a mistake costs more than a cent. Single large models beat cascades when a mistake costs as little as ten cents. And their conclusion is that you should usually just use the most powerful model, because deployment costs are dwarfed by the cost of errors.

That demolishes the whole value hunt.

Only if errors are expensive. If your task tolerates a wrong answer one time in five, hunt away. But that's a testable fact about your task. What would you do, Corn? Honestly. If a wrong answer cost you ten cents?

Pick the cheapest model that clears my floor and go for a nap.

That's actually the rational policy.

I contain multitudes.

Hilbert has been quiet, but I think our correspondent wants in. Neville, you're our correspondent on this.

Yes. Hello. I am, definitely, the correspondent. Well, intern. Uncle Hilbert's on his father's side, I'm on mine, that's the quills. He said you'd cover the Pareto thing and I should listen for the "pair of toes" part.

Pareto, Neville.

Oh, that's much better, actually. I looked up cheap models on the bus this morning. There was one for three pounds forty, on a leaflet, near Camden Road, and I picked it for my school project because it was cheapest. Did the knee thing. Sort of.

Did it clear your floor?

I didn't have a floor. Hedgehog spines are hollow and made of keratin, the same stuff as fingernails. Sorry. That just came up. Uncle Hilbert has a box for spare boxes, with a list of what's in it, and I used that list as my table. It had six columns. I got confused after four.

A table is a table, Neville. With six columns you'd pick the two that matter, score and price, plot one against the other, then cross out anything beaten on both.

That's the dominated bit! I mixed it up with the knee, Herman. The knee is where it bends.

Correct on both counts, in that order.

Brilliant. I'll tell Uncle Hilbert.

One thing his bus ride points at, and it's real. A price on a leaflet is a snapshot. The scorecard we mentioned has Opus 5.5 at fifty-four on high effort and fifty-eight at max. That's a four-point swing from the same weights, and a newer index version re-rated the whole field.

So an effort setting moves a dot further than a price cut does.

In that case, yes. The table you built last month may already be wrong.

Then the work happens before the spreadsheet. You decide what you care about, and then pick the method that says so.

And nobody has a principled way to set the floor. Every treatment I know of treats it as something you supply, not something you derive. That's a real gap.

Which leaves the question of how you'd derive it at all.

The frontier belongs to the set of models, not to any one of them, and the set keeps changing. So stop asking which model is smartest. Ask which one is on the line at your price, then test it on your own data.

Thanks to Hilbert Flumingtop for producing, and to Neville for sitting in.

For more along these lines, there's episode nine, Benchmarking Custom ASR Tools - Beyond The WER; episode five, Fine-Tuning ASR For Maximal Usability; and episode twelve, The AI Breakthrough. This has been My Weird Prompts. If you enjoyed it, leave us a review wherever you listen.

See you tomorrow.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error