{"id":"choosing-class-breaks-pricing-tiers-32129","slug":"choosing-class-breaks-pricing-tiers-32129","title":"Choosing Class Breaks for Skewed Price Data","description":"Splitting continuous API prices into discrete tiers requires choosing between competing statistical philosophies. Cartography solved this problem decades ago through choropleth class interval selection, showing how variance-minimising methods like Jenks natural breaks contrast with quantiles, density cuts, and head-tail breaks. Preprocessing choices like logarithmic transforms, zero-price offsets, and goodness of variance fit often shape the resulting boundaries far more than the clustering algorithm itself.","excerpt":"In one dimension, k-means clustering avoids random restarts entirely and runs in polynomial time via dynamic programming.","pubDate":"2026-10-31T15:45:25.801Z","tags":["statistics","open-source","ai-inference","cloud-computing"],"heroImage":"https://files.myweirdprompts.com/covers/choosing-class-breaks-pricing-tiers-32129.png","podcastAudioUrl":"https://episodes.myweirdprompts.com/audio/choosing-class-breaks-pricing-tiers-32129.m4a","podcastDuration":"18:39","episodeNumber":6383,"transcript":"Corn: Daniel has a column of batch API prices from OpenRouter, per million output tokens, and he can see the shape by eye. A very cheap cluster, a middle, a few expensive ones. He wants four tiers for a quadrant chart, and he wants the cut points to follow the real groupings instead of getting yanked around by a handful of extreme models.\n\nCorn: He named Jenks, quantiles, one-dimensional k-means, Gaussian mixtures, density valleys, and log scales. What he wants to know is how you choose, and what's the established way.\n\nHerman: Herman Poppleberry, and this one is a pleasure, because it's a solved problem in cartography. It's called class interval selection. People have been arguing about where to cut choropleth maps for decades.\n\nCorn: Choropleth being the maps where regions are shaded by a value.\n\nHerman: Right. Unemployment by county, rainfall by district. Every one of those maps forces the same decision Daniel has. You have a continuous number and you have to chop it into a handful of shades. Change where you chop, and the map tells a different story.\n\nCorn: Give me a concrete case. How different can the story get?\n\nHerman: Take a map of household income by county. Cut it into equal-width bands and one wealthy suburb county can turn nearly the whole country into the palest shade, with a single dark blob on the coast. Cut the same numbers into equal-count bands and the country looks evenly mixed, with dark patches everywhere. Same data, and two opposite impressions of how unequal the place is.\n\nHerman: George Jenks was a cartographer at the University of Kansas. His worry was making maps visually honest. He talked about a blanket of error that should be spread evenly across the map instead of piling up in one class.\n\nCorn: So a man worried about map shading wrote the algorithm Daniel's about to run on language model prices.\n\nHerman: Yes. And here's the reframe. Daniel's list isn't five algorithms. Jenks and one-dimensional k-means are the same optimisation. Jenks is the cartographic name, k-means is the machine learning name. Wikipedia says it plainly, that Jenks \"seems to be one dimensional k-means.\"\n\nCorn: Well, that's one item off the menu for free.\n\nHerman: What's left is four philosophies. Variance-minimising, which is Jenks and k-means. Count-equalising, which is quantiles. Density-valley, which is kernel density and Gaussian mixtures. And hierarchy, which is head and tail breaks. You're choosing a philosophy, not a library.\n\nCorn: Then start with the variance one, since it's two for the price of one.\n\nHerman: The objective is to make each class tight around its own mean while pushing the class means far apart. Minimise within-class variance, maximise between. You get a fit number out of it called goodness of variance fit. It's the total squared deviation around the overall mean, minus the squared deviation around the class means, divided by the total. Zero is useless, one is perfect.\n\nCorn: So you can say these four cuts explain ninety-four percent of the variance.\n\nHerman: Right, and that's the number to put next to your breaks. It also gives you a way to compare class counts. Run it with three classes, then four, then five, and watch how the number climbs. If going from three to four buys you two points and four to five buys you half a point, you've found where the returns flatten.\n\nCorn: Like an elbow plot.\n\nHerman: Same idea. Now, why one dimension is special. Finding the globally best k-means is NP-hard in multiple dimensions. So the usual Lloyd's algorithm gives you a local optimum that depends on the random seed. Run it twice, get different tiers.\n\nCorn: Which is a bad look on a chart you're defending.\n\nHerman: Imagine someone asks why a model sits in tier three, and the honest answer is that it did on Tuesday's run. In one dimension there are polynomial-time algorithms. The kmeans1d library uses dynamic programming, order k times n plus n log n. You get the true optimum. No seeds, no reruns.\n\nCorn: So for a few dozen prices, the answer is deterministic.\n\nHerman: Completely. The tooling is small and boring, which is the point. jenkspy is MIT licensed, Cython, wheels on PyPI and conda-forge. You can call a function with the class count, or use the scikit-learn style object with four classes. One trap. The README example runs it on zero through eleven and gets breaks at zero, two, five, eight, eleven.\n\nCorn: That's five numbers for four classes.\n\nHerman: Because the minimum and maximum are included. The real cuts are the inner breaks, two, five and eight. People plot the wrong lines on that one all the time. Beyond that there's kmeans1d, a ckmeans package wrapping the R implementation, and fast1dkmeans if memory matters, which for Daniel it doesn't.\n\nCorn: How would that trap show up on his chart, in practice?\n\nHerman: He'd draw four vertical lines, and two of them would sit at the very edges of the data. The chart would look like it had five tiers, or the outer lines would just hug the axis and he'd wonder why they're there. It's a quick check. Count the breaks and make sure you have one fewer than the class edges.\n\nCorn: I'd have guessed quantiles were the safe default. Equal counts, no drama.\n\nHerman: That's the common belief, and it's wrong for skewed data. Each class gets the same number of items. Esri's documentation is blunt that the map can be misleading, because similar features land in adjacent classes and widely different values land in the same one.\n\nCorn: So a dense cheap cluster gets sliced in half.\n\nHerman: Picture twenty models priced between fifty cents and a dollar fifty. Quantiles will draw a boundary straight through the middle of them, so a seventy-cent model and a ninety-cent model end up in different tiers. And a two-dollar model and a sixty-dollar model can share a tier. It's the most robust to outliers and the least faithful to groupings. Robust to the tail precisely because it doesn't look at the values.\n\nCorn: It ignores the prices. Bold for a price analysis.\n\nHerman: Then the density approach. A kernel density estimate draws a smooth curve and you cut at the valleys between humps. Conceptually that's closest to \"where does the data naturally split.\" The catch is bandwidth. With a few dozen points, a slightly wider bandwidth erases a valley and a slightly narrower one invents three.\n\nCorn: So you tune the smoothing until it shows the tiers you expected.\n\nHerman: Which is the danger. You can always find a bandwidth that confirms what you already believed. Gaussian mixtures are the standard machinery for choosing the count, you fit several and compare with BIC. I'm not going to cite a particular paper on that, it's just standard practice. But with small samples, mixtures get unstable too.\n\nCorn: Unstable how, in practice?\n\nHerman: Drop one or two models from the list and the fitted components can rearrange themselves. Each component wants enough points to estimate a mean and a spread, and a tier of three expensive models barely gives it that.\n\nCorn: So it's trying to learn a width from three numbers.\n\nHerman: And three numbers can describe almost any width. The fit can shrink that component to a spike on one model, or stretch it to swallow its neighbours, and either one can score well.\n\nCorn: What's the hierarchy one?\n\nHerman: Head and tail breaks, from Bin Jiang, in The Professional Geographer in 2013. Split at the mean. The values above the mean are the head, the minority. Recurse on the head until it's no longer heavy-tailed. The number of classes falls out on its own.\n\nCorn: That sounds built for model pricing. Huge cheap pile, handful of frontier models.\n\nHerman: Exactly that shape. Think of city sizes or word frequencies, where the average is dragged up by a few giants so only a small share of items sit above it. Split, then do it again on the giants. Though it doesn't aim for four. It gives you whatever count the tail structure produces, so if Daniel needs exactly four for a quadrant chart, he may have to merge or stop the recursion early.\n\nCorn: And which wins?\n\nHerman: None, and the literature admits it. A 1986 comparison found none of the traditional methods consistently produced homogeneous classes, each did well only on certain distributions. A 2025 paper makes the same point, that the same data can be presented differently depending on how it's binned. Your own checking of the methods against each other, which I did across a few benchmarking episodes, comes down to the same thing: the metric shapes the conclusion.\n\nCorn: Fine, the methods are settled. But I suspect what you feed them matters more than which one you pick.\n\nHerman: It does. Log scale isn't a method, it's preprocessing, and it changes the answer more than the algorithm. Jenks minimises squared error in whatever units you give it. In raw dollars, the sixty-dollar model dominates the objective and the algorithm spends its effort on the top end.\n\nCorn: So the cheap cluster, where most of the models are, gets lumped together.\n\nHerman: A gap between sixty and a hundred and twenty dollars costs enormous squared error, while the difference between fifty cents and two dollars costs almost nothing. On log ten of price, the squared error tracks ratios. And an order of magnitude more expensive is what a reader actually means by tier. Esri even ships geometrical interval for this, class breaks on a geometric series. Their example is rainfall where most stations record zero.\n\nCorn: Fifteen of a hundred stations record anything, if I remember.\n\nHerman: Yes, I flipped it earlier, eighty-five recording zero is the same thing. A long tail, built for it.\n\nCorn: And a zero in prices would break a log.\n\nHerman: Good catch. Free models, price zero, you either drop them from the fit and make them their own tier, or add a small offset. Decide on purpose, because the size of the offset changes how far the free models sit from the cheap ones on the log axis.\n\nCorn: Make that concrete. What does the offset do?\n\nHerman: Add one cent and a free model sits at log ten of point zero one, which is minus two. Add a tenth of a cent and it sits at minus three. A fifty-cent model is around minus point three either way. So the free models are two units away from the cheap cluster in one case and nearly three in the other, and that gap can decide whether they join the cheap tier or stand alone.\n\nCorn: What about outliers? Daniel specifically doesn't want a few expensive models setting the boundaries.\n\nHerman: Three honest options. Trim, transform, or use a method that ignores the tail. Quantiles ignore the tail by construction. Head and tail breaks quarantines it. The practical question is whether the top few are a real tier or noise.\n\nCorn: My instinct is a handful of frontier models at five hundred dollars is a tier. They're priced differently on purpose.\n\nHerman: Then keep them. I'd say look at the log plot first, and if they sit in a visible gap, they're a tier. If they trail off smoothly, they're a tail.\n\nCorn: So a tail you could cut anywhere, versus a cluster you can't.\n\nHerman: Yes. If you can slide a boundary up or down the tail and nothing visibly changes, the cut is arbitrary and you should say so.\n\nCorn: And then there's what happens when he makes a second chart.\n\nHerman: Esri warns that natural breaks are data-specific, not useful for comparing maps built from different data. If he later charts input prices next to output prices, or batch against standard, the breaks won't line up. A \"tier two\" on one chart would mean a different price range on the other. For charts that share a scale, quantiles or fixed geometric breaks travel better.\n\nCorn: So a reader flipping between the two charts would think a model moved tier when only the ruler changed.\n\nHerman: And they'd draw conclusions about pricing strategy from what is really an artefact of the binning.\n\nHerman: Class count, too. Every tool treats it as your choice. Jenks designed for fewer than seven because of monochrome shading, so four is well inside the intended use.\n\nCorn: Does that mean the data can't tell him four versus three?\n\nHerman: Not on its own. The fit number can show diminishing returns, but it's a judgment. If the jump from three to four is real but small, the quadrant layout he wants may be reason enough to take four. Just say that's why.\n\nHerman: So the workflow. Plot the sorted values and a density curve on a log axis. Run Jenks or k-means on log prices, and quantiles as a sanity check. Report the fit number. Prefer cuts where both land near each other.\n\nCorn: So the defensible part is the agreement, not the algorithm.\n\nHerman: Yes. And only then hand the four ranges to a language model for names. Name first and you tune the cuts to fit the labels. Also a model will happily name four tiers for data that supports three.\n\nCorn: Well, that is a good place to pause the technical side, I think.\n\nDesmond: Before you go on, this is Desmond's Money Tip. Only laugh at jokes that are worth it. I work out the savings in advance, then decide whether the joke has earned a laugh. Roughly forty laughs a year less, at a tenth of a penny of effort each, is four pence saved over twelve months.\n\nDesmond: I keep a small tin by the kettle where the laugh money would have gone. It is still empty. That is the system working.\n\nDesmond: Money is just energy you have not spent yet, and you will spend it.\n\nCorn: I might try that. Not the tin, the discipline.\n\nHerman: I'm appalled. Four pence.\n\nCorn: You were saying, about the second chart.\n\nHerman: Which one's the problem is when the breaks drift as the prices update. Then the tiers move every month.\n\nHilbert: Which is my whole complaint. A man at the hardware merchant on Jaffa Road, Mr Abramson, sold me a quarter-inch drill bit in nineteen shillings' worth of grief, I mean forty-two shekels, because he re-sorted his bins every spring. Small, medium, large. I lost two hundred and ten shekels in a year on screws that moved shelves.\n\nHilbert: Herman, I rule my own chart in pencil. Four bands. The boundaries have not moved since I drew them, and the prices underneath have wandered a long way. If a new cut lands within a pencil-width of the old line, it stays. I don't need a library for that. I re-binned once. I'm not saying what.\n\nHerman: A pencil-width is a tolerance. It's a hysteresis band, honestly, which is a real technique for stopping boundaries flickering. Thermostats do it. The heating doesn't switch on at nineteen degrees and off at nineteen and a bit, it waits for a gap, so it doesn't chatter.\n\nCorn: Though a pencil gets shorter. So the tolerance shrinks as you sharpen it.\n\nHilbert: I use a mechanical one. Point five millimetre. Fine, that's a correction, Corn, I take it.\n\nHerman: That actually connects to the second-chart problem. Stable cuts you only move when the data moves them past a tolerance.\n\nCorn: How would that work for Daniel monthly?\n\nHerman: Rerun the cuts each month on log prices. For each new break, compare it to the published one. If it's inside the tolerance, say a few percent in price terms, keep the old line. If it's outside, move it and note the change.\n\nCorn: And if a price drifts but stays inside the band?\n\nHerman: Then the model keeps its tier and nobody sees a change. The only moves readers notice are the ones that actually mean something, like a provider cutting its price by half and crossing a line.\n\nCorn: Then the real question for Daniel is what the chart is for. Comparing models to each other wants portable cuts. Showing the shape of one market wants data-specific ones.\n\nHerman: And since no method wins on its own, the agreement check is what he can actually defend. Keep a tolerance on the old lines, as we just heard, so the tiers don't wobble each time a price changes.\n\nCorn: Thanks to Hilbert Flumingtop for producing, as always.\n\nHerman: If you enjoyed this one, try episode nine, Benchmarking Custom ASR Tools - Beyond The WER; episode seven, Building Custom ASR Tools; and episode twelve, The AI Breakthrough. This has been My Weird Prompts. If you enjoyed it, a review helps other people find the show.\n\nCorn: See you tomorrow.","showNotes":"Dividing continuous data into discrete tiers is a classic challenge borrowed from cartography, where selecting class intervals dictates how choropleth maps communicate information. When segmenting skewed figures like language model pricing, algorithmic choices shape the narrative. Variance-minimizing approaches like Jenks natural breaks and one-dimensional k-means solve the exact same optimization problem, offering deterministic, optimal splits through dynamic programming. Meanwhile, common defaults like quantiles enforce equal counts but can slice dense clusters in half, and density-based techniques often suffer from sample instability. Preprocessing decisions, such as taking the logarithm of values to emphasize proportional ratios over raw dollar extremes, frequently matter far more than the specific algorithm chosen, while comparing charts requires consistent rulers rather than data-specific cut points.","promptSanitized":"Say I have a small numeric dataset, for example batch API prices on OpenRouter: price per million output tokens, maybe input too. Looking at it, I can roughly see the shape. There's a very cheap cluster, a mid range and an expensive range. I want to split it into about four tiers for a quadrant-style chart or an analysis. The question is where the cut points should go, so that they follow the main groupings in the data and don't get dragged around by a few outliers.\n\nThis is a really common statistical job, so what's the established tooling and method for it? Are there tools that take a distribution and find where it naturally splits? I'm thinking of things like Jenks natural breaks, quantiles, k-means on one dimension, Gaussian mixtures, kernel density valleys, and log scales for prices that span orders of magnitude. How do you choose between them, and how do you keep outliers from setting the boundaries? The step after that, naming the tiers, is probably just a language-model job, so the focus here is the numbers: how to find the divisions properly.","transcriptUrl":"https://episodes.myweirdprompts.com/transcripts/choosing-class-breaks-pricing-tiers-32129.md","url":"https://myweirdprompts.com/episode/choosing-class-breaks-pricing-tiers-32129/","_links":{"self":"https://myweirdprompts.com/api/episodes/choosing-class-breaks-pricing-tiers-32129.json","html":"https://myweirdprompts.com/episode/choosing-class-breaks-pricing-tiers-32129/","audio":"https://episodes.myweirdprompts.com/audio/choosing-class-breaks-pricing-tiers-32129.m4a"}}