Daniel's been staring at his agent bills and sent us a repository to walk through. Headroom — it's a Python tool from headroomlabs-ai that sits between your agent and the LLM and compresses everything before it hits the context window. Tool outputs, logs, files, RAG chunks — all of it. The README claims roughly twenty percent token savings for coding agents, sixty to ninety-five percent for JSON-heavy payloads, and — this is the part Daniel flagged — supposedly unchanged answers. He wants to know what's actually happening under the hood, how the compression strategies work, where the trade-offs bite, and how the project guards against silently degrading agent output. I pulled the repo. It's not small.
Two thousand eighty-six files. Eight point seven million tokens just in the codebase. I spent the morning in the transforms directory and the proxy pipeline and honestly — this is one of the more carefully engineered pieces of agent infrastructure I've read. The core insight is that most of what flows through an agent's context window is waste. Not noise exactly — it's real data — but the LLM doesn't need all of it to produce the same next action. The trick is knowing which parts it does need.
That's the whole ball game. "Same answers" is the claim that makes this worth talking about instead of just another truncation script. How do they actually deliver that?
They don't truncate. That's the first thing to understand. Truncation is what everyone reaches for first — chop the middle, keep the head and tail, hope nothing important was in the middle. Headroom never does that. Every compression strategy is content-aware. It reads what it's compressing, classifies it, and applies a transform that's designed for that specific shape of data. The pipeline has something like fifteen different compressors, and the router picks which ones to apply based on what the content actually is.
So it's not one compressor. It's a dispatcher that looks at the payload and says "you're a diff, you're a log, you're JSON, you're prose."
And the dispatcher itself is interesting. It's not regex. There's a Rust module — the content detector — that uses tree-sitter grammars for code, a unidiff parser for diffs, and a classifier for JSON versus prose versus logs. The classification runs before any compression, and it's designed to be fast. Sub-millisecond for typical payloads. The whole thing is in the headroom-core crate, and the Python side calls into it through PyO3.
That's a deliberate choice — they're not running an LLM to decide how to compress for the LLM.
Right, that would defeat the purpose. The compression itself has to be nearly free in both time and tokens. The pipeline is deterministic and rule-based. No model calls. The only ML anywhere in the compression path is an optional ModernBERT model for prose — and even that has a fast deterministic fallback called TextCrusher that they built specifically because the ML path was too slow for large payloads.
Walk me through the actual strategies. What does it do when it sees a diff?
The diff compressor is one of the cleanest examples. A typical agent diff — say Claude Code or Codex outputs a patch — has a lot of structural repetition. The compressor strips the index lines, normalizes the hunk headers, and folds unchanged context lines into a compact representation. But here's the key: it preserves every changed line verbatim. The model needs to see exactly what was added and removed to verify correctness. The compressor just removes the scaffolding the model already knows how to reconstruct.
And the model actually can reconstruct it? It doesn't get confused by the compressed format?
That's where the "same answers" engineering comes in. Every lossy compressor has a corresponding retrieval path. The compressed output includes markers — they call them CCR markers, for Compressed Content Retrieval — that are essentially hash keys. If the model ever needs the original content, it can call a tool called headroom_retrieve with that hash and get back the exact original bytes. The proxy injects this tool into the model's available tools automatically.
So it's not really lossy compression in the traditional sense. It's lossy display with a lossless recovery path.
That's exactly the right way to think about it. The model sees a compressed representation that preserves everything the compressor believes is salient. If the compressor guessed wrong — if it stripped something the model actually needed — the model can retrieve the original. The system is designed to be recoverable.
How often does the model actually retrieve?
According to their telemetry, very rarely. The compression feedback learner tracks retrieval rates and uses them to tune how aggressive the compression is. If a particular type of content keeps getting retrieved, the system backs off and preserves more. The CHANGELOG mentions a bug fix where they discovered the feedback was inverted — a successful eviction was being counted as a retrieval, which made the learner think it was compressing too aggressively when it was actually doing fine. They fixed that.
That's the kind of bug that silently makes your product worse for months and you only catch it because someone notices the numbers don't make sense.
And it's exactly the kind of knock-on effect that makes this project interesting to read. The compression itself is clever, but the infrastructure around making it safe — the retrieval path, the feedback loop, the content-aware routing, the fail-open design — that's where the real engineering is.
Let's talk about the JSON compression, because that's where the big numbers come from. Sixty to ninety-five percent savings on JSON payloads. What's actually happening there?
That's the SmartCrusher. It's the most sophisticated compressor in the pipeline, and it lives in both the Python and Rust sides — they ported it to Rust for parity, and there's a whole parity test suite to make sure they produce identical output. The idea is that most JSON in agent contexts is either tool output or API responses, and it's full of structural repetition. Arrays of similar objects, nested dictionaries with repeated keys, long numeric sequences.
So it's not just minifying.
Not at all. Minifying JSON — stripping whitespace — saves maybe ten to fifteen percent. SmartCrusher does several things. First, it analyzes the structure: it identifies repeated keys across array elements and factors them out into a schema-like header. Then it compresses numeric arrays using changepoint detection — if you have a time series that's mostly linear, it stores the breakpoints instead of every value. For string arrays with low cardinality, it deduplicates. For arrays of similar objects, it extracts the common structure and represents each element as a delta from the template.
And the model can read that compressed representation and still understand what the data says?
That's the design goal. The compressed output is still valid JSON — it's not binary or base64. It's a more compact JSON representation that preserves the semantic content. A model looking at a SmartCrusher-compressed array of API responses can still see the field names, the value ranges, the structure. It just doesn't see the same field name repeated forty times, or a thousand identical numeric values.
What about the edge cases? What if the JSON is irregular — every object has different keys?
SmartCrusher has a classifier that runs first. If the content doesn't match the patterns it knows how to compress, it passes through unchanged. There's a whole set of passthrough rules: small objects, short arrays, non-JSON content, content with protected patterns. The compressor is conservative about what it touches.
Protected patterns?
They added an audit-safe mode. You can configure patterns — file paths, error codes, identifiers — that must survive compression verbatim. If SmartCrusher can't guarantee those patterns will be preserved, it fails closed and returns the original uncompressed content. This was added for compliance use cases where certain data absolutely cannot be altered, even if it's going to be retrievable.
That's a lot of engineering for what started as "let's save some tokens on tool outputs."
It reflects the reality that this tool sits in the critical path. If it breaks something, the agent breaks. The entire proxy is designed to fail open — if compression fails for any reason, the original content is forwarded unchanged. There's a whole error-handling architecture around this. The Rust proxy has timeouts, the Python side has fallbacks, and there are integration tests for basically every failure mode you can imagine.
Speaking of the proxy — let's map the integration surfaces. Daniel asked about the three ways to use this: library, proxy, and MCP server.
The library is the simplest. You import headroom and call compress on your content before sending it to the model. It's a synchronous Python API — you pass in messages, tool definitions, system prompts, and it returns compressed versions. This works if you're writing your own agent loop and you want to compress explicitly.
But that means you have to remember to call it everywhere.
Which is why most people probably use the proxy. The proxy sits as a local HTTP server — by default on localhost port eight-seven-eight-seven — and intercepts every API call your agent makes to Anthropic, OpenAI, Google, or any OpenAI-compatible endpoint. It compresses the request body before forwarding it, and it can also compress the response. The agent doesn't know it's there. You point your agent's base URL at the proxy and it just works.
And the proxy handles all the provider-specific quirks?
That's one of the more impressive parts of the codebase. The proxy has separate handler paths for Anthropic's messages API, OpenAI's chat completions and responses APIs, Google's Gemini generateContent, and Bedrock's InvokeModel and Converse. Each path understands the provider's message format, where the content lives, how tool calls are structured, and how to reconstruct the request after compression. There's also a WebSocket path for Codex's real-time API.
Because Codex uses WebSockets, not HTTP.
Right. And the proxy handles that transparently — it intercepts the WebSocket upgrade, compresses the frames, and forwards them. The agent thinks it's talking directly to OpenAI.
What about the MCP server path?
That's the third integration surface, and it's the most interesting for agent frameworks. MCP — Model Context Protocol — is how tools are exposed to models in a standardized way. Headroom runs an MCP server that provides the headroom_retrieve tool. Any MCP-compatible agent can connect to it and get access to the compressed content retrieval. The proxy also injects this tool automatically, but the MCP server lets you use it even if you're not running the proxy — for example, if you're using the library mode and you want the retrieval path without the full proxy.
So the three surfaces cover different levels of integration depth. Library for explicit control, proxy for transparent interception, MCP for the retrieval tool. And they all share the same compression engine underneath.
The same Rust core, yes. The Python library, the proxy, and the MCP server all call into the same PyO3 extension module that wraps the Rust transforms.
Let's get into the trade-offs. Daniel specifically asked when lossy compression could quietly hurt an agent's output, and how the project guards against it.
The biggest risk is the compressor stripping something that looks like noise but is actually signal. A log line that seems repetitive but contains a subtle variation. A JSON field that appears in every object but has one crucial outlier. A diff context line that the model needs to understand why a change was made.
The classic compression problem — one person's redundancy is another person's critical data.
And in the agent context, the compressor doesn't know what the model's task is. It doesn't know if the model is debugging a crash, writing code, or analyzing a dataset. So it has to make conservative guesses.
How conservative?
Several layers. First, the content detector classifies the content type, and each compressor has its own safety rules. The log compressor, for example, preserves error lines and stack traces verbatim — it only compresses the repetitive informational lines. The diff compressor preserves all changed lines. The code compressor preserves function signatures and only compresses bodies.
What about the code compressor specifically? That seems risky — compressing code that the model is supposed to understand or modify.
The code compressor uses tree-sitter to parse the code into an AST. It preserves all signatures, all imports, all type definitions. What it compresses is function bodies — and even then, it doesn't remove them. It replaces the body with a compact representation that preserves the structure but not the implementation details. The idea is that if the model needs to see the implementation, it can retrieve it. If it just needs to know what functions are available and what they return, the compressed form is sufficient.
And this is configurable per language?
The tree-sitter grammar set covers all the major languages. The CHANGELOG mentions they added C-sharp support recently — tested on Newtonsoft.Json and Polly, with token savings of sixteen to thirty-eight percent and zero syntax errors in over seventeen hundred files.
That's a strong validation claim. Zero syntax errors on real-world codebases.
They clearly test this extensively. There's a parity test suite that compares the Python and Rust implementations byte-for-byte. There are fidelity regression tests. There are adversarial tests. The benchmark directory has things like worst-case benchmarks and adversarial CCR tests.
Another risk: the compressor could break the model's prompt cache. If the compression changes the prefix of the conversation, the provider has to re-cache everything, and you lose the caching discount.
This is a real problem and they've thought about it. The proxy has a whole cache stabilization subsystem. There's a prefix tracker that monitors what the provider has cached and ensures that compression only happens on the new, uncached suffix. The frozen prefix is forwarded byte-identical on every turn. There's even a mode — cache mode — where the entire strategy is to freeze the cached prefix and only compress the new content, trading some compression savings for cache hit rates.
And that mode is for when the cache savings outweigh the compression savings?
If you're in a long conversation and most of the context is already cached at the provider, compressing the prefix would bust the cache and you'd pay full price for tokens that were previously nearly free. Cache mode preserves the cache and only compresses the new turn.
How much does the cache actually save?
Anthropic charges ten percent of the base price for cache reads. So if you have a hundred-thousand-token context and ninety thousand of it is cached, you're paying for ten thousand tokens plus nine thousand cache-read tokens — effectively nineteen thousand tokens of cost instead of a hundred thousand. That's an eighty-one percent cost reduction before any compression. Busting that cache to save another twenty percent on the prefix would be a net loss.
So the system has to make a decision: compress or preserve the cache. How does it decide?
It doesn't have to decide — the cache mode handles it automatically. The prefix tracker knows which messages the provider has cached, and the compression pipeline skips those messages. The new messages get compressed normally. It's the best of both worlds.
What about the output side? You mentioned the proxy can compress responses too.
That's the output shaper. It's a separate subsystem that operates on the model's response before it reaches the agent. The idea is that models are often more verbose than necessary — they explain things the agent already knows, they repeat information, they use more words than needed. The output shaper tries to reduce that verbosity without changing the semantic content.
That feels even riskier than input compression. You're altering what the model said.
It's opt-in and experimental. The CHANGELOG mentions it's behind feature flags and tracked with holdout keys for evaluation. They're clearly being careful about it.
Let's talk about the memory system, because that's a whole other dimension. The repo has an entire memory module with vector search, graph storage, and multiple backends.
The memory system is separate from the compression pipeline, but it serves a related purpose. It's a way to store information across sessions so the model doesn't have to re-learn it. The proxy can inject relevant memories into the system prompt based on the current conversation. It uses embeddings for semantic search, and there's a graph store for relationship-based retrieval.
So it's like a RAG system but for agent memory.
And it has its own compression concerns — memories have to be compact enough to fit in the system prompt without consuming too many tokens. The memory module has budget management, ranking, and a traffic learner that observes which memories get used and adjusts accordingly.
The traffic learner is another feedback loop.
The whole project is feedback loops. Compression feedback, memory traffic learning, cache hit rate monitoring, savings tracking. Everything is instrumented and everything feeds back into tuning the system.
That's the part that makes me think this is a production tool, not a research project. The compression strategies are interesting, but the observability and safety infrastructure is what makes it usable.
And it's all exposed. The proxy has a dashboard — there's a whole HTML dashboard with savings charts, per-project breakdowns, cache hit rates, request logs. The stats endpoint returns JSON with detailed metrics. You can see exactly what's being compressed, how much you're saving, and whether the model is retrieving compressed content.
The per-project breakdown is a nice touch. If you're running multiple agents or multiple projects through the same proxy, you can see which ones benefit most from compression.
That was added because people were running the proxy as a shared service — a team all pointing their agents at the same proxy instance — and they wanted to know whose usage was driving the savings.
Let's circle back to the "same answers" claim. After reading the code, do you buy it?
I buy it with caveats. The system is engineered to be conservative. It preserves everything it can identify as potentially important. It has a retrieval path for everything it strips. It fails open. The feedback loops tune the aggression. And the benchmarks — the ones in the repo — show high answer retention rates across multiple eval suites.
But?
But the real test is in production, with real agents doing real tasks. The retrieval path only works if the model knows to use it. If the compressor strips something subtle — a configuration value that looks like a default but isn't, a log line that seems identical to the previous one but has a different timestamp that matters — the model might not realize it's missing something. It might produce a wrong answer without ever knowing it needed to retrieve.
That's the fundamental tension. The compressor is making decisions about what's important without knowing the task. And "important" is task-dependent in ways that are hard to capture in rules.
The project acknowledges this implicitly through its design. The retrieval path exists precisely because the compressor can't be perfect. The feedback loop exists because the initial compression policy might be wrong. The audit-safe mode exists because some data is too important to risk. The whole architecture is built around the assumption that compression will sometimes be wrong, and the system needs to recover gracefully.
That's a more honest engineering posture than "our compression is lossless for all practical purposes."
It's the difference between claiming perfection and building for imperfection. I respect that.
What's the install experience like? If Daniel wants to try this on his agent stack.
It's designed to be low-friction. For the proxy, it's a pip install and then you point your agent at localhost. They have a wrap command that automatically configures Claude Code, Codex, Cursor, Copilot, and about fifteen other agents. The wrap command modifies the agent's config to route through the proxy, and unwrap restores the original config.
So you can try it without committing to it.
And the proxy has a simulation mode where it shows you what it would compress without actually compressing anything. You can see the savings you'd get before you turn it on.
That's smart. Reduce the barrier to adoption by letting people see the benefit before they take any risk.
The whole project is designed around that principle. Fail open, show before you change, make it easy to undo. For a tool that sits in the critical path of agent operations, that's the right posture.
One thing I noticed in the repo structure — there's a REALIGNMENT directory with a dozen phase documents. What's that about?
That's their architectural roadmap. They're in the middle of a multi-phase migration from Python to Rust for the proxy layer. Phase A through H cover different components — cache stabilization, live zone compression, Bedrock and Vertex support, auth mode, observability, and eventually retiring the Python proxy entirely in favor of the Rust one.
So the Rust proxy is the future, and the Python proxy is being phased out.
The Rust proxy is already in production — it's in the crates directory and it's what gets compiled into the binary. The Python proxy still exists for backward compatibility and for features that haven't been ported yet. Phase H is literally called "Python retirement."
That's a significant engineering investment. Rewriting a proxy in Rust while maintaining the Python version and keeping both in sync.
It tells you they're serious about performance and reliability. A Rust proxy can handle more concurrent connections with lower latency and no GIL contention. For a tool that sits in the request path, latency matters. Every millisecond the proxy adds is a millisecond the user waits.
What's the latency overhead in practice?
The benchmarks directory has latency tests. I haven't run them, but the design suggests it should be low single-digit milliseconds for most payloads. The content detection runs in Rust, the compression transforms are Rust, and the proxy is async. The only potential bottleneck is the ML-based prose compression, and they built the TextCrusher fallback specifically to avoid that.
For most requests, the proxy adds negligible overhead.
The token savings more than compensate. If you're saving twenty percent on input tokens and the proxy adds five milliseconds, you're coming out ahead on both cost and — for large contexts — total latency, because sending fewer tokens over the network is faster.
All right. If I'm an agent builder listening to this, what's the one thing I should take from this deep dive?
Context compression is real and it works, but the difference between a naive implementation and a production-ready one is mostly in the safety infrastructure. Anyone can write a script that truncates tool outputs. What makes Headroom interesting is the content-aware routing, the retrieval path, the cache stabilization, the feedback loops, and the fail-open design. Those are the parts that let you run compression in production without waking up to find your agent has been silently producing wrong answers for a week.
The second thing — which you implied but didn't say — is that you should still test it on your own workloads. The benchmarks are encouraging, but your agent's behavior on your tasks with your data is the only validation that matters. The simulation mode exists for exactly that reason.
That's fair. The project gives you the tools to evaluate it safely. Use them.
We should thank Hilbert Flumingtop for producing, as always.
This has been My Weird Prompts. If you want to dig into the code yourself, it's at github dot com slash headroomlabs-ai slash headroom. The docs are thorough — start with the architecture page and the how-compression-works guide.
We'll be back soon.
