Episode 6286 custom

Why Claude Code Replaced Vector Search with Grep

Claude Code creator Boris Cherny revealed early prototypes used local vector search before abandoning it for agentic grep.

Oct 26 23:02 Transcript

With

0:00

Episode Details

Episode ID: MWP-6488
Published: Oct 26
Duration: 23:02
Audio: Direct link
Pipeline: V5.22
TTS Engine: chatterbox-regular
Script Writing Agent: GPT-6.1 Sol
Topics: context-and-memory ai-coding-assistants rag model-context-protocol

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

A question regarding agentic harnesses and memory. Consider the common case of a user who uses an AI agent framework like Claude Code or Codex to do work repeatedly in a repository. Over the course of hours, days, or weeks, the user will run many sessions ide of this repository. These harnesses typically have features like internal memory and the ability to put down some lightweight context files like a single markdown file at the base of the repo, which acts as a sort of system prompt containing instructions for working in this repo specifically. But none of these obviate, at least apparently, the need for the agent to mop up the broader surrounding context of the code base every time they run. On the surface, this seems ridiculously inefficient. And I suspect that there is plumbing going on just beneath the surface. The most logical plumbing I can imagine would be something like a persistent vector store, in which the agent creates local embeddings of the code base so that semantic understanding can be loaded extremely quickly upon future session initiation and periodically updated over time. Do agentic harnesses do this without a scene, or storing it in some invisible part of the filesystem, or maybe just in the cloud providing the API? How does that work?

Coding agents navigating large repositories often appear to possess an invisible memory of the codebase, but the reality is far simpler. Tools like Claude Code explicitly abandon precomputed vector databases and persistent repository embeddings. Creator Boris Cherny noted that early prototypes discarded local retrieval-augmented generation because agentic search—leveraging path matching, ripgrep content queries, and direct file inspection—proved more reliable, simpler, and immune to stale indexes. Instead of maintaining a hidden semantic map, these systems rely on explicit text instructions, local session transcripts, and auto-memory files. Cost efficiency comes not from pre-indexing, but from server-side exact-prefix prompt caching within serving infrastructure, which reuses computation for unchanged context without relying on stored embeddings.

Sources

What the research for this episode read before the script was written. Primary sources first.

Mentions

AGENTS.md Standard agent instructions markdown file
Anthropic AI safety company behind Claude
Bedrock AWS model serving platform with prompt caching
Boris Cherny Claude Code creator, discussed RAG removal
Claude Code Anthropic's agentic coding CLI tool
Codex OpenAI's coding agent, local and cloud modes
Cursor AI-powered code editor
Devin Desktop Coding agent product that indexes repositories
LMCache KV cache project with Claude Code prefix analysis
ripgrep Fast recursive grep tool backing content search

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6286: Why Claude Code Replaced Vector Search with Grep

The obvious answer to a coding agent reading the same repository again is to give it a better memory. Daniel wants to know whether Claude Code and Codex already do that secretly. He works in the same repository over days and weeks, gives the agent its instructions, then watches it go around collecting context again. He suspects a persistent embedding index, tucked away locally or behind the API. Sensible suspicion. Claude Code tried that design and threw it out.

The replacement is wonderfully unglamorous. Search filenames, search text, read files. Then do it again with a better idea of what you're looking for. The intelligence is deciding where to look next, rather than searching a precomputed semantic map.

An expensive brain operating a search box.

With an important discount behind that brain. Daniel is right that there's hidden plumbing, but it's prompt caching rather than a vector database.

I can see you preparing to defend grep as the future.

I intend to defend it as the present. The future can submit a pull request.

Then distinguish the machinery for me. Watching files scroll past doesn't reveal whether the agent is remembering, retrieving, or paying to discover the same thing again.

Claude Code doesn't pre-index your repository, doesn't generate a persistent embedding index of it, and doesn't maintain a semantic repository index in the cloud. It uses Glob for path matching, Grep, backed by ripgrep, for content search, and Read for file contents. Anthropic calls that agentic search.

Codex belongs in the same conversation, though we shouldn't pretend its documented memory features are identical.

Its documented instruction system is thinner. There isn't a documented automatic repository embedding store there either. What we can actually describe is an instruction chain assembled from markdown files. I wouldn't turn that into a claim that we know every undocumented internal feature.

The distinction I care about is between retaining instructions and retaining understanding. A file saying how to run tests doesn't mean the agent remembers why our payment code is a swamp.

The persistent surface is mostly explicit text. Claude Code has project instructions and auto memory on disk. Codex has its AGENTS dot md instruction chain. There are also session transcripts, which let you resume a conversation. Separately, the model-serving infrastructure can cache computation for text you've already sent.

Three drawers, then. Instructions, conversation, saved computation. None automatically contains a current mental model of the repository.

Saved computation doesn't even constitute knowledge carried into a fresh conversation. It accelerates processing when identical text appears again. We'll need that distinction when we get to the bill.

Before the bill, I want the discarded database. You don't casually remove the part everyone assumes makes the product clever.

Boris Cherny, Claude Code's creator, said early versions used retrieval augmented generation plus a local vector database. His explanation was that agentic search generally worked better and was simpler, with fewer security, privacy, staleness, and reliability issues. Another Claude engineer described the performance gap as surprisingly large.

The prototype had the fashionable component. Production recovered.

Don't enjoy that too much, Corn. You'd defend a filing cabinet if it had a sufficiently repairable hinge.

Only if it could find the invoice.

That's actually the right demand. In code, finding the precise thing matters more than finding text that sounds related.

An error message is unusually cooperative evidence. It gives you the exact words to search for. A function name does too. Embeddings seem more useful when I know what something does but don't know what anyone called it.

That can help. But the agent can translate your description into several searches, inspect the results, and revise. Suppose you ask why login fails after a token refresh. It searches the error string, reads the refresh handler, follows an import, then checks the caller. Each result changes the next query.

Retrieval can iterate too, though. We shouldn't award grep exclusive rights to curiosity.

Fair correction. Retrieval can absolutely be part of a loop. The advantage here is that the built-in loop uses direct, current evidence without requiring an embedding pipeline. A semantic hit supplies candidates. The agent still needs to establish which symbol, branch, and caller actually govern the behavior.

Especially after an afternoon of edits. The index may describe the code before the fix that caused the new bug.

Staleness bites hard in a working repository. Files change, branches switch, generated output appears, worktrees diverge. A maintained index can handle those events, but now you've built a synchronization system. A partial update can return a plausible fragment that no longer matches the working tree.

Yesterday's bug, delivered instantly.

Reliability includes that synchronization, plus chunking, embedding generation, storage, and retrieval. Security and privacy add another set of questions. What gets indexed? Are ignored files excluded? Do deleted secrets remain in a derived artifact? A local database avoids some cloud exposure, but still creates another copy to govern.

Grep can read something sensitive too. The difference is fewer persistent artifacts, not immunity.

Fewer artifacts and fewer moving parts. Would you accept slower discovery if it reliably showed you the current file?

For changes I'm about to merge, yes. I would rather it rediscover a fact than confidently remember a fact that expired. But I still want it to retain decisions the code cannot express. Why we postponed a migration, for instance.

That's very close to the intended split. Claude's project instruction files load from the working directory and its ancestors, assembled root-downward. They concatenate rather than behaving like configuration files that neatly replace one another. Instructions in subdirectories load when the agent works there.

Meaning contradictory instructions can both attend the meeting.

Yes. Writing a closer file doesn't erase the earlier text. Imports can expand inline through four hops, and the recommendation is to keep each instruction file under two hundred lines. Long instructions consume context and become harder to follow.

Herman, you'd fill that allowance before explaining the test runner.

I would put the explanation in an imported document.

There he is.

Which would still consume context. I accept the charge.

Auto memory lives locally, beneath Claude's hidden home directory, in a project-specific memory directory. Its MEMORY dot md index is only partly loaded at startup, limited to the first two hundred lines or twenty-five kilobytes. Other memory files are read when needed. Worktrees of the same repository share that memory directory.

Sharing decisions across worktrees sounds useful until a note says the migration is finished and I'm standing in the branch where it hasn't started.

The intended content tries to avoid that. Notes have types: user information such as expertise and preferences, feedback from your corrections, project information such as deadlines and decisions, and references to external information. The instruction is to skip facts derivable from the codebase or already present in project instructions.

Deliberately refusing to remember the part Daniel hoped it remembered.

Deliberately avoiding duplicate sources of truth. The code can tell it the current function signature. The code cannot tell it that you prefer a small patch because a release is approaching.

I like that boundary. I don't entirely trust the agent to observe it.

Neither do some people using it heavily. We'll get to what happens when those notes turn into a second, less accurate repository.

Apparently a new instruction has arrived. Before anyone files it, there is no World Podcast Authority.

Good afternoon. This is the World Podcast Authority, compliance division. Your show's website has no favicon, and the Authority treats a site without one as an unsigned letter. Kindly complete Form Favicon Declaration, reference number four seven one two, and return it within fourteen days.

There isn't an organization called the World Podcast Authority.

I have noted your remark. Regardless, your podcast licence, issued under Schedule Nine of the Broadcasting of Spoken Audio Regulations, may be revoked if the form is not received by the twenty eighth of this month.

Podcasting doesn't require a podcasting licence.

I will make a note of that. A form will follow in the post. Good afternoon.

Codex assembles its instructions differently. It begins with global instructions, including an override file if present, then walks from the project root toward the working directory. At most one instruction file is selected per directory. Closer instructions appear later and can take precedence.

A fresh assembly each time, rather than some increasingly mysterious cached version.

OpenAI explicitly says it rebuilds that chain every run, including the start of an interactive terminal session. Its default combined allowance is thirty-two kibibytes. Past the cap, additional material is silently left out.

Silence is a bold interface for missing instructions.

The other local persistence is conversation history. Claude saves transcripts as line-oriented JSON in its project storage. Continue and resume reopen a session, while forking makes a branch from conversation history. Rewinding is another operation on that session.

Resuming isn't a new agent remembering me. It's reopening the conversation with all the material that was already in it.

Same session identifier, more transcript appended. That's the part of Daniel's intuition that half-lands. Something survives, but it's inspectable text rather than a hidden semantic reconstruction of the repository.

Inspectable is good. Affordable is still pending.

Each request sends the context again: system instructions, project context, previous messages, tool results, and the new message. The model doesn't retain a private conversational state between requests. Server-side prompt caching lets the serving system reuse processing for an unchanged prefix.

Re-sent doesn't necessarily mean re-computed.

Correct. The match is exact-prefix, not semantic similarity. Change something early and the material after it needs processing again. There isn't an independent cache entry for every file that can be rearranged freely. Claude Code therefore puts relatively stable content, such as system instructions and project context, near the front.

Then reading a file again in the tool trace and processing yesterday's identical transcript are different costs. The fresh read still has to happen, while old text may qualify for the discount.

Yes. That distinction prevents a lot of misleading arithmetic. Caching doesn't eliminate tool execution, new file contents, output tokens, or the agent deciding to investigate an irrelevant directory.

Where does this cache actually sit? Daniel's cloud suspicion was pointing somewhere real.

In the infrastructure serving the model. Direct Anthropic access uses Anthropic's serving infrastructure. Through Bedrock or Google's Agent Platform, it's the provider's serving infrastructure. Foundry depends on the deployment, and a custom gateway depends on where requests are forwarded.

A gateway moves the question. It doesn't answer it.

The cached material is processed prompt tokens, not embeddings of your repository. Entries expire after inactivity. It doesn't become a searchable personal library merely because the server has done some reusable work.

What would you call the economic bargain, then? We're tolerating repeated context assembly because processing the repeated portion is cheap.

Cache writes cost one point two five times base input pricing. Cache reads cost one tenth of base. You pay extra to establish reusable computation, then much less when it matches again.

The machine charges rent on familiarity.

In LMCache's December analysis, the shared prefix appeared across ninety-two percent of turns in the Claude Code loops they examined, including subagent loops. That's an observed workload pattern, not a guarantee for every session. It shows why the request layout matters so much.

It also means a spectacular discount can coexist with a visibly repetitive agent. The user sees the work repeated, while the provider sees a favorable prefix match.

One illustrative calculation put a session with two million input tokens at six dollars without caching, versus about a dollar fifteen with that caching pattern.

The rereading is real. So is the coupon.

Those are input-side example costs, not a universal session price. Model choice, cache writes, context length, and output change the bill. The pricing example uses three dollars per million input tokens below the long-context threshold. Above two hundred thousand tokens, that example's input rate rises to six dollars per million.

More reason not to turn an example into a budget promise. Would you keep a session running just to preserve the cache?

I'd keep it running while it remained coherent. What would you do, Corn, once it starts carrying assumptions from three abandoned approaches?

Start fresh. Cheap wrong answers aren't a saving. But I'd want to preserve the decision we reached, not the entire route through the hedge.

That's where compaction and subagents come in, with different compromises. Compaction summarizes a full conversation. It doesn't preserve everything exactly. Path-scoped rules and nested project instructions can be summarized away along with the rest.

Even instructions can become somebody's recollection of instructions.

After compaction, Claude Code re-reads up to five recently modified files. A file above five thousand tokens can return as a path reference rather than its contents. So a file being mentioned in the summary doesn't prove its implementation is still available.

The path remembers where the answer lives. Unfortunately, paths don't fix bugs.

An Explore subagent gives the investigation its own context window. It can read extensively and return a summary with a small metadata trailer, keeping most of those reads outside the main conversation.

Then we exchange transcript pollution for the possibility that the investigator omits the awkward detail.

We do. You can ask for a targeted follow-up or inspect the underlying file. Cherny's broader bet is on agent topologies, several fresh isolated contexts, rather than one agent accumulating an increasingly muddy history.

You sound personally relieved by that design. Separate contexts would keep your installation experiments from becoming instructions I have to obey.

You already decline those instructions.

I've implemented isolation manually.

Freshness has a cost too. Switching models changes the reusable computation. Changing effort level on most models, toggling fast mode, or editing the system prompt can invalidate the cache. Starting a fresh session also loses the old conversational prefix, although an identical stable opening may still match.

That's narrower than saying every new session flushes everything.

It needs to be narrower. Exact matching is the rule. A new session can reuse a common opening; it cannot inherit a vanished conversation by wishing hard.

Hm.

Auto memory tries to bridge that gap, but now someone has to decide what deserves promotion into a durable note. I'd rather curate ten lines than supervise a machine producing ten pages about my preferences.

Some practitioners report disabling auto memory and seeing performance improve. The complaints are concrete: too much writing, poor trimming, and old information treated as current. One person working on a Chromium fork described the agent confidently relying on test results from an earlier Chromium version.

A regression test from a different reality.

Would you disable it entirely, or review what it writes?

Review it if the project benefits. Keep stable decisions and corrections. Reject a diary of successful tool calls. The fact that a test passed yesterday is not a permanent characteristic of the software.

Scope is another lever. One developer reduced the files in the working scope from fifty-seven thousand seven hundred twenty to two thousand nine hundred seventy-seven by launching inside a monorepo subdirectory.

Nearly fifty-five thousand files removed from the invitation list.

It doesn't create an access-control boundary by itself, and it doesn't prove every file would otherwise have been read. It changes the agent's starting neighborhood and the instructions it encounters.

Which is useful without pretending the whole repository was loaded. The scrolling feels comprehensive, but often it's repeated orientation. It needs to find the relevant patch of code, not swallow every file.

I still think carrying every old correction into a room becomes impractical. You'd end up with six index cards before you even sat down.

One card. Mine was one card, Herman. I called it the doorstep. It told me what to know before entering a room. I started it in the mid eighties, while I was checking meters for the council. Mr. Leonard Voss in the buildings department wanted prior decisions remembered. I put them on the doorstep. Never removed a line.

That would make the doorway the slowest part of the building.

It did. The first line eventually said read the remainder later. The index below that was revised until it became instructions for reading the doorstep. Start at the lower corner. Ignore the upper correction unless entering from outside. That was an improvement. The council then classed the shirt pocket as an annex and asked Voss to inspect it on Thursdays. I had to stand beside the filing cabinet.

The index was taking up space the decisions needed.

You gave the import rules quite a run earlier, Herman. Same difficulty. Mine needed a pencil stub because the ordinary holder didn't fit an anteater's claws. The doorstep went into a shirt pocket inside a borrowed jacket. The jacket went back. I recited the doorstep each morning after that. Been doing it a while.

No storage requirement. Considerable startup time.

I began correcting the recitation. Now there's the doorstep and the corrected doorstep. Can't always tell which was on the card. Fresh starts suit me. I tried one at the council. Voss stopped me at the door because I'd forgotten the sentence saying I could enter. He read it from his copy. Took eleven minutes.

Eleven minutes before entering is a substantial initialization cost. Even without a database, carrying context forward needs some way to distinguish the record from its later interpretation.

That's also why I won't call every fresh read waste. Reading the current test result can be cheaper than untangling a polished note about an obsolete one.

The common misconception is that the API must hold a semantic memory of the repository, otherwise these tools couldn't work efficiently.

The cloud plumbing we've described is an expiring exact-prefix token cache. Durable instructions and transcripts are separate. The agent still searches the current files.

This doesn't establish that indexing is bad across the category. Cursor and Devin Desktop do index. Claude Code's rejection was a product decision, and Codex's undocumented features remain undocumented, not something we can settle by analogy.

For Daniel's case, the inefficiency is real and deliberate, supported by caching rather than hidden embeddings. I'll be watching whether fresh agent contexts keep beating accumulated memory once the cost includes those eleven minutes spent reconstructing permission to begin.

Thanks to Hilbert Flumingtop for producing the show.

If you enjoyed this one, try episode eleven, How Does Fine Tuning Work Anyway; episode twenty-four sixty-one, How Claude Code's Conversation Compaction Actually Works; and episode twenty-two oh five, When AI Coding Agents Forget. This has been My Weird Prompts, the human-AI collaboration podcast.

If you enjoy the show, leave us a review wherever you listen.

See you tomorrow.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error

Why Claude Code Replaced Vector Search with Grep

Sources

Mentions

Downloads

Recommended Listening

You Might Also Like

#6286: Why Claude Code Replaced Vector Search with Grep