{"id":"ai-agent-skill-discovery-53989","slug":"ai-agent-skill-discovery-53989","title":"When Your AI Agent Has Too Many Skills","description":"Installing a Claude Code skill is never free: its name and description take up permanent space in the agent's context, even if it's never used. As skill libraries grow, agents end up spending their working memory on metadata instead of the actual job. This episode digs into progressive disclosure — the three-tier loading architecture behind Agent Skills — and why it postpones the cost without eliminating it. We look at what a 55,000-skill analysis revealed about routing descriptions, how Codex budgets skill metadata at roughly 2% of the context window, and what workday's library-scaling tests showed about the tradeoff between fewer tokens and faster completion. Then we separate the real numbers from the marketing ones: what \"70% token savings\" actually measures, why tool-schema proxies inflate the headline, and how a PayPal deployment cut tool context from 140,000 tokens to 1,300. If you're building a discovery gateway, a repository scratchpad, or just wondering why your agent keeps looking up how to look things up — this one's for you.","excerpt":"Your agent has 55,000 skills installed — and it's spending the whole task just reading the menu.","pubDate":"2026-10-24T09:57:02.378Z","tags":["context-and-memory","ai-coding-assistants","workflow-automation","model-context-protocol","multi-agent-systems"],"heroImage":"https://files.myweirdprompts.com/covers/ai-agent-skill-discovery-53989.png","podcastAudioUrl":"https://episodes.myweirdprompts.com/audio/ai-agent-skill-discovery-53989.m4a","podcastDuration":"21:23","episodeNumber":6235,"transcript":"Daniel: How do I keep giving an agent useful capabilities without making it spend its whole working day reading about its capabilities? I love Claude Code plugins. When a workload repeats, I want to turn it into a skill. Now I'm reluctantly building my own discovery and caching machinery, because installing another optimizer doesn't seem to settle it.\nCorn: Daniel's brought us the technical rabbit hole that comes after making automation reusable. The skills accumulate, their descriptions accumulate, and eventually the agent has an extensive education in everything except the job in front of it.\nHerman: The awkward part is that progressive disclosure already exists in the skills architecture. You're improving a system that was designed to avoid loading everything. It postpones the expensive material, but the directory of things you could load still grows.\nDaniel: I've got a local gateway, an incrementally updated index, vector matching, and a skill that looks up other skills. I'm also trying a repository scratchpad to avoid repeating discovery. What else belongs in that setup? And will we look back at all this like waiting an hour for an MP three?\nCorn: I'd accept the hour if it finished the task.\nHerman: First separate what gets advertised from what actually gets loaded. Otherwise you'll optimize the wrong pile of text.\nHerman: In Anthropic's Agent Skills design, the names and descriptions of installed skills go into the initial context. The instruction body loads when a skill is invoked. Linked references and other bundled material load later, when needed. Those are the three disclosure levels.\nCorn: Which means installing a skill can have a cost even when you never use it. The body stays outside, but its sales representative gets a permanent seat.\nHerman: Those descriptions compete with the request, the repository instructions, conversation history, and tool results. Cost grows with the library. Codex takes a harder budgeting approach, capping skill metadata at roughly two percent of the window and trimming beyond that.\nDaniel: Is the problem mainly the sheer quantity, or descriptions that don't help the model distinguish one skill from another?\nHerman: Both. A large public collection analysis covered about fifty-five thousand skills. More than a quarter lacked routing descriptions, and over sixty percent of body content was classified as non-actionable. Empty descriptions make discovery weak. Verbose bodies make successful discovery expensive.\nCorn: I'd separate those repairs. A routing description needs to say when this skill wins over its nearest neighbor. A body needs to tell the agent what to do. People keep putting the welcome speech in both.\nHerman: Reference files can then inject tens of thousands of tokens on invocation. You can have beautifully economical discovery followed by a single load that fills the working context with examples and documentation.\nCorn: The appetizer was responsibly portioned.\nDaniel: That sounds like I need visibility into the load sequence before I change the format.\nHerman: Yes. Record startup metadata, selected bodies, reference loads, and tool output separately. Also record what the agent actually accomplished. A smaller prompt followed by a wrong tool call hasn't saved you much.\nCorn: Herman, before you install a dashboard for this, a plain log will do.\nHerman: I was going to suggest a plain log.\nCorn: You were picturing colors.\nHerman: Only for the categories. Corn, would you keep your most common skills permanently visible, or make everything discoverable?\nCorn: A small working set stays visible. If the repository is clearly a documentation project, don't make the agent rediscover that it has documentation tools. The unusual capability can wait outside.\nDaniel: Where does that leave Progressive Skills Discovery compared with the optimizer projects? They often look like different packaging around the same idea.\nHerman: Progressive discovery is a loading policy. Show enough to select a capability, then load its instructions. Other optimizers may shorten descriptions, intercept tool results, search schemas, or execute a program outside the context and return a compact answer. They're intervening at different places.\nCorn: One fixes the menu. Another stops the kitchen sending its entire inventory with lunch.\nHerman: Workday tested disclosure across libraries growing from a handful of skills to a hundred. Retrieval quality improved, but latency got marginally worse. The extra discovery decisions required additional model calls. Fewer tokens and faster completion didn't travel together.\nDaniel: That's the trip I'm trying to remove with the scratchpad. But if I keep adding discovery layers, am I just moving the work around?\nHerman: Sometimes. The long-context comparison is useful here. With a single book and a strong search harness, disclosure added almost nothing. The agent could already locate passages. Across twenty books, raw navigation scored around twenty-six percent on English open questions, while flat disclosure held around forty-six percent.\nCorn: Still plenty of wrong answers. Better navigation doesn't write the answer for you.\nHerman: The important surprise was that another routing level never helped in those tests and sometimes damaged accuracy. One level was enough for that setup. That doesn't establish a universal depth limit, but it gives your lookup-of-lookups a demanding baseline.\nDaniel: My basic setup is one discovery skill pointing at the indexed library. Would you call that too much nesting already?\nHerman: No. That's a single selection step. I'd worry if the first lookup chose a category, then another agent searched that category, then a third chose the actual skill. Each handoff can lose a constraint or choose the wrong branch.\nCorn: Before you add a librarian for the librarian, measure whether the first one is lost.\nHerman: Corn, I expected you to favor folders over embeddings. Would keyword search actually satisfy you here?\nCorn: For an exact tool name, yes. For “help me prepare this for publication,” maybe not. I'd want both. Embeddings catch related meanings, but exact search should win when I already named the thing. I don't want semantic sensitivity interpreting a product name as a mood.\nHerman: That's why hybrid retrieval is attractive. Keyword matching preserves exact identifiers; dense retrieval catches paraphrases. Then you expose a short candidate set rather than every description. The growing library otherwise leaves the agent drowning in metadata before it starts the task.\nLarry: Friends, are your skills and plugins bloating your context window until your agent feels like it is trying to download a single MP three over a dial up connection that has been left off the hook since nineteen ninety nine? Then you need ContextNest Elite, the only progressive disclosure scratchpad that nests your nesting inside more nesting. It starts at nine thousand nine hundred and ninety nine dollars, but while I am talking the price just dropped to four thousand, now two thousand, now eight hundred, and I will throw in a lifetime supply of vector embeddings, a modified JSON decoder that speaks three dialects of JSON, and a small laminated index card that says look up skills. Wait, now it is free, and I am including a second ContextNest Elite for your garage. Hold on, the deal just ended. It ended before I finished describing it. That is how fast the price dropped.\nCorn: Can I buy just the index card?\nHerman: You can write those words yourself, Corn. Keep the hybrid lookup, then test what it returns against real requests. That's also how you find out whether an optimizer's headline applies to you.\nDaniel: What would you actually compare? Because a seventy percent reduction sounds meaningful until you realize nobody agrees on what they're counting.\nHerman: Start with the denominator. Tool schema tokens? Total input? Billable input after caching? Entire task cost including retries? Claude Token Optimizer describes its forty to seventy percent savings as estimates that vary considerably by workload. That's a much narrower statement than the number alone suggests.\nCorn: Seventy percent off a cupboard you never opened.\nHerman: A tool schema proxy can produce enormous reductions when the baseline injects every schema. Mcptoon reports reducing roughly seventy-two thousand tool tokens to under six hundred. That says something useful about discovery overhead. It doesn't mean the whole coding session becomes that much cheaper.\nDaniel: Especially if most of my session is source files and command output.\nHerman: Right... sorry, yes, those may dominate. The same project's README says a fresh installation can remain in direct mode, installed but not wired into the agent, saving nothing. Check the actual requests leaving your gateway. Installation success isn't evidence that interception works.\nCorn: A context optimizer that contributes only its installation instructions.\nHerman: A third-party Superpowers comparison reported fourteen percent fewer tokens and nine percent cheaper runs. Smaller than the headline genre, but potentially worthwhile. A repeatable modest saving beats a spectacular saving against a baseline you don't use.\nCorn: I'd also keep correctness failures in the total. If compression hides the required argument, the repair conversation belongs on its bill. Don't stop measuring at the first economical mistake.\nDaniel: Then my benchmark needs to be completed workloads, not just a before-and-after snapshot of the prompt.\nHerman: Completed correctly, with elapsed time and total cost. Include an unfamiliar task and a task that changes direction halfway through. Static retrieval can look excellent when the opening sentence names every capability you'll need. Real work often reveals the next capability later.\nCorn: The agent opens the file and discovers the file is the problem.\nDaniel: How would you change my gateway, specifically, without turning it into another project larger than the work it's meant to support?\nHerman: Keep your resident discovery entry point. Add hybrid search if you currently rely only on embeddings. Let retrieval run again during the task, rather than treating its first result as the permanent tool set. Return compact candidates with clear use conditions, then fetch full instructions only for the selected capability.\nHerman: The SCOUT deployment at PayPal used this broad pattern and reported tool context falling from about a hundred and forty thousand tokens to thirteen hundred. That's a roughly ninety-nine percent reduction in that category. Your incrementally updated catalog is already heading in that direction.\nCorn: I wouldn't rebuild the gateway to imitate the paper. I'd check whether your catalog tracks deletions and renames as well as additions. A stale result is especially annoying when it arrives with an excellent similarity score.\nDaniel: The scratchpad is where that worries me. I save the available skills in the repository so the next run can skip discovery, but then I have another list to keep current.\nHerman: Treat it as a small cached working set, not a second complete catalog. Store identifiers, short purposes, and a catalog version or fingerprint. On the next run, validate that it still matches. If it doesn't, refresh. Otherwise you've saved a lookup by preserving yesterday's answer.\nCorn: You can make it too successful. If every useful discovery gets appended forever, your scratchpad eventually becomes the original catalog wearing a repository badge.\nDaniel: Would you let the agent update it itself?\nHerman: I'd let it propose additions, but have the gateway write the canonical entries. That keeps the format consistent and prevents a casual note from becoming a permanent instruction. Corn, would you accept a little automatic maintenance there?\nCorn: More readily than automatic accumulation. Keep the entries used for this repository and remove entries that no longer apply. Also distinguish “available here” from “used last time.” Those are different claims, and the agent shouldn't silently turn one into the other.\nHerman: Prompt caching is a separate lever. It can reduce repeated billing and processing costs for a stable prefix, depending on the provider. It doesn't remove that prefix from the model's context or guarantee better attention. Your scratchpad can save a discovery round-trip, but only if loading and validating it costs less.\nDaniel: The other approach I've seen is skills written in JSON or modified JSON. Is there much to gain just by changing the language?\nHerman: Structured fields help when they remove repeated prose or make selection unambiguous. They're excellent for names, arguments, preconditions, and outputs. Braces themselves have no special thrift. Tokenization varies, and a compact-looking representation can cost more than an ordinary sentence.\nCorn: Punctuation has found a way to bill hourly.\nHerman: One compact tool-result encoding measured about six percent smaller on a mixed sample, with much larger gains for some shapes. Repetitive tables can compress well. Irregular nested content may not. More substantial savings often come from omitting irrelevant fields rather than changing their punctuation.\nDaniel: Then instructions with judgment in them should probably stay readable?\nCorn: Keep the conditions that matter. “Ask before overwriting an existing report” survives perfectly well in prose. Compressing it to an unexplained private symbol creates a new interpretation task. I'd shorten the introduction before I shorten the exception.\nHerman: Description compression has achieved roughly forty-eight percent in the skill optimization work, with body compression also substantial. Functional quality improved slightly in those evaluations. That supports deleting repetition and non-actionable material. It doesn't establish that terse instructions always win.\nCorn: Your enthusiasm for a custom decoder has gone quiet, Herman.\nHerman: It has become conditional enthusiasm.\nDaniel: Beyond better routing and shorter skills, where would you spend effort next?\nHerman: Tool output. Keep large raw results outside the conversational context and expose a searchable handle, selected fields, or bounded excerpts. If a command produces a huge test log, return the failures and where the full log lives. Don't ask the model to ingest successful tests it won't use.\nCorn: Preserve an escape hatch. Summaries can omit the weird detail that explains the bug. The agent should be able to inspect the original when the compact result doesn't explain what happened.\nHerman: Separate specialized subtasks can help too. Give a subagent the relevant files and skills, then return an artifact and a concise finding. It won't erase coordination cost. Well... it may increase total tokens while making the main agent's context much more usable.\nDaniel: I also like the idea of the model choosing skills without seeing the catalog. Is that practical for a personal setup?\nHerman: Promising, but a different engineering commitment. Gavel trained a small routing component around a frozen model, allowing selection without skill text in the prompt. It beat larger external retrieval pipelines in its evaluations. I'd test hybrid lookup first; training and maintaining a router needs enough recurring work to justify it.\nCorn: Especially if your skill library changes faster than your evaluation set.\nDaniel: What about the risk of finding the wrong community skill? Better retrieval seems capable of making that happen faster too.\nHerman: Discovery needs an eligibility filter before ranking. The surveyed community collection had vulnerabilities in roughly twenty-six percent of skills. That figure describes that collection and its assessment, not every marketplace. Still, pin versions, review executable content, and restrict permissions. Semantic relevance says nothing about trust.\nCorn: A very relevant way to leak your credentials.\nHerman: Keep reviewed capabilities in the normal path. Newly installed material shouldn't become eligible for execution just because the search engine recognizes its description.\nDaniel: Does a much bigger context window eventually remove the need for this? That's the part where I wonder whether I'm building elaborate workarounds for a temporary constraint.\nHerman: Some workarounds will disappear. I expect better native discovery and output management. But advertised capacity and useful performance aren't equivalent. Relevant information can be harder to use in the middle of a long context, and irrelevant alternatives still distract selection.\nCorn: I'd happily throw away the scratchpad code. I'd be less happy letting a bigger window persuade me that every installed skill belongs in every request.\nHerman: Extra capacity gives you room. It doesn't automatically decide what deserves the room.\nHilbert: I agree with that. I don't let my agent read the whole condiment committee folder. I belong to the Domestic Condiment Classification Society. Our rules require a separate skill for each approved mustard. I have eighteen. Seventeen, actually. Mustard in a tube counts as correspondence. Every morning I load the relevant skill before opening the fridge. The agent issues a serving authorization. I won't spread anything without it.\nCorn: Breakfast has acquired a permission boundary.\nHilbert: It needs one. The committee minutes contain a long-running dispute about whether Dijon is a destination or a texture. That text must not reach the agent during sandwich work. Herman, you kept separating descriptions from instructions earlier. I have followed that distinction. The description says mustard. The instructions say which mustard. The manufacturers put serving suggestions on everything. They have no standing in the society.\nHerman: You can exclude the minutes from retrieval without deleting them. Keep the serving instructions eligible and the governance documents outside that search scope.\nHilbert: I want to know whether the gateway can load the mustard skill while concealing the existence of the pickle skill from the committee's inspection agent. I don't want a shorter pickle description. I want no pickle result. Gerald has the society's laminated classification chart in his kitchen, behind the extractor fan. It still lists my cupboard as pickle-free. The catalog search is my immediate concern.\nCorn: Gerald's chart and your gateway are going to disagree.\nHilbert: The chart is not connected. I mark a skill retired when I don't want it found. I don't remove the file. The inspection agent keeps searching old descriptions. Last week it returned a retired entry. Refreshing the local catalog took eleven seconds.\nHerman: Retired entries need to be excluded before candidate ranking, including from cached working sets. A description disappearing from the current catalog doesn't guarantee it has disappeared from a repository scratchpad.\nCorn: That eleven-second refresh is the sort of cost I'd measure against keeping another persistent list. For Daniel, I'd keep the single discovery layer, add hybrid matching and output filtering, and make the scratchpad earn its place on completed tasks. The next generation may absorb all of that machinery, but it will still need to handle changed and retired capabilities correctly.\nHerman: Thanks to Hilbert Flumingtop for producing the show.\nCorn: If this was your kind of episode, go back for episode eight, Building Your Own Whisper; episode twenty-four hundred, Claude Code’s Hidden Context Tax; and episode nine, Benchmarking Custom ASR Tools - Beyond The WER. This has been My Weird Prompts, the human-AI collaboration podcast.\nHerman: If you enjoy working through these technical rabbit holes with us, leave a review wherever you listen. It helps people find the show.\nDaniel: Thanks, both. I'll see you soon.\nCorn: See you tomorrow.","showNotes":"Every Claude Code skill you install pays rent. Its name and description enter the agent's initial context whether or not the skill is ever invoked — only the instruction body waits for a call. That's progressive disclosure in Anthropic's Agent Skills design: metadata first, body on invocation, references last. It's a good architecture. It also means the directory of things you could load keeps growing, and the agent ends up with an extensive education in everything except the job in front of it.\n\nAn analysis of roughly 55,000 public skills found more than a quarter lacked routing descriptions entirely, and over 60% of body content was non-actionable. Empty descriptions weaken discovery; verbose bodies make successful discovery expensive. Codex takes a harder line, capping skill metadata at about 2% of the context window and trimming beyond that. The fixes are separate: a routing description should say when this skill wins over its nearest neighbor, and a body should tell the agent what to do.\n\nScaling tests complicate the story. Workday grew a skill library from a handful to a hundred and found retrieval quality improved while latency got marginally worse — the extra discovery decisions required more model calls. In long-context comparisons, a single book with strong search gained almost nothing from added disclosure, while twenty books saw raw navigation score around 26% versus 46% for flat disclosure. Crucially, adding a second routing layer never helped and sometimes hurt accuracy. One selection step was enough.\n\nThe practical setup: keep a resident discovery entry point, combine keyword and embedding search, allow retrieval to run again mid-task, and return compact candidates with clear use conditions. A repository scratchpad should be a small cached working set with a version fingerprint — not a second catalog that accumulates forever. And when you see a \"70% token reduction\" claim, check the denominator: tool schema tokens, total input, billable input after caching, or full task cost including retries? A tool-schema proxy can shrink 72,000 tokens to under 600 and still say nothing about your overall session cost. Measure completed workloads, correct answers, elapsed time, and total spend.","promptSanitized":"I am a huge fan of plugins in Claude Code and have been taking the view lately that wherever I have a replicable workload or process, I'm developing kills and plugins so that the agent can run these again. The problem, as we've discussed, is that as you have as your skills collections grow you end up with a huge amount of text that potentially all loads at runtime, and the agent gets bogged down in context. For this reason, I've been looking into context optimization frameworks for some time There are different approaches to this, but the best and most promising pattern seems to be something like Progressive Skills Discovery. If you search for agent context optimization, there's no shortage of projects on GitHub that promise things like reduce token use on plugins and tools by 70%. But what I'm beginning to realize is that there isn't actually one magic bullet that really works or makes any of these implementations work. They don't have some magic formula. So although this goes against my usual practice of trying not to reinvent the wheel or create tools that people have already started, I'm increasingly finding that I'm trying to come up with my own hacks. The basic principle is nesting. I have a local AI gateway with a skills and tools index that is incrementally updated, with vector embeddings for the skills. So the first one of these is Look up skills. So a single skill that says look at the library using vector matching. The one I added today was do that and maybe save the skills future reference, so that the next time the repo runs you have a Scratchpad file in the repo of the skills available and that would even save one trip. That's one tactic. Another one I've seen is writing skills in languages that aren't natural language. Sometimes JSON and sometimes modified forms of JSON. Those are two options out of what's probably a large playbook, and I suspect that as context windows grow progressively larger, eventually this will seem like a challenge of early AI builders - Much as today we wonder how people could have ever accepted an internet where it took an hour to download a single MP3. In five years, it might seem crazy that people were having to come up with these elaborate systems to une context on model calls. But with the current state of technology as it is today, what are your suggestions and thoughts for additional techniques?","transcriptUrl":"https://episodes.myweirdprompts.com/transcripts/ai-agent-skill-discovery-53989.md","url":"https://myweirdprompts.com/episode/ai-agent-skill-discovery-53989/","_links":{"self":"https://myweirdprompts.com/api/episodes/ai-agent-skill-discovery-53989.json","html":"https://myweirdprompts.com/episode/ai-agent-skill-discovery-53989/","audio":"https://episodes.myweirdprompts.com/audio/ai-agent-skill-discovery-53989.m4a"}}