220,000 prompt tokens a turn: we stopped sending every MCP tool to the model

Our first real customer deployment exposed 84 tools, generated from a GraphQL API. The assistant worked. Then I looked at the token cost of a single conversation and found turns costing 220,000 prompt tokens. Nobody had asked a 220,000-token question. The visitor had typed one line, and we were re-sending a tool catalogue the size of a short book on every round of every turn.

This is the part of MCP that nothing warns you about. A tool is not free to have. Its name, description and full JSON Schema go into the request on every round, whether or not it ever gets called. Here is what we measured, what we changed, and the three ways the change broke before it worked.

Where 220,000 tokens actually go

An agent turn is not one request. The model proposes a call, you execute it, you send the result back, and it proposes the next one. Each of those rounds carries the whole tool list again. A 25-round session with 84 tools bound pays for those schemas 25 times.

Tool schemas also compound badly when the tools are generated. A hand-written tool has a two-line description and four parameters. A tool generated from a GraphQL mutation can carry a hundred-field input object with every field documented, which is exactly what you want when the model calls it and exactly what you do not want when it does not.

Retrieval, applied to tools instead of documents

The fix is the one everybody already uses for knowledge: embed the corpus, embed the query, send the top matches. Only here the corpus is the tool list.

We embed the name and description of every tool on every connected server, keyed by a hash of that text so nothing is re-embedded until a description changes. Rows are deleted when a tool disappears from the source, and a 10-minute memo per server stops the check from running on every message.

export function toolEmbeddingText(name: string, description?: string | null): string {
  const desc = String(description ?? "").trim();
  return `${name}\n${desc}`.slice(0, 4000);
}

export function diffToolEmbeddings(desired, existing) {
  const existingByName = new Map(existing.map((r) => [r.tool_name, r.text_hash]));
  const desiredNames = new Set(desired.map((d) => d.name));
  const toEmbed = desired.filter((d) => existingByName.get(d.name) !== d.hash);
  const stale = existing.map((r) => r.tool_name).filter((n) => !desiredNames.has(n));
  return { toEmbed, stale, unchanged: desired.length - toEmbed.length };
}

At request time, if more than 24 tools are bound, the visitor's message is embedded and matched against those vectors, and the top 20 survive into the prompt. Below 24 we skip the whole thing, because the embedding round trip costs more than the schemas it would save. Schema tokens per round fell by about 75% on the 84-tool deployment, with no change to what the assistant could do.

Break 1: the query is two words long

Retrieval quality depends on the query, and chat hands you terrible queries. "Yes, do it." "How many?" "The second one." Embedding those against a tool corpus returns noise, so the turn that most needs the right tool is the turn most likely to lose it.

Two rules fixed it. If the last user message has fewer than four words and an earlier one exists, the previous message is prepended to form the query. And when the turn is a reply to a clarifying question the assistant asked, the query is built from the original request plus the answer, never the answer alone. "Acme" is a useless retrieval query. "Create a draft post for Acme" is not.

Break 2: a tool that worked in turn 2 vanished in turn 3

Pure top-20 selection is stateless, and a conversation is not. The visitor asks for posts, gets them, then says "publish the first one". The publish tool did not rank in the top 20 for that four-word message, so it was gone, and the model did the worst available thing: it answered as if it had published.

Three pin categories go in before ranking, and none of them are optional:

• Tools belonging to the assistant's configured use cases. If an operator said this assistant files bugs, the bug tools are present on every turn.
• Identity tools (me, whoami, viewer, get_current_user). Without one the assistant cannot establish whose data it is reading.
• Every tool already called in this conversation, read from the last 12 tool rows. A tool that ran once is very likely needed again.

The failure mode is worth stating plainly, because it connects to a different bug we wrote about: a turn with no suitable tool bound does not reliably say so. It narrates success. We now treat "zero tools in the prompt" as a state that has to be justified, and the write-claim guard is the second net under it.

Break 3: cosine similarity prefers the catalogue to the counter

A visitor asked how many posts were published. The ranking put statuses at 0.62 and get_post_status at 0.60. Both read well against that question, and the one that merely lists the possible status values won by two hundredths. The model took it, then paginated through posts to count them by hand, spending calls to produce a number another tool already had.

Embeddings rank topical closeness, not usefulness for the shape of the question. So the ranking takes a small intent-aware nudge. On a turn the router already classified as a count question, names that promise an aggregate gain 0.05, names that look like a catalogue lose 0.05, and a description that promises a count gains half the boost.

export const COUNT_TOOL_BOOST = 0.05;

export function countToolBoost(c) {
  const name = String(c.originalName ?? "");
  const desc = String(c.description ?? "");
  if (CATALOGUE_TOOL_NAME_RE.test(name)) return -COUNT_TOOL_BOOST;
  if (COUNT_TOOL_NAME_RE.test(name)) return COUNT_TOOL_BOOST;
  if (COUNT_TOOL_DESC_RE.test(desc)) return COUNT_TOOL_BOOST / 2;
  return 0;
}

0.05 is not a magic constant. It is slightly larger than the gaps we were seeing between a catalogue and its matching aggregate, and small enough that it cannot drag an unrelated tool into the set. If your tool names are less uniform than ours, your number will be different. Measure the gap before picking one.

Selection has to fail open, and say so

Retrieval now sits in front of every tool call, which makes it a fresh way for the product to go silently dumb. An embedding provider having a slow minute must never become "the assistant cannot do anything today". Every path that is not a confident match returns the full tool set: no candidates, fewer candidates than the cap, an empty query, no servers, no matches, a thrown error, or a selection that takes longer than 4 seconds.

if (!Array.isArray(candidates) || candidates.length === 0) return all("no_candidates");
if (candidates.length <= maxTools) return all("under_cap");
const query = String(input.query ?? "").trim();
if (!query) return all("empty_query");

Failing open costs tokens, which is the correct direction to fail. The reason is logged once per worker and the mode is recorded per turn, so a week of expensive turns shows up as all:error in the stats instead of as a mysterious bill.

If you host tools for someone else

• Count prompt tokens per round, not per conversation. What matters is schema cost multiplied by rounds, and that is invisible in a per-conversation average.
• Give operators an allowlist per assistant before you give them retrieval. A support assistant that binds 12 tools never reaches the threshold, and no selection beats good selection.
• Treat the pinned set as a product decision. Identity and recently-used tools are not an optimisation detail, they are what keeps a multi-turn conversation coherent.
• Log the selection mode with the candidate and selected counts on every turn. We would not have found the catalogue-versus-counter bug without being able to ask which tools were in the prompt when the wrong one ran.

Tool selection is one half of a budget. What the tools send back has to fit as well, which we wrote about in the evidence budget post. Both run for every assistant hosted on Command+K, including the 84-tool deployment the 220,000-token turn came from.