How do I prevent an AI agent from taking actions on the wrong account?

Short answer: never let the account be a value the model fills in. Derive it on the server from the credential the request arrived with, resolve it once per turn, freeze it into any call a human approves, and treat a permission denial as a wrong-id signal instead of a sign-out. Every wrong-account bug I have shipped came from breaking one of those four rules, and none of them were fixed by better prompting.

I run the assistant that ships with Command+K, so this is not a thought experiment. The agent performs real writes against real customer APIs on behalf of a signed-in end user. Here is what actually went wrong and what the fixes look like in code.

Identity has to be known by construction

The first version of almost every agent integration has an account_id or workspace parameter on the tools. It looks harmless. It is the bug. The moment an id is a parameter, choosing it is a language-model decision, and a language-model decision made from a page full of ids is a coin flip with good manners.

What we do instead: the gateway resolves the caller from the signed token, and the answer to "do we know who this is" is a boolean the rest of the turn reads.

export function isIdentityKnownByConstruction(
  claims: { sub?: unknown } | null | undefined,
  endUserCtx: { identity?: unknown } | null | undefined,
): boolean {
  if (!endUserCtx || !endUserCtx.identity) return false;
  const sub = claims?.sub;
  if (typeof sub !== "string" || sub.length === 0) return false;
  return !sub.startsWith("anon:");
}

Two things matter here. The identity comes from claims.sub, which is the subject of a token the host signed, not something the visitor or the model can type. And an anonymous visitor is not a weak identity that we treat optimistically, it is a different state entirely: the anon: prefix makes it false, so nothing downstream can mistake a guessed id for a verified one.

A cached identity must never cross a subject boundary

Calling me on every turn costs a full gateway round trip, so we cache the result in the conversation. That optimisation is where a wrong-account bug hides, because conversations are durable. They get reopened, shared, resumed from a different device, and in our case replayed from history to restore approval state. Cache against the conversation and the second reader inherits the first reader's ids.

The cache therefore refuses to answer unless the subject on the current request is the same subject the cached answer was recorded for.

export function shouldServeIdentityFromCache(input: {
  entry: IdentityCacheEntry | null | undefined;
  convSub: unknown;
  claimsSub: unknown;
  originalName: string;
  args: unknown;
  now?: number;
}): boolean {
  const { entry } = input;
  if (!entry || entry.result == null) return false;
  if (typeof input.convSub !== "string" || typeof input.claimsSub !== "string" || !input.claimsSub)
    return false;
  if (input.convSub !== input.claimsSub) return false;
  // ... same identity tool, same identity field, empty args, resolved result
  if (!entry.createdAt) return false;
  const at = Date.parse(entry.createdAt);
  if (!Number.isFinite(at)) return false;
  const now = input.now ?? Date.now();
  if (now - at < 0 || now - at >= IDENTITY_CACHE_MAX_AGE_MS) return false;
  return true;
}

Read the failure branches, not the success branch. A missing claimsSub is a refusal. A mismatch is a refusal. An unparseable timestamp is a refusal, and so is a timestamp in the future, because a clock that runs backwards is not a reason to trust stale identity. IDENTITY_CACHE_MAX_AGE_MS is 24 hours. Anything older goes back to the gateway.

The general shape: a cache that holds identity is an authorization decision with a TTL. It has to be keyed on the credential, and it has to fail closed. Ours costs one extra round trip when it refuses, which is the cheapest possible price for not acting as the wrong person.

Ids that come from the page are not identity

Our widget reads the host page so it can answer questions about what the user is looking at. That context is genuinely useful and it is also full of plausible-looking identifiers: a project id in the URL, a customer id in a table row, an account slug in the breadcrumb. An agent that has just been told "the user is asking about this screen" will happily use the id on that screen as the id to act on.

When an identity tool resolves, we inject one line that outranks everything else in the context window.

export const IDENTITY_OVERRIDE_LINE =
  "IDENTITY: use the ids in this result (user id, active project id) for every later call; " +
  "ignore ids from earlier turns or from the page.";

This is the one place a prompt line earns its keep, and it is worth being precise about why. It is not protecting the account boundary, the credential already does that. It is resolving an ambiguity inside one account: which of several ids the user meant. Two sources get named explicitly as losers, earlier turns and the page, because those were the two sources we watched the model actually pull from.

An approval has to execute the call it approved

This is the subtlest one, and the one I would not have predicted. Writes go through a human approval card: the agent proposes, the user sees exactly what will happen, the user says go ahead. The obvious implementation is to re-run the model after the approval so it can carry on. That implementation is broken, because the second plan is a new plan. Same intent, possibly different arguments, possibly a different target.

So the approval does not resume the agent, it replays the frozen calls that were recorded when the card was shown.

export function planResume(
  approvedCalls: unknown,
  resolve: (name: string | undefined) => { originalName: string; isConnectPlaceholder?: boolean } | undefined,
  opts?: { max?: number },
): ResumeCall[] {
  if (!Array.isArray(approvedCalls)) return [];
  const out: ResumeCall[] = [];
  const seen = new Set<string>();
  for (const raw of approvedCalls as ApprovedCallEcho[]) {
    const args = raw.args;
    if (!args || typeof args !== "object" || Array.isArray(args)) continue;
    const name = typeof raw.name === "string" && raw.name ? raw.name : undefined;
    const original = typeof raw.original === "string" && raw.original ? raw.original : undefined;
    const routedName =
      name && resolve(name) ? name : original && resolve(original) ? original : undefined;
    if (!routedName || seen.has(routedName)) continue;
    seen.add(routedName);
    out.push({
      id: `approved_${out.length}_${Date.now().toString(36)}`,
      type: "function",
      function: { name: routedName, arguments: JSON.stringify(args) },
      original: resolve(routedName)!.originalName,
    });
  }
  return out;
}

The arguments are re-serialised from what was recorded, not regenerated. An unroutable name is skipped rather than guessed at. Duplicates are collapsed, because one "yes" is one execution and a retry loop that turns it into two is its own kind of wrong action. The approval card and the executed call are the same object, and that is the property you want to be able to state in a sentence when a customer asks.

A 403 is a wrong-id signal, not a signed-out signal

Our first error classifier treated every authorization failure the same way and told the user to reconnect. That was wrong in the exact case that matters. If identity is known by construction, the user is signed in, and a denial means the agent reached for something that is not theirs. The reconnect flow then fixes nothing and the next attempt reuses the same bad id.

if (kind === "permission" && ctx?.identityKnown) {
  return {
    code: "permission_denied",
    message: `The connected tool denied access to that resource (not a sign-in problem) — ${PERMISSION_WRONG_ID_HINT}.`,
    retryable: false,
  };
}

The hint tells the model to go back to the ids from the identity result, and retryable: false stops it from hammering the same call. Split 401 from 403 in your classifier and the difference between "prove who you are" and "you asked for the wrong thing" stops being lost.

The checklist

1. No account, tenant or workspace id is a model-supplied tool parameter. The gateway derives it from the credential.
2. Anonymous is its own state, not a weak version of signed in. Nothing that acts runs without a verified subject.
3. Any identity cache is keyed on the credential subject, has a hard TTL, and fails closed on anything it cannot verify.
4. Approved calls execute the recorded arguments. Nothing between the approval and the execution is allowed to re-plan.
5. 403 with a known identity routes to a wrong-id message, not a reconnect, and is not retried.
6. Every executed call is logged with the subject it ran as, so "which account did that happen on" is a query and not an investigation.

If you want the longer version of the identity model, the per-customer identity post covers how the credential gets bound in the first place, and connection modes documents which of them can carry an end-user identity at all. Every mechanism above is on by default for assistants hosted on Command+K, which is the only reason I can describe them this precisely: we found each one by getting it wrong first.