MCP production secrets through brokers or environment variables

An MCP agent should receive production credentials through a broker that issues a narrowly scoped, short-lived credential for one approved action. Direct environment injection is acceptable only when the agent runs as a single-purpose process, the injected credential is equally narrow and short lived, and no untrusted tool or child process can inspect its environment.

That answer sounds like a vote for Vault, but the product name is not the control. A badly designed broker can hand the agent a reusable Vault token with broad read access, while a disciplined workload launcher can inject a five-minute database credential into an isolated process. Compare the authority that reaches the agent, how long it remains useful, and where its value can appear. The delivery mechanism comes after those questions.

Secret delivery is separate from MCP authorization

MCP authorization protects the connection between an MCP client and an MCP server; it does not automatically protect the credentials that the server uses to call a database, cloud API, payment provider, or deployment system. Treat these as two credential planes with separate issuers, audiences, lifetimes, and audit records.

The Model Context Protocol authorization specification says an HTTP MCP server must validate that an access token was issued for that server. It also forbids token passthrough: a server must not take the token it received from the MCP client and forward it to an upstream API. That rule draws a useful boundary. The inbound token proves who may call the MCP server. A different downstream credential authorizes what the server may do after it accepts the call.

Local stdio servers are different. The specification says stdio implementations should retrieve credentials from the environment instead of using the HTTP authorization flow. This is transport guidance, not permission to fill the process environment with production administrator tokens. A local server still runs with the client's operating-system privileges, and the MCP security guidance warns that a compromised local server can access files, networks, and other resources available to that process.

Write the two planes down before choosing a delivery pattern:

If one bearer token crosses both rows, the design has collapsed two trust decisions into one. Vault cannot repair that after the fact. The server must exchange or request a credential meant for the downstream resource, and the audit system must retain the link between the inbound identity and the downstream action without copying either secret.

Environment injection widens the process boundary

An environment variable is easy to deploy and hard to contain because every component inside the process boundary may be able to read it. The model does not need a tool named read_secret if a shell tool, diagnostic endpoint, package hook, crash reporter, or child process can expose the environment on its behalf.

Injection usually happens before the process starts. A scheduler, CI runner, container platform, or Vault Agent obtains a value and places it in DB_PASSWORD, CLOUD_TOKEN, or a similar variable. Application code reads it through the standard runtime API. There is no network trip to a broker during the task, which reduces moving parts and avoids a dependency in the request path.

The cost appears during rotation and isolation. Most runtimes snapshot the environment at process creation, so changing the scheduler's secret does not update a running child. The operator must restart or supervise the process, then prove that old replicas stopped. A long-running MCP server may keep yesterday's credential until its next deployment even though the control plane shows a new value.

Environment variables also spread through ordinary operational behavior. Debug handlers print configuration. Test frameworks include the environment in failure bundles. Process supervisors expose launch settings. A tool runs env while diagnosing a path problem. None of those actions looks like a secret read at the secret manager, so the secret manager's audit trail cannot explain the disclosure.

Do not overstate the operating-system issue. On a well-configured host, an unrelated unprivileged process should not freely read another process's environment. The practical concern is the set of processes, tools, extensions, and support paths that intentionally share the same identity or runtime. Agent systems make that set larger because their purpose is to invoke tools chosen at runtime.

Direct injection can still be the smaller risk for a tiny worker that performs one operation and exits. Give that worker a credential created for its exact role, keep the lifetime shorter than the maximum job time plus a small grace period, disable shell and diagnostic tools, and destroy the execution sandbox after the job. In that case, injection is a transport detail inside an already narrow boundary.

A broker is safe only when it holds the parent authority

A broker improves the design when the agent asks for an operation and the broker returns limited authority, or performs the operation itself. It adds little when the agent receives a broad Vault token and can choose arbitrary secret paths.

The strongest pattern is a capability broker. The agent submits a structured request such as deploy release 184 to staging. The broker checks the authenticated workload, user approval, target environment, tool name, and policy. It then calls the deployment service without returning the production credential to the MCP process. The tool response contains a deployment ID and status, not an access token.

A credential broker is the next best option when the downstream client must hold a credential. It authenticates the workload using platform identity, requests a dynamic credential for one Vault role, and returns it through a private channel to a constrained executor. The language-model context and general MCP tool dispatcher never receive the value. The executor sends it only to the intended service and discards it when the call finishes.

A secret lookup proxy is weaker. If the agent can say read secret/data/anything and holds a token that authorizes many paths, the proxy has moved the API endpoint but preserved broad authority. Path allowlists help, yet a static database administrator password retrieved through an allowlist remains a reusable administrator password.

The parent credential matters most. A broker should authenticate with workload identity or keep its Vault token in a separate process that the agent cannot inspect. Never inject a Vault token into the MCP server merely so the server can fetch other secrets. That token is a map to all paths allowed by its policy, often for longer than any downstream credential. One prompt injection or verbose exception can turn a narrow task into a secret inventory.

Vault response wrapping helps during handoff. HashiCorp's documentation explains that a wrapped response contains a single-use wrapping token instead of the original secret, with a separate and often shorter lifetime. The receiver can check the creation path before unwrapping. If someone already unwrapped the token, the intended receiver sees a failure that deserves investigation. Wrapping reduces exposure while a value moves, but the unwrapped credential still needs a narrow role and short lease.

A broker becomes production infrastructure, so define its failure behavior before agents depend on it. If Vault is unreachable, the broker should reject new privileged operations and preserve enough non-secret context for a retry. It should not fall back to a static administrator token hidden in configuration. That fallback gives an attacker the strongest credential during the period when monitoring and operator attention may already be degraded.

Cache policy deserves the same precision. Caching a Vault response in the MCP process defeats separation. A broker may cache its own renewable authentication token, and an executor may reuse a downstream credential within one bounded job if the service requires several calls. Neither should return cached material to a different user, task, target, or approval. Include those dimensions in the cache key and erase an entry when its job closes, its lease expires, or policy changes.

Keep the broker interface smaller than the secret manager interface. Methods such as issue_orders_reader(task_id, approval_id) and deploy_release(target, digest) give policy code stable objects to validate. A generic get_secret(path) method exports Vault's namespace into the agent and encourages callers to assemble authority from strings. That convenience is exactly what an attacker needs after a prompt injection.

The broker must also authenticate the caller without another permanent secret. Prefer a platform workload identity bound to the running service account, task, or machine. Validate its audience and deployment context, then map it to a broker policy. If the platform cannot provide workload identity, bootstrap with a single-use wrapped credential delivered by the scheduler and exchange it immediately. Do not solve secret delivery by injecting a long-lived broker password.

Short lifetimes must match real task boundaries

A short TTL limits the time available to abuse a leaked credential; it does not reduce what the credential can do during that time. A five-minute organization administrator token is still an organization administrator token.

Match the lease to an operation that the system can name and observe. For a database migration, issue a database identity limited to the target schema and migration commands. For an object export, limit the credential to one bucket prefix and read actions. For a deployment, bind it to one environment and the required release operations. The agent's general ability to reason does not justify general production authority.

HashiCorp distinguishes dynamic secrets from values stored in its KV engine. Dynamic secrets have leases and can be revoked by Vault; static KV values do not gain that property merely because an agent fetched them from Vault. This distinction is routinely blurred in architecture diagrams. Moving a year-long vendor token from a CI setting into Vault KV centralizes storage and access logging, but the vendor still accepts the copied token until someone rotates it at the vendor.

Set TTL from measured task duration rather than a slogan. A credential that expires midway through a transaction can leave a migration or deployment in an uncertain state. The executor should know whether it may renew, and renewal policy should cap the total lifetime. HashiCorp notes that a requested renewal increment is advisory and that clients must inspect the returned lease rather than assume the requested duration was granted.

Use a policy that denies browsing and fixes the role path. This fragment lets a broker request one database role while preventing it from listing neighboring roles:

path \"database/creds/mcp-orders-reader\" {
  capabilities = [\"read\"]
}

path \"database/roles/*\" {
  capabilities = [\"deny\"]
}

path \"secret/data/*\" {
  capabilities = [\"deny\"]
}

The policy is intentionally boring. Do not add list for operator convenience to the runtime identity. Put troubleshooting access behind a separate human role with its own approval and audit record.

Revocation includes the downstream system

Revocation works only when the issuer can make the downstream service reject the credential and the application stops using cached copies. Deleting an environment variable, revoking a Vault token, or killing an MCP session may leave the actual production credential valid.

Vault associates leases with dynamic secrets. Its lease documentation says revoking a lease invalidates that secret and prevents renewal; revoking a token also revokes leases created by that token. Some secret engines remove the downstream identity immediately. This gives operators a precise response path that static injection cannot provide on its own.

There are failure modes behind that promise. Vault may be unable to contact the database or cloud service when it tries to remove an identity. HashiCorp documents irrevocable leases for this condition. Your incident procedure must check the downstream system, not stop when vault lease revoke returns or when the lease disappears from a dashboard.

For injected static credentials, revocation means rotating at the authoritative provider, updating the source, restarting every consumer, and rejecting the old value. If the provider supports two active credentials during rotation, define which one is old and set a deadline for its removal. If it supports only one, plan for connection failures and retries rather than quietly extending the credential's life forever.

Revocation speed is an end-to-end property. Measure the time from an incident decision until a test request using the exposed value fails at the downstream service. Include broker caches, connection pools, queued jobs, warm workers, and replicas in that measurement. A thirty-second Vault lease means little if a pool keeps authenticated database sessions alive for an hour.

An audit trail needs causality, not secret values

A useful audit trail can answer who asked, what policy allowed, which credential or lease was issued, which production resource changed, and whether the action succeeded. It should answer those questions without storing the bearer value.

Vault audit devices record requests and responses that pass through Vault. That creates evidence for authentication, secret reads, lease creation, renewal, and revocation. It does not record every SQL statement or cloud API call made afterward. Downstream service logs remain necessary, and they become far more useful when dynamic credentials create a distinct identity for each workload or task.

Carry one correlation ID across the boundary. Create it when the MCP server accepts the tool call, then attach it to the broker request, approval record, executor log, and downstream request metadata where the service supports it. Record the inbound user or workload identity separately from the Vault entity and downstream principal. Those identities describe different decisions and should not overwrite each other in a single actor field.

Avoid an audit design that records complete MCP arguments and responses by default. Tool payloads often contain user data, query results, headers, and error bodies. Store an operation type, resource identifier, policy decision, response class, byte count, and hashes of stable non-secret identifiers. Put full payload capture behind a temporary, approved diagnostic mode with a retention limit.

The distinction between an access log and a causality trail matters during an incident. An access log may prove that mcp-prod-role read database/creds/orders. A causality trail connects that read to user 482's approved get_monthly_total call, broker decision 9137, database principal v_mcp_7f2, and the exact query class. Without the connection, investigators see activity but cannot decide whether it was expected.

Logs and tool responses are the likely leak path

Most agent secret exposures do not require breaking Vault. They happen after a legitimate read, when a value enters a prompt, tool result, exception, trace, or support bundle.

Assume every string returned to the MCP layer may reach model context and observability systems. A database tool should return rows or an error category, not its connection string. An HTTP tool should remove authorization and cookie headers before it constructs either a success result or an error. A deployment tool should return a release ID, target, and state. If a tool needs to mention a credential, it should use a non-secret accessor, lease ID, or last four characters chosen specifically for display.

Redaction by field name is too weak. Secrets appear inside nested JSON, command lines, URLs, stack traces, and vendor error text. Redact known values at the executor boundary before serialization, limit response size, and treat high-entropy detection as a secondary alarm rather than the main control. Detection misses short passwords and flags harmless identifiers.

Run one controlled leak drill before production. Issue a canary credential that has no useful authority, then trigger each path below and search every destination where data may land:

1. Make the downstream service reject the credential and return a verbose error.
2. Crash the executor while it constructs the client.
3. Ask the diagnostic tool for environment and process information.
4. Return an oversized nested tool response that contains the canary at several depths.
5. Open the tracing, log aggregation, model transcript, support export, and dead-letter queue records.

The test passes only when the canary value is absent and the records still contain enough identifiers to investigate the failure. Redaction that deletes the correlation ID, role, and operation makes the system quiet but unaccountable.

Prompt injection changes who chooses the tool call, not the operating system's secret rules. Treat model output as untrusted input to policy. The broker should validate enumerated operations, typed resources, approval state, and workload identity. It should reject free-form secret paths and arbitrary upstream URLs even if the model presents them confidently.

Approval belongs at the authority boundary

Human approval reduces risk only when the approved statement matches the credential and operation that the executor will use. A generic confirmation such as Allow this tool? gives the approver too little information and gives the implementation too much room to substitute a different target after the click.

The broker should build the approval request from validated fields, not from prose produced by the model. Show the operation, production account or environment, resource, effect, requesting identity, and expiry. For a deployment, that may mean release 184, production, service billing-api, change window 9137, requested by user 482, valid for one execution within ten minutes. The approver should not need to inspect a secret path, and the approval record should never contain the secret.

Bind the decision to a digest of the normalized request. The executor recalculates that digest before it asks for a credential or performs the operation. If the model, tool dispatcher, or queue changes the target after approval, the digest no longer matches and the broker rejects the request. Binding prevents a common time-of-check and time-of-use failure in which a harmless staging request wins approval, then mutable arguments point the executor at production.

Approvals also need consumption rules. A one-time approval should become unusable after the matching action starts, whether the action succeeds or fails. A retry policy may allow a small, explicit number of attempts for the same normalized request, but it must not turn one approval into an open session. Store the retry count and final state beside the original decision.

Do not require a human click for every read. That pattern trains people to approve prompts they cannot evaluate and creates pressure to bypass the broker. Divide operations by consequence. Low-risk, read-only calls can run under fixed policy and data filters. Expensive reads, exports, writes, permission changes, production deployments, and secret administration deserve stronger checks or human approval. The credential role must reflect the same division; an approval screen cannot compensate for a role that allows unrelated actions.

Separate approval authority from runtime authority. The person who may approve a production deployment does not need permission to read the Vault path, and the broker identity that reads the dynamic role does not need permission to approve its own request. For a small company, those functions may run in one service, but keep distinct policy decisions and log fields so growth does not require redesigning the protocol.

Failure handling needs an explicit rule. If the approval service is unavailable, privileged calls should stop. Falling back to a broad injected credential turns an availability incident into a security exception precisely when operators are distracted. Read-only operations with preapproved policy may continue if their credentials and data filters remain valid, but the system should record that it used the degraded path.

Test approval binding with substitution, replay, and delay. Change the resource after approval, submit the same approval twice, and wait past its expiry before execution. Each attempt should fail before Vault issues a lease or the downstream service sees a request. Those negative tests say more about the boundary than a successful demo in which every component behaves honestly.

Choose the pattern by blast radius

Use a broker for any agent that can select among multiple tools, touch more than one production system, run for a long time, accept untrusted content, or launch code. Those traits describe most useful MCP agents. Keep Vault authentication and secret material outside the model-controlled process, and prefer a broker that performs the privileged operation over one that returns a credential.

Use direct environment injection when all of these conditions hold:

• The process has one fixed purpose and one downstream role.
• The credential is dynamic or otherwise short lived at its authoritative issuer.
• The runtime cannot invoke a shell, load arbitrary extensions, or expose diagnostics.
• Rotation restarts the process automatically and removes old replicas.
• Logs and crash handling have passed a canary leak test.

Choose neither pattern if the downstream provider only offers a broad, long-lived token and the agent can make arbitrary calls with it. Put a small service in front of that provider and expose named business operations. The service keeps the static token, validates inputs, enforces limits, and returns reduced results. This does not make the vendor credential short lived, but it stops the agent from holding it and narrows the commands available to an injected prompt.

For teams already running Vault, the production reference path is straightforward: workload identity authenticates a separate broker, policy fixes one allowed role, Vault issues a leased credential or wrapped response, a constrained executor uses it, and the downstream audit record carries the correlation ID. The MCP process receives only a typed result. For teams without Vault, a cloud secret manager plus workload identity and a small operation broker can enforce the same boundary; do not deploy a secret platform merely to add another HTTP hop.

A Team & AI Audit from oleg.is can map these authority paths as part of examining an AI engineering setup, but the immediate engineering decision should not wait for an audit. Remove parent credentials from the MCP process, pick one production tool, and prove with a canary that its secret cannot appear in the tool response or logs. If that proof fails, the agent is not ready for production access.