Is your MCP server monitoring what agents actually need?

An MCP server can return HTTP 200 and still be useless to an agent. The process may be alive while authentication fails, a required tool has disappeared, its input schema has changed, or the dependency behind the tool is timing out. Production monitoring has to test the contract the agent consumes, not merely the socket that accepts traffic.

I treat every MCP server as a runtime dependency with three separate questions: can I reach it, does it still expose the contract I approved, and are real agent calls succeeding at an acceptable cost and speed? Mixing those questions into one green status hides the failures that hurt users. Separating them produces alerts an engineer can act on and evidence a founder can use when deciding whether an agent workflow is ready for more traffic.

Is the server alive, ready, or actually usable?

An MCP server needs distinct liveness, readiness, and synthetic checks because each protects a different decision. Liveness tells an orchestrator whether to restart a process. Readiness tells a load balancer whether to send traffic. A synthetic MCP check tells the on-call engineer whether an agent can complete an approved operation.

Keep liveness deliberately shallow. It should confirm that the event loop or request handler responds, without querying a database, an identity provider, or every downstream API. If a dependency outage makes liveness fail, an orchestrator can restart healthy processes until it creates a second outage. A local process check or a small HTTP endpoint outside the MCP protocol is enough.

Readiness should test dependencies that make every request impossible. That may include loaded configuration, a reachable credential store, or a database connection pool. Do not put optional tool dependencies into global readiness. If one accounting integration is down while five read-only tools work, removing the whole server from rotation expands the failure.

The synthetic check must speak MCP. For deployments on protocol revisions through 2025-11-25, it should complete initialize, record the negotiated version and capabilities, send notifications/initialized, then issue a harmless list or call request. For 2026-07-28, the core is stateless: request metadata carries the protocol version and client details, and server/discover can return supported versions, capabilities, and server identity before a real operation. The current specification also removes ping from that revision, so a monitor that equates MCP health with a successful ping can reject a healthy new server or bless an old path that agents no longer use.

The official Model Context Protocol lifecycle specification says clients and servers must respect the negotiated version and use only negotiated capabilities. That is more than protocol etiquette. Your probe must use the same version family, transport, authentication flow, and client capability profile as production, or its green result describes a different system.

Use one status for each decision:

• live: the process can answer a shallow local request.
• ready: the instance can accept ordinary traffic.
• contract_ok: the MCP discovery or initialization result matches policy.
• synthetic_ok: a harmless agent-shaped operation finishes correctly.

This separation also prevents a familiar alerting mess. A failed synthetic check should page the workflow owner when agents are blocked, while a failed liveness check may stay inside the platform team's automated recovery loop.

A production probe must behave like a small client

A useful probe implements enough of the client lifecycle to catch failures at the boundary. A generic HTTP monitor cannot detect a rejected protocol version, malformed JSON-RPC response, missing capability, expired MCP credential, or a tool result that violates its declared output.

Run two probe classes. An instance probe runs frequently, stays cheap, and avoids side effects. A workflow canary runs less often and exercises one safe business path end to end. For a sales data server, the instance probe might list tools and validate schemas; the canary might read a dedicated synthetic account whose contents never change. Never use a mutating customer operation as a heartbeat.

The monitor identity needs its own least-privilege credential. Give it access only to discovery and designated canary resources. Excluding authentication makes the check safer to operate but weaker as evidence: an agent can still fail because token audience, scope, issuer, or expiry handling broke. The 2025-11-25 authorization specification requires HTTP requests to carry authorization on every request and distinguishes invalid tokens with 401 from insufficient permission with 403. Record that distinction instead of flattening both into auth_failed.

This request is a compact probe for a legacy stateful server. Use a synthetic client name so logs and rate limits can separate monitoring from user traffic.

{"jsonrpc":"2.0","id":"health-1","method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"prod-monitor","version":"1.0.0"}}}

A valid response is not simply parseable JSON. Check that the response ID matches, result.protocolVersion belongs to your supported set, serverInfo.name matches the expected service, and required capability objects exist. Then finish the lifecycle and request the catalog.

{"jsonrpc":"2.0","method":"notifications/initialized"}
{"jsonrpc":"2.0","id":"health-2","method":"tools/list","params":{}}

For a stateless 2026-07-28 endpoint, send server/discover or a harmless request with the required per-request metadata and HTTP headers. Monitor both paths during migration. A fallback that quietly negotiates an older version may keep traffic flowing, but it should increment a downgrade counter and attach the chosen version to the trace.

Set a deadline on every probe and a larger absolute deadline on long operations. The lifecycle specification recommends request timeouts and cancellation for timed-out requests. A probe that waits forever consumes the same pools and file descriptors as a stuck agent call, so the monitor itself can worsen an incident.

Transport changes what the monitor can prove. For Streamable HTTP, test through the production DNS name, gateway, TLS policy, authorization server path, and routing rules. Also probe one instance directly from inside the cluster so the responder can separate an application failure from an edge failure. The public probe and instance probe should share contract assertions even though their network paths differ.

For stdio, the host launches the server as a child process, so a remote uptime service cannot observe the real boundary. Instrument the host to record launch failures, startup duration, unexpected exits, stderr volume, request deadlines, and restarts. The transport specification permits logs on stderr but requires stdout to contain only valid MCP messages. A stray startup banner on stdout can corrupt the protocol while the process remains alive, so include framing validation in the synthetic client.

Do not convert every local stdio server into HTTP just to make monitoring convenient. That adds a network and authorization surface the workflow may not need. Export health from the host process, run a scheduled local canary under the same operating-system identity as the agent, and send the resulting metrics and traces to the central backend.

Do not retry until the first failure disappears. Record the first attempt, then make at most one bounded retry with jitter if the operation is safe. Otherwise the dashboard reports retry success while users still pay the added latency and token cost.

Capability drift is a release failure

Capability drift means the server contract observed in production differs from the contract your agent, policy, and evaluation suite approved. It includes more than a missing tools capability. Tool names, descriptions, input schemas, output schemas, annotations, resource templates, prompt arguments, server instructions, and protocol versions can all drift.

Some drift is intentional. A deployment adds a tool, narrows an enum, or rewrites a description. That change still needs review because agents select tools from names and descriptions, then construct arguments from schemas. A wording change can alter selection behavior even when no conventional API client would notice. Removing an optional property can be as disruptive as deleting an endpoint if prompts were producing it reliably.

Distinguish advertised capability from usable inventory. The server may advertise tools while tools/list returns an empty catalog because registration failed. It may advertise list change support while clients never receive or consume the notification. Under older revisions, listChanged tells the client that the server can announce catalog changes; it does not prove that a given catalog is correct. Under 2026-07-28, list results include cache hints and deterministic ordering, so monitors must also verify that caching policy does not preserve an obsolete catalog past the rollout window.

Create a versioned contract manifest for each environment. Store the protocol versions you accept, server identity, required capabilities, and an allowlist of callable items. Give every item a stability class:

• required, where absence blocks the deployment;
• optional, where absence degrades a named workflow;
• forbidden, where appearance creates an unreviewed action path;
• informational, where the monitor records change without blocking.

The forbidden class matters. Teams often alert only when a tool disappears. A newly exposed delete_record tool can be a larger incident because the model now sees an action that policy, prompts, and approvals never considered.

Compare contracts in continuous delivery and again from outside the cluster after deployment. The build-time comparison catches an intended but unreviewed change. The external comparison catches wrong routing, partial rollout, stale discovery caches, environment-specific registration, and a gateway that sends the monitor to a different version than agents receive.

Do not approve a contract using a hash alone. A hash says something changed but gives the reviewer no consequence. Generate a semantic diff that says tool removed, required argument added, enum narrowed, description changed, or output property type changed. Block incompatible changes, require explicit approval for new mutating tools, and record harmless description edits for evaluation.

Normalize catalogs before you compare them

A stable fingerprint requires canonicalization, otherwise array order or cosmetic JSON changes create noise. Save the raw response for diagnosis, but compare a normalized representation with volatile transport fields removed and catalog entries sorted by stable identity.

The following jq filter creates a small tools manifest from a tools/list response. It retains the fields that affect model choice and argument generation, sorts tools by name, and calculates a digest outside the JSON transformation.

jq -S '{tools: [.result.tools[] | {name, description, inputSchema, outputSchema, annotations}] | sort_by(.name)}' tools-list.json > tools-contract.json
sha256sum tools-contract.json

The output should look like this in source control or an artifact store:

{"tools":[{"annotations":{"readOnlyHint":true},"description":"Find an invoice by number","inputSchema":{"properties":{"number":{"type":"string"}},"required":["number"],"type":"object"},"name":"find_invoice","outputSchema":{"type":"object"}}]}

Canonicalization needs a policy, not a blind deletion list. Descriptions may look like prose noise to an API engineer, yet they influence tool selection. Annotations may be hints rather than enforced security controls, yet a client can use them when deciding whether to ask for approval. Keep both in the semantic contract. Remove request IDs, cursor values, cache ages, and other response-instance data.

Run schema checks beyond equality. Verify unique tool names, valid JSON Schema, bounded string and array inputs where your policy requires them, and the absence of secrets or internal URLs in descriptions. If the server declares an output schema, call the synthetic tool and validate the returned structured content against it. A catalog can remain unchanged while an upstream adapter starts returning a different shape.

Fingerprint prompts and resources separately. They have different owners and failure consequences. One combined hash turns every alert into an investigation of the entire server, while separate hashes point directly to the changed surface.

Store these fields with every observation:

{"server":"billing-mcp","environment":"production","observed_at":"2026-08-09T12:00:00Z","protocol":"2026-07-28","server_version":"4.3.1","tools_digest":"sha256:...","prompts_digest":"sha256:...","resources_digest":"sha256:...","deployment":"git:..."}

That record connects runtime behavior to a deployment without putting high-cardinality commit values into every time-series metric. Keep the digest and deployment ID in an event or trace attribute, then expose a simple contract_match gauge by server and environment.

Usage metrics must follow the whole tool call

The metrics that matter describe demand, correctness, latency, and cost across the agent-to-server path. CPU and memory still belong on the dashboard, but they cannot tell you whether the model chose a tool, whether the client obtained approval, or whether the result helped the workflow finish.

Measure each stage separately: tool offered to the model, tool selected, call attempted, server accepted, downstream work completed, result validated, and agent workflow completed. These events answer different questions. A fall in calls may mean low demand, a catalog that vanished, a model that stopped selecting the tool, or an approval screen users reject. One requests_total counter cannot distinguish them.

At the server boundary, collect request count, in-flight requests, duration, response bytes, and outcomes. Use labels such as server, environment, protocol version, method, tool name, outcome class, and transport. Keep error details in logs or traces. Never put user ID, session ID, request ID, arbitrary resource URI, prompt text, or raw error message into metric labels. Those values create unbounded cardinality and may expose sensitive data.

A practical metric set can start here:

mcp_requests_total{server,method,tool,outcome,protocol}
mcp_request_duration_seconds{server,method,tool,outcome}
mcp_inflight_requests{server,method}
mcp_contract_match{server,environment}
mcp_capability_changes_total{server,change_type}
mcp_auth_failures_total{server,reason_class}
mcp_downstream_duration_seconds{server,dependency,operation,outcome}
mcp_result_validation_failures_total{server,tool}
agent_tool_selections_total{workflow,server,tool}
agent_workflow_outcomes_total{workflow,outcome}

Count protocol errors separately from tool errors. A JSON-RPC Method not found response, an HTTP 401, an MCP tool result with isError: true, invalid structured content, and a downstream timeout have different owners and fixes. Map them into a small controlled outcome vocabulary, then retain the original error code on the trace.

Histograms should match decisions you can make. Observe server processing time and end-to-end client time separately so queues, gateways, retries, and model-side approval delays do not get blamed on the handler. Publish median and tail percentiles from histograms, but alert on service-level objective burn or sustained failure ratios rather than a single noisy percentile window.

Usage also needs a denominator. tool_calls_total rising looks impressive until you learn that completed workflows stayed flat and retries doubled. Track calls per completed workflow, selection-to-success conversion, and the share of workflows that fall back after a tool error. These ratios reveal whether the MCP layer reduces work or merely creates activity.

Establish a baseline before setting thresholds

Collect a baseline by server, tool, workflow, and protocol version before turning every unusual value into an alert. Traffic has natural cycles, and agent changes can shift selection rates without changing server health. Compare a deployment with the previous known-good deployment under similar traffic, then annotate model, prompt, client, gateway, and server releases on the same timeline.

Separate capacity from demand. A queue can grow because agents suddenly select a useful tool more often, because retries amplify steady demand, or because each call became slower. Plot arrival rate, completion rate, concurrency, queue time, and downstream latency together. That view tells you whether to add capacity, stop a retry loop, or fix the dependency.

Cost deserves a measured path as well. For each workflow, record tool attempts, response bytes, downstream billable units when available, model turns added by failures, and successful completion. Do not estimate savings from request counts. A cheap tool that returns ambiguous data can force another model turn and cost more than a slower tool with a precise result.

Treat zero traffic carefully. Zero calls to a server during a quiet period is normal. Zero offered tools while agents are active can mean discovery failed. Many clients do not expose a clean tool_offered metric, so instrument the host application at the point where it constructs the model request. That is the only place that can distinguish "the model ignored this tool" from "the client never supplied it."

Create a small reconciliation job between client and server telemetry. For a time window, client attempts should approximately match server receipts after accounting for transport failures, cancellation, and sampling. A widening gap catches gateway rejection, DNS trouble, client-side timeout, and telemetry loss. Do not demand exact equality because retries and clock boundaries create legitimate differences.

Metrics also need change control. Define label vocabularies, bucket boundaries, and retention with the same care as an API. An engineer who adds raw resource IDs to a popular counter can overload the monitoring backend during the incident that generated those IDs. Review metric changes in code and put a cardinality limit on the exporter or collector.

Traces explain failures that metrics can only count

Propagate one trace across the host application, model turn, MCP client, gateway, server handler, and downstream API. Without that chain, an engineer sees a slow agent, a normal MCP median, and a slow database query in separate tools with no proof that they describe the same request.

The 2026-07-28 release documents W3C Trace Context fields in MCP metadata, which gives SDKs and gateways common names for traceparent, tracestate, and baggage. Use that support where your client and server implement it. For older versions, propagate trace context through supported transport metadata or create links between client and server spans at the gateway. Do not smuggle tracing fields into tool arguments because the model can alter them and downstream business logic may log them as user data.

Model a tool call with spans for client selection, approval wait, transport, server dispatch, credential exchange, and downstream operation. Put controlled attributes on the spans: MCP method, tool name, protocol version, server version, result class, retry count, contract digest, and response size. Record arguments only through an explicit redaction policy. Tool inputs routinely contain source code, customer records, access tokens, file paths, and queries that do not belong in an observability backend.

Sampling needs two lanes. Keep a low baseline sample for successful calls and retain all errors, contract mismatches, authorization failures, and unusually slow traces. Head sampling alone decides before the failure occurs, so it often discards the evidence you wanted. Tail sampling can retain traces based on their final status, provided the collector has enough capacity during an incident.

Logs remain useful for state transitions and detailed errors, but protocol logging is not your observability pipeline. Older MCP revisions can advertise structured logging to a client; the 2026-07-28 revision deprecates and removes that core feature. Server operators still need ordinary application logs independent of whether a connected client requests or displays MCP log notifications.

Redact before export, not only in the dashboard. Restrict observability access, set retention by data class, and test redaction with synthetic secrets. A trace processor that removes authorization but leaves a token inside tool.arguments has not solved the problem.

Alerts should describe agent impact

Page on sustained user impact, fast contract risk, and exhausted capacity. Do not page because one synthetic request failed or a catalog digest changed during an approved rollout. An alert needs a clear owner, a runbook decision, and enough context to locate the affected workflow.

Define service-level indicators at two layers. The MCP service layer covers accepted requests that return a valid protocol response within a threshold. The workflow layer covers agent jobs that use the server and reach the intended business outcome. The first helps the server team operate the dependency; the second stops everyone from celebrating a technically healthy tool that agents cannot use successfully.

Useful paging conditions include a fast burn of the request success objective, synthetic failure from multiple locations, all instances failing readiness, a required tool disappearing, a forbidden tool appearing, and saturation with a growing queue. Ticket or notify for an approved description change, a gradual traffic shift, or one optional tool entering degraded mode.

Use multi-window alerts for error budgets so a brief spike does not wake someone and a slow failure does not hide below a high threshold. Pair the alert with dimensions the responder needs: affected server and tool, protocol version, deployment, region, error class, first observed time, and a trace example. Keep raw customer arguments out of the page.

Capability drift deserves different severity rules from availability. A removed required read tool can block a rollout. A new mutating tool should block exposure immediately even when every request succeeds. A changed description may trigger an evaluation suite and human review without paging. Encode these decisions in the manifest instead of asking an on-call engineer to invent policy at 2am.

Test the alerts. Disable a canary credential, remove a required tool in a staging deployment, inject a downstream timeout, and return content that violates an output schema. Confirm that each failure reaches the expected dashboard and owner. If the only proof of an alert is its configuration file, assume it will surprise you.

One stale catalog can survive a clean deployment

A common failure starts with a harmless-looking tool rename. The server deploys search_orders_v2, removes search_orders, and passes its HTTP health check. Half the clients reconnect and fetch the new catalog. The rest keep an older cached list or remain on a legacy session that never processes a list-change notification.

The model on an old client still selects search_orders. Calls reach a healthy new instance and receive Method not found or an unknown-tool error. Client retries raise latency. The agent may then improvise with a less suitable tool, so workflow failures appear several minutes later and far from the MCP server dashboard. CPU stays low, HTTP availability stays high, and the deployment looks clean.

This is why I argue against treating tool discovery as startup configuration. The approach is popular because it saves list calls and makes clients simpler. It is wrong unless the cache has an explicit freshness rule, a change signal, and a safe response to an unknown tool. Discovery is part of the runtime contract.

Prevent the failure with a compatibility window. Add the new tool, keep the old name as a deprecated alias, deploy clients that understand both, observe old-name usage falling, then remove the alias in a later release. If an alias could perform a mutating action, preserve the same authorization and approval behavior while it exists.

Monitor the migration with four signals: catalog digest by instance, negotiated protocol distribution, calls by old and new tool name, and unknown-tool errors by client version. The deployment can finish only when every serving instance reports the intended digest and old clients no longer call the alias.

The stateless 2026 protocol reduces hidden connection state, but it does not eliminate stale catalogs. Cache hints make freshness explicit, which helps only when servers set sensible values and clients obey them. Your canary should fetch through the same gateway and cache path as an agent, then compare what it sees with the release manifest.

Ownership turns telemetry into production control

Every production MCP server needs one service owner and one workflow owner. The service owner controls availability, contract publication, credentials, capacity, and downstream dependencies. The workflow owner controls prompts, model behavior, approvals, fallbacks, and the business success measure. An incident often crosses that boundary, so dashboards and runbooks must name both.

Put a contract check in continuous delivery before traffic shifts. Deploy to one slice, run discovery and a safe canary through the production gateway, compare the normalized catalog, validate one result, and watch error-budget burn. Promotion should stop automatically on a required capability mismatch or a new forbidden tool. Rollback should restore both server code and the matching contract manifest.

Review weekly usage by workflow, not only by server. Retire tools that agents never select, investigate tools selected often but completed rarely, and cap expensive operations whose retry behavior multiplies downstream cost. When a tool is intentionally unused as an emergency path, label it that way so nobody deletes it from a popularity report.

For founders running several agent workflows, this operating work is part of the engineering system, not optional monitoring polish. On oleg.is, the Team & AI Audit examines where AI workflows, team ownership, and production controls can reduce engineering cost without hiding operational risk.

Start the production review with evidence from one server: its accepted protocol versions, normalized contract, last seven days of tool outcomes, one complete trace, and the owner for every alert. If you cannot assemble those five items, the server is still an experiment regardless of how many agents depend on it.