MCP Servers in Production: Authentication, Permissions, Logging, and Failure Recovery

Model Context Protocol (MCP) is a protocol for connecting AI clients to tools, resources, and prompts exposed by servers. In production, the hard problem is not making a tool callable; it is preserving the security and operational boundaries around every call.

The thesis is simple: an MCP server is a small distributed system with an unusually powerful caller. Authentication tells you who is at the door. It does not tell you what that identity may do, what the model actually attempted, or how to recover after a timeout halfway through a mutation. Production MCP needs four separate control planes: identity, authorization, evidence, and recovery.

That distinction matters as teams move from local prototypes to hosted servers that read private systems and take external actions. The MCP specification separates the base protocol, authorization, server features, and utilities. Your architecture should preserve that separation too.

The quick answer

How should an MCP server authenticate?

For HTTP transports, implement the MCP authorization flow around OAuth 2.1 conventions, validate the token on every request, bind it to the server's canonical resource URI, and reject tokens issued for another audience. For stdio, use environment or host-provided credentials and keep the process local.

How should it enforce permissions?

Treat the model as an untrusted planner, not as an authority. Derive the effective principal from the validated token, map that principal to a narrow tool and data policy, and enforce the policy again immediately before the downstream action. A hidden system-prompt instruction is not an access-control layer.

What should be logged?

Log the request, authenticated principal, tool name and version, policy decision, downstream request, result class, latency, retry count, and correlation IDs. Redact tokens, secrets, unnecessary personal data, and full tool arguments by default.

How should it recover?

Classify failures before retrying. Retry only transient, safe-to-repeat work; use idempotency keys for mutations; persist a durable operation state; return partial progress explicitly; and send ambiguous outcomes to reconciliation rather than blindly replaying a write.

The foundations

An MCP deployment has at least four boundaries: the client-to-server transport, the server's tool dispatcher, the server-to-upstream API, and the system that owns the resulting side effect. Each boundary has a different question.

The transport asks whether the request is authentic and intended for this server. The dispatcher asks whether this caller may invoke this tool with these arguments. The upstream adapter asks whether it may translate the call into a database query, CRM write, ticket update, or cloud operation. The side-effect owner asks whether the operation is safe to commit, repeat, or roll back.

The MCP overview makes the same separation at protocol level: JSON-RPC and lifecycle management are mandatory, while authorization, tools, resources, prompts, and logging are distinct capabilities. Implementing the protocol successfully is not the same as implementing a production boundary successfully.

A useful request envelope carries more than JSON-RPC parameters:

{
  "request_id": "req_01J...",
  "principal": { "subject": "user_123", "tenant": "acme", "scopes": ["tickets:read"] },
  "tool": "tickets.search",
  "tool_version": "2026-07-01",
  "arguments": { "query": "refund" },
  "policy_snapshot": "pol_9f...",
  "deadline_ms": 8000,
  "idempotency_key": null
}

The model should not supply the principal, tenant, scopes, policy snapshot, or deadline. The server derives or constrains those values. The model supplies intent-shaped arguments; the control plane supplies authority.

That distinction matters even for apparently low-risk research tools. X's official MCP for SEO research is useful precisely because the server exposes a defined API surface; production teams still need to constrain which accounts, searches, and write-capable operations a connected client can reach.

Why is authentication not authorization for an MCP server?

Authentication proves that a credential was accepted. Authorization decides whether the authenticated subject may perform a particular operation on a particular resource under current policy. Treating the first as the second turns a valid token into a standing invitation to every tool and every tenant the server can reach.

The current MCP authorization specification requires an HTTP server to validate access tokens before processing requests, including checking that the token was issued specifically for that MCP server. It also distinguishes 401 for missing or invalid credentials from 403 for insufficient permissions. Those codes define whether the client should re-authenticate or change the requested operation.

Use the token's subject, issuer, audience, expiry, and scopes as inputs to a policy decision. Do not accept a token merely because its signature verifies. A token issued for a different API can be perfectly valid and still be wrong for this server. The spec's resource-indicator requirement exists to make that audience binding explicit.

For interactive users, use authorization code flow with PKCE and exact redirect-URI validation. For service-to-service jobs, use a machine identity or the MCP client-credentials extension, not a copied human refresh token. For stdio servers, use environment-based credentials rather than bolting a browser OAuth flow onto a local process.

The dangerous implementation is token passthrough. If a client token is valid for your MCP server and your server calls a CRM API, forwarding the original bearer token makes the CRM believe the client authenticated directly to it. That collapses audit identity and creates a confused-deputy path. Exchange or mint a separate downstream credential whose audience is the CRM and whose scopes reflect the narrow operation.

This is the same boundary that appears when teams wire an agent runtime into external systems. In Claude Agent SDK support triage, the MCP server is the seam between the agent loop and Zendesk. That seam needs its own identity and policy; the model session is not a substitute for either.

What should permissions look like when the caller is a model?

Permissions should be explicit, capability-shaped, and checked at execution time. A tool named update_ticket is not one permission; its risk changes with tenant, ticket owner, field, status transition, and whether the action is public or internal.

Start with two-phase execution. Let the model propose an action and return a structured preview: target, fields, old values when safe to show, new values, reason, and risk class. Then re-authorize the exact action at commitment. Between planning and execution, the user's role, record state, or policy may have changed.

Do not rely on tool names or descriptions as the permission model. Tool descriptions are model-facing metadata and can be stale, manipulated, or misunderstood. The server must enforce argument schemas, tenant boundaries, allowed fields, maximum result counts, and state transitions in ordinary code.

The MCP security best-practices guide treats sessions as continuity, not authentication. Bind a session to the authenticated principal, use unpredictable identifiers, and verify every inbound request. A session ID that grants access by itself is a bearer credential with a longer shelf life than intended.

Make dangerous operations harder to express than safe ones. Expose tickets.preview_update and tickets.commit_update as separate tools, with the second requiring an idempotency key and a policy decision. Do not expose a universal run_sql, http_request, or execute_shell tool and hope the system prompt keeps it bounded.

The same principle makes Claude Code useful for real SEO work: the value comes from a defined tool surface and workflow, not from an unbounded shell and a request to behave.

What does useful MCP logging include, and what must it leave out?

Useful logs reconstruct a decision without becoming a second copy of the user's private data. The minimum event should answer who called, what tool version ran, what policy was applied, what downstream action occurred, how long it took, and whether the result was committed, rejected, retried, or left ambiguous.

{
  "event": "mcp.tool.completed",
  "request_id": "req_01J...",
  "trace_id": "tr_01J...",
  "tenant_id": "acme",
  "principal_hash": "sha256:...",
  "tool": "tickets.commit_update",
  "tool_version": "2026-07-01",
  "policy_decision": "allow",
  "downstream": "zendesk",
  "outcome": "committed",
  "latency_ms": 742,
  "retry_count": 0,
  "idempotency_key": "idem_01J...",
  "argument_hash": "sha256:..."
}

Keep raw arguments and upstream responses in a restricted audit store only when policy requires them. Hash or selectively project values in ordinary application logs. Never log bearer tokens, API keys, refresh tokens, authorization codes, full customer messages, or unredacted tool output by default.

The MCP logging utility is a client-facing structured log channel, not a replacement for server observability. It defines severity and logger metadata for messages sent to clients and explicitly calls for removing sensitive information, rate limiting messages, and controlling log access. Your server still needs metrics, traces, and an append-only audit trail for consequential actions.

Use three destinations with different purposes:

1. Metrics answer whether the system is healthy: request rate, p95 latency, 401/403/429/5xx rate, tool error rate, upstream health, retry volume, and ambiguous-outcome count.
2. Traces answer where time and failure occurred: client request, policy check, tool handler, token exchange, upstream call, and commit confirmation should share a trace ID.
3. Audit events answer what happened to a protected resource: who requested the action, which policy allowed it, what changed, and which human approved it.

A line that says tool failed is not an audit trail. A database row that stores every prompt forever is not observability; it is a retention liability. Choose fields based on the investigation you need to perform, and test redaction with adversarial values that resemble secrets and personal data.

How should an MCP server recover when a tool call fails halfway through?

Recovery starts by separating “the operation failed” from “the client does not know the outcome.” A timeout after a downstream write may mean the write committed and the response was lost. Retrying immediately can create a duplicate charge, email, or ticket update.

Classify every failure into four states:

• Rejected before side effect: authentication, authorization, schema, quota, or precondition failure. Do not retry; return a stable client error.
• Transient before side effect: connection reset, rate limit, or dependency overload before upstream acceptance. Retry with bounded exponential backoff and jitter if the operation is safe.
• Committed: downstream confirms the mutation. Return success and record the external operation ID.
• Unknown outcome: the response was lost or the downstream cannot prove whether it committed. Persist unknown, stop automatic replay, and reconcile using the idempotency key or an upstream lookup.

For reads, retry policy can be mechanical. For writes, idempotency is the boundary. Generate or require a key at the start of the logical operation, persist it before calling upstream, and send it to every downstream that supports it. If upstream does not support idempotency, create a durable operation record and use a provider-specific deduplication or reconciliation query. Never claim “exactly once” when the infrastructure only provides “at least once.”

Every tool should accept a deadline. The deadline travels through the call chain and leaves enough time for a useful response. Without a deadline, a client timeout can outlive the server's work, producing a zombie operation that continues spending money or mutating data after the caller has given up.

Return partial results as data, not as a vague apology. A batch tool should identify completed, skipped, failed, and unknown items. A long-running operation should expose a durable status resource or task handle. The client can then poll or resume without asking the model to infer state from an error message.

The MCP tasks specification is relevant for operations that outlive one request: task creation, completion, retrieval, and audit events need durable state. Even if your server does not implement the full capability, adopt the same state-machine thinking.

The edge cases that break careful deployments

A valid token with a stale role. Role changes do not necessarily invalidate already-issued tokens. Keep access tokens short-lived for sensitive systems, or check a policy version before a high-impact commit.

Prompt injection in tool results. A retrieved document, ticket, or web page can contain instructions aimed at the model. Mark untrusted content as data, keep it out of privileged policy fields, and ensure the model cannot turn tool output into new authority. Authorization must be independent of returned text.

Schema drift. A tool description may update before all clients do. Version tools, validate arguments server-side, and return typed validation errors rather than generic 500s.

Rate-limit storms. Return retry-after information, cap retries per request and principal, and use a circuit breaker so one broken dependency does not consume all worker capacity.

Partial batches. If a batch mixes reads and writes, make idempotency explicit per item. “Retry the batch” is unsafe unless every item is independently deduplicated.

Session confusion. A session ID copied from one server to another should not resume a stream or receive events. Bind sessions to server, tenant, and principal, and expire them.

Human approval that expires. Bind approval to an action hash, target version, approver, and short expiry. If the record changed, require a new preview and approval.

The honest take

The protocol cannot decide your business policy. MCP gives you interoperable message and authorization conventions; it does not know whether a support agent may refund $50, whether a research tool may read a private repository, or whether a bulk delete is reversible. Those decisions must live in code, policy data, and operator practice.

OAuth is not a complete security architecture. Audience validation and PKCE close important classes of mistakes, but an authorized user can still request a harmful action, and an injected document can still influence planning. The effective defense is layered: narrow tools, object-level checks, dry-run previews, human gates, outbound allowlists, and adversarial testing.

Logs do not automatically create accountability. If the audit store is editable by the same service that performs the action, an attacker or bug can erase evidence. Separate write paths and access controls, define retention, synchronize clocks, and test that an incident responder can reconstruct a decision without reading every prompt.

Failure recovery is not magic exactly-once execution. You can make retries safe for specific operations; you cannot infer a lost side effect from a timeout alone. The honest design is to represent uncertainty, reconcile it, and make the model wait rather than letting it guess.

A production checklist

• Every HTTP request validates an access token, issuer, audience, expiry, and scope.
• Tokens are never accepted in query strings, logged, or passed through to another resource server.
• Every tool has a server-enforced policy for tenant, object, fields, limits, and state transitions.
• Mutations have preview/commit separation, idempotency keys, and durable operation state.
• Sensitive actions have explicit approval or a documented reason not to require it.
• Metrics, traces, and audit events use a shared correlation ID.
• Logs redact secrets and unnecessary personal data, and audit retention is defined.
• Retries are bounded, jittered, dependency-aware, and disabled for unknown write outcomes.
• Sessions are random, short-lived where appropriate, and bound to principal and server.
• Tool schemas and versions are tested against stale clients and malformed model arguments.
• The failure path is tested: 401, 403, 429, timeout-after-commit, duplicate webhook, partial batch, and revoked access.

The bottom line

An MCP server is production-ready when it can answer four questions for every tool call: who was allowed to ask, what exactly was allowed, what actually happened, and what the system will do when it cannot tell. Authentication handles the first part only. Permissions constrain blast radius; logs preserve evidence; idempotency and reconciliation turn ambiguous failures into manageable states.

Use the protocol's OAuth conventions for HTTP, keep stdio credentials local, and treat the model as a planner whose proposed actions must be re-authorized by ordinary server code. Start with read-only tools, add previews before commits, and ship the audit and recovery path before adding more tools. The fastest way to make an MCP server trustworthy is to make unsafe behavior explicit, bounded, and boring.

Frequently asked questions

Should every MCP server use OAuth?

No. HTTP servers handling protected resources should follow MCP authorization conventions. Local stdio servers generally use environment or host-provided credentials. The transport and threat model determine the mechanism.

Can I use one API token for the MCP server and the upstream service?

Avoid it. Validate the inbound token for the MCP server, then use a separate downstream credential scoped to the upstream resource. This preserves audience boundaries and makes revocation and audit clearer.

Should I retry a failed tool call?

Only after classifying the failure. Retry reads and known-safe transient failures with limits. Do not blindly retry a write after a timeout; mark it unknown and reconcile first.

What is the most important permission rule?

The model never supplies its own authority. Derive identity and policy from the authenticated request, enforce object-level checks at execution time, and keep high-impact mutations behind a distinct commit path.

Is MCP logging enough for compliance?

No. MCP's logging capability is a structured message channel between server and client. Compliance-grade evidence also needs server-side metrics, traces, and an access-controlled audit store for consequential actions.