Production Deployment & Observability

Serving topology

Most LLM apps fan out into three tiers. Keep the boundaries clean so you can scale, cache, and observe each independently.

client ──▶ API gateway ──▶ orchestration service ──▶ model providers (OpenAI, Anthropic, vLLM…)
                                  │                            
                                  ├──▶ vector store / DB       
                                  ├──▶ tool services (search, code exec, MCP)
                                  └──▶ cache (semantic + exact)

• Stateless orchestration — push state (threads, checkpoints) to Postgres/Redis so any pod can serve any request.
• Streaming first — SSE or WebSockets end-to-end; buffering destroys perceived latency.
• Async + queues — long agent runs go to a worker (Celery, BullMQ, Temporal); the HTTP path stays under ~30s.

Deployment targets

Configuration & secrets

• Inject provider keys from a secret manager (AWS Secrets Manager, Doppler, Vault) — never bake into images.
• Separate keys per environment; rotate on a schedule and on offboarding.
• Front all providers with an LLM gateway (LiteLLM, Portkey, OpenRouter) for failover, rate-limit smoothing, and per-team budgets.
• Pin model versions (gpt-4o-2024-08-06) — "latest" aliases change behavior overnight.

The three pillars of observability

Tracing LLM calls

OpenTelemetry is the lingua franca; the OpenInference and OpenLLMetry conventions add LLM-specific span attributes (model, prompt, tokens, cost). Pick one backend and ship to it consistently.

• LangSmith — first-class for LangChain/LangGraph; datasets + evals + traces in one place.
• Langfuse — open source, OTel-native, self-hostable.
• Arize Phoenix — OSS, OpenInference; strong RAG/agent debugging UI.
• Datadog / Honeycomb / Grafana Tempo — general APM with LLM dashboards.

Step-by-step config for OTel, LangSmith, and Langfuse lives in the Observability Setup guide.

# LangSmith — auto-trace every LangChain / LangGraph call
import os
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_PROJECT"] = "prod-checkout-agent"

# Add custom metadata to the current run
from langsmith import traceable

@traceable(run_type="chain", metadata={"version": "v3", "tenant": "acme"})
def answer(question: str) -> str:
    ...

SLOs to set

• Time-to-first-token (TTFT) p95 < 1.5s for chat UX.
• End-to-end latency p95 budget per route; agents get a separate, looser budget.
• Cost per request p95 and per-tenant daily cap.
• Tool error rate < 1%; retriever empty-result rate tracked separately.
• Eval score on a golden set sampled from prod — alert on drift.

Cost & latency control

• Prompt caching — Anthropic/OpenAI prompt cache for long system prompts and tool schemas.
• Semantic cache — Redis + embeddings for repeat questions; gate on similarity threshold.
• Model routing — cheap model for easy intents, frontier model only when needed.
• Streaming + early stop — abort generations when the client disconnects.
• Batch + structured outputs — JSON mode / tool calls beat parsing free text.
• Token budgets — hard caps per request and per user/day; reject before calling the model.

Reliability patterns

• Retries with jitter on 429/5xx; cap attempts; never retry non-idempotent tool calls without an idempotency key.
• Fallbacks — secondary provider or smaller model when the primary fails or times out.
• Circuit breakers per provider to shed load fast.
• Per-user rate limits at the gateway to contain abuse and runaway agents.
• Idempotency keys on writes triggered by agents so retries don't double-charge.

Rollout strategy

• Prompt & model versioning — treat prompts as code; pin model snapshots.
• Shadow traffic — run a candidate prompt/model on a copy of prod traffic, compare offline.
• Canary + feature flags — ramp 1% → 10% → 50% with eval + cost + latency gates.
• A/B with online evals — judge or thumbs-up/down feedback wired into the gate.
• One-click rollback — prompts, model IDs, and tool schemas all behind the same flag.

Feedback loop

Capture 👍/👎, edit-distance on agent outputs, and task-completion signals. Pipe negatives into your eval dataset and into the next prompt iteration. Production traffic is the best eval set you will ever have — only if you collect it.

Incident response

• Provider outage runbook with failover targets and DNS/feature-flag toggles.
• Cost-spike alert (per-tenant $/hour) with auto-throttle.
• Quality regression alert from online evals — auto-route to last known good prompt.
• Privacy incident: trace + log redaction job, retention policy enforcement.

Production checklist

• Stateless service, externalized session/checkpoint state.
• Streaming responses end-to-end.
• OTel traces with LLM attributes shipped to a backend.
• Dashboards for latency, cost, eval score, error rate.
• Per-user and per-tenant rate limits + budget caps.
• Retries, fallbacks, circuit breakers per provider.
• Pinned model snapshots and versioned prompts behind flags.
• Online evals + user feedback flowing into the dataset.
• PII redaction before logs/traces leave the VPC.
• Runbooks for provider outage, cost spike, quality drift.