LLMOps Playbook

1. Scope of this playbook

This playbook is for teams operationalizing LLM-powered features and agents in production — versioning prompts and configurations, running continuous evaluation, monitoring for drift, and managing the release process for changes to any of these. It assumes you already have LLM features live or nearly live; it does not cover initial model selection or the agent platform/orchestration layer itself (see the AI Agent Platform Playbook), though the two are closely related and often owned by overlapping teams.

2. LLMOps maturity model

Self-assessment checklist

• Can you see the full history of every change made to a production prompt, including who changed it and why?
• Does every prompt or model change go through an evaluation gate before reaching production, or does quality get checked after the fact?
• If a model provider silently updates the model behind an API you call, would you notice through monitoring, or only when a user complains?
• Can you roll back a prompt change independently of a full application deploy?
• Is there a feedback loop from production issues back into your evaluation dataset, or does every incident get investigated from scratch?

3. Is dedicated LLMOps tooling worth it yet?

A single team with one or two LLM features and a lightweight, disciplined process (prompts in version control, a small manual eval checklist run before each change) can operate without dedicated LLMOps tooling for a while. The case for investment strengthens once you have multiple teams shipping LLM changes, prompts changing frequently enough that manual review is a bottleneck, or an incident that traced back to an untracked prompt or silent model change.

The clearest signal: someone asks "what changed?" after a quality regression, and the honest answer takes hours of git archaeology and Slack history instead of a few minutes in a change log.

4. Readiness checklist

• Prompts and model configurations are stored as versioned artifacts, separate from (but linked to) application code.
• An evaluation dataset exists — even a small, imperfect one — representative of real production inputs, not just hand-picked happy-path examples.
• There's a defined process for who can approve a prompt/model change reaching production, proportional to the feature's risk.
• Monitoring infrastructure can track quality signals in production (not just uptime/latency), even if those signals start out simple, like user feedback or escalation rate.
• A rollback path exists that doesn't require a full application redeploy.

5. Build vs. buy

6. Phased rollout playbook

Phase 0 — Baseline

Move prompts out of ad hoc code changes into version control as a first, low-cost step. Build a small evaluation dataset from real production examples, even a few dozen cases — this is more valuable early than a large but synthetic one.

Phase 1 — Pre-deploy gates

Require every prompt/model/config change to pass the eval suite before reaching production. Keep the bar achievable at first; a gate nobody can pass gets bypassed, not respected.

Phase 2 — Production monitoring

Add continuous quality monitoring in production, not just pre-deploy checks — this is what catches drift from upstream model updates and real-world input distribution shifts that a static eval suite won't see.

Phase 3 — Closed-loop iteration

Feed production monitoring signals back into the evaluation dataset automatically, and move toward A/B testing prompt/model changes rather than shipping changes to all traffic at once.

7. Governance & risk checklist

• Every production prompt/model change has a named approver, proportional to the feature's risk level.
• Changes to the model behind an API (a provider's silent model update) are detected and reviewed, not discovered accidentally.
• There's a documented, tested rollback process for prompt and configuration changes, exercised at least occasionally so it's trusted when actually needed.
• Evaluation datasets are reviewed periodically for staleness — a suite that hasn't changed while production traffic has is a governance gap, not a stable baseline.
• Incidents traced to a prompt or model change feed back into the review process for future changes, not just a one-time fix.

8. Security checklist

• Evaluation datasets containing production data are handled per your data-classification policy — eval data is still sensitive data.
• Prompt injection resistance is part of the evaluation suite, not treated as a separate, occasional security review.
• Access to modify production prompts is scoped and logged, not open to anyone with repository access.
• Monitoring captures enough detail to investigate an incident (what prompt version, what model version, what inputs) without over-logging sensitive user data by default.
• Third-party LLMOps tooling that has access to production prompts and evaluation data is vetted with the same rigor as any other vendor with access to your systems.

9. Cost model & ROI

The ROI case is strongest where it's hardest to see in advance: incidents avoided don't show up as a line item, only as an absence of incidents. Track regression rate and mean-time-to-detect explicitly if you want to make this case visible to leadership.

10. Organizational playbook

A platform or ML-engineering team should own the LLMOps tooling and process — the eval framework, versioning system, and monitoring infrastructure. Feature teams should own their own prompts, evaluation datasets, and the decision of when a change is ready to ship, using the platform team's tools and gates. This mirrors the DevOps platform-team pattern: the platform team builds the guardrails, product teams move fast within them, and neither role becomes a bottleneck for the other if the tooling is genuinely self-service.

11. Common failure patterns

• Prompts as code, forever: never separating prompt versioning from application code, so every prompt tweak requires a full deploy cycle and loses independent change history.
• Eval-suite rot: building an evaluation suite once at launch and never updating it as production traffic patterns shift, so it stops catching the regressions that actually occur.
• Silent model drift: a provider updates the model behind an API call, application behavior shifts, and nobody notices until users complain — because nothing was monitoring for it.
• Gate without teeth: an evaluation gate that exists on paper but can be bypassed under deadline pressure, which defeats its purpose the first time it matters most.
• One-time evaluation: treating evaluation as a pre-launch checkbox rather than a continuous practice, missing the drift and edge cases that only appear at production scale and over time.

12. LLMOps maturity decision scorecard

An organization that scores well on release velocity but poorly on evaluation gates and monitoring is optimizing for shipping speed at the expense of the regression detection that keeps shipping speed sustainable. Weight accordingly.