Enterprise RAG Playbook
A decision framework for enterprise architects and technology leaders running a RAG initiative — maturity model, readiness checklists, build-vs-buy, governance, security, cost, and a reusable platform scorecard.
- RAG maturity is an organizational problem before it's a technical one. Most stalled RAG programs fail on ownership, governance, or scope — not embeddings.
- Before you build anything, confirm RAG is the right pattern. Long-context stuffing and fine-tuning solve different problems and are sometimes the better answer.
- Data readiness is your primary risk, not model choice. No architecture recovers from bad source documents.
- Build vs. buy is a spectrum. Most enterprises land on managed retrieval infrastructure plus custom orchestration, not either extreme.
- A RAG security review must treat retrieved content as an untrusted input, not just the user's query.
- Cost scales with tokens injected and reranked per query, not documents indexed. Model this before you commit to a chunking strategy.
- Ship one narrow pilot with a measurable success metric before building platform-level infrastructure for use cases that don't exist yet.
1. Scope of this playbook
The RAG Systems & Vector Databases post in the AI Architect's Blueprint series covers the technical mechanics — chunking, embeddings, vector databases, retrieval strategies, evaluation. This playbook assumes that material and focuses on the layer above it: the decisions, checklists, and organizational structure that determine whether a RAG initiative actually ships and survives contact with production.
Use it sequentially if you're scoping a RAG program from zero. Use individual sections as reference if you're mid-flight and need a specific decision — build vs. buy, a security review, a cost model — without re-reading the whole thing.
This is a framework, not a template to follow blindly. Enterprise context — regulatory exposure, existing cloud investment, data sensitivity, team maturity — changes which answers are right. The checklists tell you what to decide, not what to decide.
2. RAG maturity model
Most organizations don't have one RAG system — they have several, at different maturity levels, often without realizing it. Use this to locate where a given initiative actually is, not where you'd like it to be.
Prototype
Notebook or script, static docs, no eval, no owner
Pilot
One use case, one source, golden dataset, manual gate
Production-hardened
CI eval, retrieval-time access control, observability
Platform
Shared infra, self-serve onboarding, governance review
Governed & Optimized
Freshness SLAs, continuous eval, cost tuning
Most enterprise RAG efforts — including ones already "in production" — sit at Stage 1 with Stage 2 branding.
Self-assessment checklist
- Does an automated evaluation pipeline run before every change ships, not just once at launch?
- Is access control enforced by the retrieval layer itself, independent of the application UI?
- Can you name the owner responsible for a specific data source's freshness?
- Is there a documented process for onboarding a second use case without rebuilding the pipeline?
- Is cost per query visible and attributable to a specific use case or team?
- Has the system had a dedicated security review covering retrieval-specific risks (Section 8)?
Zero or one checked: Stage 0–1. Two to four: Stage 2. Five or six: Stage 3–4.
3. Is RAG even the right pattern?
RAG is the default reach for "make the model know our stuff," but it isn't the only tool, and it isn't free. Confirm the fit before investing in the pipeline.
Large / current / access-controlled corpus
→ RAG. This is the core case it's built for — knowledge separated from reasoning, updated without retraining.
Small, static corpus (a few dozen docs)
→ Long-context stuffing. No retrieval infrastructure, no chunking decisions, no retrieval failure mode.
New skill, style, or output format needed
→ Fine-tuning. RAG adds knowledge, not behavior. Style and skill live in weights, not context.
Multi-hop reasoning across entities
→ GraphRAG / agentic RAG. Chunk-based retrieval struggles to connect information that isn't co-located.
Gut check: if the honest answer to "why not just paste the relevant documents into the prompt" is "the corpus is too big, changes too often, or has to be filtered per user," you need RAG. If it's "we haven't tried," try the simpler option first.
4. Readiness checklist
Run this before Phase 0 kicks off. Gaps here are the single biggest predictor of a pilot that stalls.
Data readiness
- Source documents identified and access confirmed
- An owner or SME assigned per source, not just per project
- Update frequency and freshness requirement defined per source
- Sensitive or PII content identified and classified
- Document format inventory complete (PDF, HTML, structured, scanned)
- Representative sample available for chunking experiments
Infrastructure readiness
- Cloud vs. on-prem decision made, residency constraints known
- Vector DB tier selected or trial environment provisioned
- LLM and embedding model provider(s) selected
- Logging and observability stack identified
- Environment exists for golden dataset and CI regression testing
Team readiness
- Product owner assigned for the use case
- Engineering owner assigned for the retrieval pipeline specifically
- SME available to validate the golden dataset
- Security reviewer identified before launch, not after
- Support/on-call plan defined for production issues
Governance readiness
- Existing data classification policy mapped to this use case
- Access control model defined — who can query what
- Retention and deletion policy for logged queries defined
- Escalation path for a user reporting an incorrect answer
- Named sign-off owner for launch
If more than a couple of items in any category are unchecked, the gap belongs on the project plan before architecture discussions start.
5. Build vs. buy
| Approach | What you own | Best for | Watch out for |
|---|---|---|---|
| Fully managed Bedrock KB, Vertex AI Search, Azure AI Search | Configuration and data | Fastest time-to-pilot, standard document types | Less control over chunking/retrieval internals |
| Managed retrieval + custom orchestration Managed vector DB, custom pipeline | Pipeline logic, retrieval strategy | Non-standard chunking, hybrid search tuning | More engineering investment; you own eval |
| Fully self-hosted Self-hosted vector DB, open-source embeddings | The entire stack | Strict residency, cost-sensitive at scale | Highest operational overhead; slowest to pilot |
| Enterprise search + LLM layer Existing search platform extension | Little — mostly configuration | Mature search deployment already indexing the corpus | Retrieval quality bounded by vendor roadmap |
Start from data sensitivity and existing cloud footprint, not from what's technically most capable. If you're already Azure-native with the target documents in SharePoint, evaluate Azure AI Search first — the integration cost you avoid usually outweighs marginal retrieval-quality differences between platforms.
6. Phased rollout playbook
Business case
Success metric, baseline cost, single use case, budget
Pilot
Golden dataset, manual review, internal only
Production hardening
CI eval, retrieval-time access control, security review
Scale-out
Second use case, self-serve onboarding, chargeback
Phase 0 — Business case & scoping
- A measurable success metric defined (deflection rate, time saved, accuracy threshold — not "users like it")
- A baseline established for how the task is done today, including its current cost
- A single use case chosen — resist scoping multiple use cases into the first pass
- Budget and timeline approved against a realistic pilot window (six to ten weeks is typical)
Phase 1 — Pilot
- Single data source, chunking strategy chosen and tested against samples
- Golden dataset built — twenty to fifty questions is a workable floor
- Retrieval and generation quality manually reviewed against the golden set
- Internal-only rollout; no external users yet
- Go/no-go gate defined and agreed before Phase 2 starts
Phase 2 — Production hardening
- Automated evaluation (Ragas or equivalent) wired into CI, running on every pipeline change
- Access control enforced at retrieval time — pre-filtering by permission, not post-filtering the response
- Observability in place: queries, retrieved chunks, latency, and cost logged per request
- Security review completed (Section 8)
- Rollback plan and incident runbook written and reviewed by whoever's on call
Phase 3 — Scale-out
- A second use case onboarded reusing the existing retrieval infrastructure
- A documented, largely self-serve process for adding a new data source
- Cost attribution or chargeback model in place across consuming teams
- A governance function reviewing new use cases before they launch
Skipping straight to Phase 3 infrastructure before Phase 1 has validated a single use case is the most common — and most expensive — sequencing mistake in enterprise RAG programs.
7. Governance & risk checklist
- Every retrieved chunk is traceable back to a source document and an owner
- Source attribution is surfaced to end users wherever feasible
- A freshness policy defines the maximum age before a document is flagged for review
- PII or sensitive content is redacted or access-gated at retrieval time, not only at generation time
- Query and response logs are retained per your actual compliance requirement — not indefinitely by default
- Users have a way to flag an incorrect or harmful answer, and that feedback reaches the content owner
- Superseded or retracted content is removed from the index itself, not just from the source system
The failure mode governance is protecting against isn't usually a single catastrophic answer — it's a slow accumulation of stale, unattributed, or over-exposed content that nobody notices until an audit or incident forces the question.
8. Security checklist
RAG introduces an attack surface that standard application security reviews often miss: the retrieved content itself.
| Risk | Description | Mitigation |
|---|---|---|
| Indirect prompt injection | A document in the corpus contains text instructing the model to ignore its system prompt or exfiltrate data | Treat retrieved content as untrusted input; scan for instruction-like patterns; constrain tool access from RAG-triggered generations |
| Cross-tenant leakage | Retrieval returns chunks the querying user shouldn't have access to | Enforce document-level access control as a pre-filter on vector search, not a post-hoc check on the response |
| Vector store exposure | Embeddings can sometimes be partially reconstructed toward source content | Treat the vector store with the same access controls as the source data |
| Data residency violation | Embedding or LLM API calls send data to a provider/region outside compliance bounds | Confirm residency and retention terms for every provider in the pipeline, not just the primary LLM |
| Poisoned knowledge base | An untrusted process injects false or malicious content into the corpus | Access-control the ingestion pipeline; review new sources before they enter the index |
| Over-broad retrieval scope | A query returns chunks well outside the intended domain | Apply metadata filters and domain scoping before vector search runs, not after |
The single highest-leverage line item here for most enterprises: access control enforced at retrieval time. An application-layer check on the final response is not a substitute — by the time it runs, the model has already seen content it shouldn't have.
9. Cost model & ROI
| Driver | Scales with | Notes |
|---|---|---|
| Embedding | Chunks indexed (one-time) + queries (ongoing) | Usually the smallest line item |
| Vector storage | Chunks × embedding dimensions | Starts to matter past ~10M vectors |
| Retrieval compute | Queries/sec × top-K | Managed platforms often price per query |
| Reranking | Queries × candidates reranked | Most expensive per-call step if a cross-encoder is used |
| Generation (LLM tokens) | Queries × (context + output tokens) | Usually the dominant cost — a direct function of chunking |
cost_per_query ≈ embedding_query_cost + retrieval_cost + rerank_cost + (context_tokens_injected + output_tokens) × token_price
Because injected context tokens usually dominate this equation, chunk size and count aren't just a retrieval-quality decision — they're a direct line to unit economics. A chunking strategy that improves recall by returning more, larger chunks can quietly double cost per query. Model this before finalizing the strategy, not after the first invoice.
ROI framing: compare against the baseline established in Phase 0. RAG tends to pay off when query volume is high enough to amortize infrastructure cost, the manual alternative has a well-understood cost today, and the golden dataset shows retrieval quality clears a "good enough" bar. It tends not to pay off for low-volume, low-stakes use cases — those are often better served by long-context stuffing (Section 3).
10. Organizational playbook
RAG programs stall as often on unclear ownership as on unclear architecture. A minimal RACI:
| Activity | Product Owner | Platform/ML Eng | Data/SME Owner | Security |
|---|---|---|---|---|
| Use case scoping | R / A | C | C | I |
| Data source onboarding | C | R | A | C |
| Chunking & embedding strategy | I | R / A | C | I |
| Golden dataset creation | C | C | R / A | I |
| Evaluation & regression testing | I | R / A | C | I |
| Access control design | C | R | I | A |
| Production launch sign-off | A | R | C | R |
The recurring gap in practice is the Data/SME Owner column — the role most often left unassigned, and the one whose absence shows up months later as silently stale or wrong retrieval results nobody is accountable for.
11. Common failure patterns
Scope creep before validation
Onboarding five data sources before proving the pipeline works for one.
No golden dataset
"It seems good" is not a launch criterion, and gives you nothing to regression-test against.
Access control bolted on later
Retrieval that doesn't respect document-level permissions from day one.
Retrieved content treated as trusted
No defense against instructions embedded in indexed documents.
No owner for stale content
The index rots silently until a user notices an answer is wrong.
Chasing model upgrades
Most RAG quality problems are retrieval failures, not generation failures — check retrieval first.
No cost visibility until the invoice
Token costs from injected context compound fast at real query volume.
Platform-first, use-case-second
Building shared infrastructure before a single validated use case justifies it.
No path to say "I don't know"
Answering on low-confidence retrieval erodes trust faster than an occasional honest non-answer.
Ignoring embedding lock-in
Switching embedding models later means re-indexing the entire corpus.
12. RAG platform decision scorecard
A reusable template for evaluating a build-vs-buy or vendor decision. Score each criterion 1–5 for every option under consideration; weight by what matters most in your context.
| Criterion | Weight | Score (1–5) | Notes |
|---|---|---|---|
| Data sensitivity / residency fit | — | — | — |
| Alignment with existing cloud investment | — | — | — |
| Native hybrid search support | — | — | — |
| Self-serve onboarding for new data sources | — | — | — |
| Cost predictability at scale | — | — | — |
| Vendor lock-in risk | — | — | — |
| Operational overhead to run | — | — | — |
| Time to first working pilot | — | — | — |
Weight the criteria before scoring any option — it's easy to unconsciously weight toward whichever platform is already familiar.
- RAG maturity is organizational before it's technical — locate your actual stage with Section 2 before assuming you're further along than you are.
- Confirm RAG is the right pattern before investing in it. Long-context and fine-tuning solve different problems.
- Readiness gaps in data, team, or governance are the leading predictor of a stalled pilot — check Section 4 before architecture discussions start.
- Access control enforced at retrieval time, not the application layer, is the single highest-leverage security control.
- Cost is a function of tokens injected per query, which is a function of chunking strategy — model it before you commit.
- Sequence matters: one validated use case before shared platform infrastructure, every time.