AI Architecture

Enterprise AI Architecture & LLMOps — AI Gateway, AI Mesh, Model Registry, and the Platform Decision

The AI Architect's Blueprint · Blog 4 of 5

Previously: Blog 3 — Agentic AI, MCP & Multi-Agent Systems →

TL;DR — Key takeaways

→The AI Gateway is the control plane for all model traffic. Without one, you cannot govern costs, enforce policy, or manage model diversity at enterprise scale.

→The AI Mesh solves the internal communication problem — giving AI services a standard way to discover each other, communicate securely, and fail gracefully as your agent ecosystem grows.

→A Model Registry is your governance foundation. You cannot do responsible AI without knowing what models you operate, what data trained them, who approved them, and what they are approved to do.

→LLMOps observability has five layers — from infrastructure metrics to drift detection. Evaluation is the hardest and most important practice, and the one most teams skip until something goes wrong.

→Azure AI Foundry, Vertex AI, and Amazon Bedrock are all enterprise-grade. The platform decision matters less than your abstraction strategy — and most large enterprises end up using more than one.

→The full enterprise AI stack has seven layers. Each layer has a job. None is optional at scale.

→Design for the hundredth version, not the first. The hardest part of enterprise AI is governing systems you built with teams you have not hired yet, on models that have not been released yet.

1. The hardest moment in enterprise AI

There is a moment every enterprise AI team knows well. The proof of concept worked. The demo impressed the steering committee. The model performed well in the sandbox. Then someone asked: when can we ship this to production?

That is when the real architecture work begins.

Blogs 1 through 3 covered how to build AI systems — models, retrieval, agents, protocols. This blog covers something different: how to run AI systems in production at enterprise scale. How to govern them, observe them, control their costs, enforce policy across them, and keep them working reliably when the models underneath them change.

This is the domain of enterprise AI architecture and LLMOps. It is not glamorous. It is where most enterprise AI initiatives succeed or fail.

Moving from prototype to production AI is not a scaling problem. It is a governance problem, an observability problem, and an organizational problem — wrapped around an engineering problem.

2. The enterprise AI stack — seven layers

Before going deep on individual components, it helps to see the full picture. Enterprise AI architecture has seven layers, each with a distinct job, each dependent on the one below it.

The seven-layer enterprise AI stack. Each layer has a job. Build from the bottom up — but instrument from the top down.

Layer 1 — Foundation Models & Platforms: The models themselves — OpenAI GPT-4o, Anthropic Claude, Google Gemini, Meta Llama, and any fine-tuned domain models your organization has built. Hosted on Azure AI Foundry, Vertex AI, Amazon Bedrock, or your own inference infrastructure.

Layer 2 — AI Gateway: The control plane. Every model call flows through it. Authentication, routing, cost management, guardrails, caching, and audit logging — applied uniformly across every model your organization uses.

Layer 3 — AI Mesh: The internal fabric. How AI services discover each other, communicate securely, and handle failures. Increasingly relevant as agent systems become the norm rather than the exception.

Layer 4 — AI Services: Your enterprise AI capabilities — RAG pipelines (Blog 2), agents and multi-agent systems (Blog 3), fine-tuned domain models, embedding services, re-rankers, and classifiers.

Layer 5 — Model Registry: The governance layer. Versioning, lineage, evaluation results, deployment status, and approval workflows for every AI asset your organization operates.

Layer 6 — LLMOps Observability: The visibility layer. Infrastructure metrics, LLM-specific metrics, distributed traces, output evaluation, and drift detection. The feedback loop that connects production back to development.

Layer 7 — Applications: The value layer. The copilots, assistants, automation workflows, and intelligent applications that end users interact with.

Every layer has a job. None is optional at scale. The question is not whether you need all seven — you do — but how quickly you need to invest in each as your AI footprint grows.

3. The AI Gateway — your control plane for model traffic

What the AI Gateway does

Think of an AI gateway as the API gateway for your AI layer — but purpose-built for the unique demands of LLM traffic.

A traditional API gateway handles authentication, rate limiting, and routing for REST APIs. An AI Gateway does all of that and adds: token budget management, prompt validation, content filtering, model failover, semantic caching, and cost attribution — all in real time, often with sub-100ms overhead.

The AI Gateway sits between your applications and the constellation of AI models your enterprise uses. Whether that is OpenAI, Anthropic, Azure-hosted models, self-hosted Llama variants, or fine-tuned domain models in your own VPC — the Gateway provides a unified control plane across all of them.

The AI Gateway is your control plane. Every model call flows through it — regardless of which provider, which team, or which application made the request.

Core Gateway capabilities

Unified authentication and authorization. Every model call is authenticated at the gateway. Downstream services never hold raw API keys. The gateway handles credential rotation, per-team scoping, and integrates with your existing IAM (Azure AD, Okta, AWS IAM) for role-based access to specific models.

Intelligent routing and fallback. When GPT-4o is unavailable, the gateway routes to Claude. When cost thresholds are hit, it downgrades to a cheaper model for low-priority tasks. When latency is critical, it selects the nearest endpoint. This is not load balancing — it is policy-driven routing based on model capability, cost, latency, and availability.

Token budget management. LLM costs scale with token consumption, not request count. The gateway tracks token usage per team, per application, per user — and enforces budgets in real time. No more surprise bills from a runaway agent loop consuming a month's budget overnight.

Prompt and response guardrails. The gateway inspects every prompt for PII, prompt injection attempts, jailbreak patterns, and policy violations before the request reaches the model. Response filtering handles output toxicity, confidential data leakage, and format validation. This is your last line of defense before model outputs reach users.

Semantic caching. Identical or semantically similar prompts return cached responses, reducing latency and cost for high-volume use cases like FAQ bots or repetitive document analysis.

Observability and audit logging. Every request, response, routing decision, and guardrail trigger is logged with full context: user, team, application, model, token count, latency, cost, and outcome. This feeds your LLMOps observability layer and satisfies the audit requirements that compliance teams will ask for.

Gateway patterns

Centralized gateway (hub). A single gateway cluster handles all AI traffic. Simple to operate, easy to govern — but can become a bottleneck and single point of failure.

Federated gateways (domain). Each business domain runs its own instance, governed by central policy templates. More resilient and closer to the data, but requires strong policy synchronization.

Sidecar gateway (mesh). Gateway logic runs as a sidecar process alongside each AI-consuming service, as part of the AI Mesh. Maximum flexibility, zero single point of failure — but operationally complex.

4. The AI Mesh — the internal fabric for AI services

Why the gateway is not enough

The AI Gateway solves the external control plane problem — how applications call AI models. As your enterprise AI ecosystem matures, you face a different challenge: how do AI components discover each other, communicate securely, and stay resilient when your agent workflows span multiple services, teams, and runtimes?

This is the domain of the AI Mesh.

The AI Mesh is an architectural pattern — borrowing from service mesh concepts — that provides service discovery for AI capabilities, secure communication between AI components, policy enforcement at the component level rather than just the edge, distributed tracing across multi-hop AI workflows, and circuit breaking for AI service chains.

AI Mesh vs. AI Gateway

These are complementary layers, not competing concepts.

The AI Gateway handles traffic between your applications and AI models — its primary concerns are cost, policy, and authentication. The AI Mesh handles traffic between AI components themselves — its primary concerns are discovery, resilience, and observability across service-to-service calls.

Think of the AI Gateway as what your applications call, and the AI Mesh as what your AI components use to talk to each other — especially relevant as multi-agent systems (Blog 3) become the norm.

MCP as an AI Mesh primitive

If you read Blog 3, you will recognize the Model Context Protocol (MCP) as a natural building block for the AI Mesh. MCP standardizes how AI agents discover and invoke tools and data sources. An AI Mesh built on MCP gives you a common protocol layer across AI services from different teams, vendors, and runtimes.

5. The Model Registry — your system of record for AI assets

Why you need a registry

By the time your enterprise has ten production AI models, you will wish you had a model registry from day one.

Without one, you face questions you cannot answer: which version of the customer sentiment model is in production? Is it the one trained on Q3 data or Q4? Who approved the deployment? When was it last evaluated? What is its performance on the EU customer segment? Does it contain any training data that is subject to a current legal hold?

A Model Registry is the system of record for your enterprise's AI models — tracking lineage, metadata, versions, evaluation results, approvals, and deployment status in a single governed repository.

What a registry stores

Model metadata. Name, version, description, owner, team, model type (LLM, classifier, embedding, re-ranker), base model used, training data description, framework and runtime requirements.

Lineage and provenance. Training pipeline run that produced this version, dataset versions used for training and evaluation, fine-tuning configuration, hyperparameters, parent model version for iterative fine-tuning chains.

Evaluation results. Benchmark performance on standard datasets, domain-specific evaluation results, bias and fairness audit outcomes, safety and alignment evaluation scores, business metric performance such as task completion rate or customer satisfaction impact.

Governance metadata. Data classification of training data, regulatory compliance flags (GDPR, HIPAA, SOC2), usage restrictions and approved use cases, approval workflow status and named approvers, full review and audit history.

For LLM-specific workloads, the registry needs capabilities beyond traditional ML: versioning prompt templates alongside models, storing system prompts and their history, tracking PEFT adapter weights separately from base models, and maintaining model cards that communicate capabilities and limitations to non-technical stakeholders.

6. LLMOps observability — seeing what your AI is actually doing

The observability gap

Traditional software observability is built on three pillars: metrics, logs, and traces. These tell you what your system did and how it performed. They do not tell you why your AI responded the way it did, whether the response was actually good, or how your model's behavior is changing over time.

LLM systems introduce a fourth pillar: evaluation. And without it, you are flying blind in production.

The five-layer LLMOps observability stack

The LLMOps observability stack. Layers 1 and 2 tell you what happened. Layers 3, 4, and 5 tell you why it matters — and warn you before it becomes a problem.

Layer 1 — Infrastructure metrics (the floor). CPU and GPU utilization, memory, inference latency, throughput in tokens per second, error rates, queue depth. This is standard observability extended for AI infrastructure. Tools: Prometheus, Grafana, Datadog, CloudWatch.

Layer 2 — LLM-specific metrics (the signal). Token consumption by team and application, cost per request, context window utilization, cache hit rates, model availability, p50/p95/p99 latency per model endpoint. Most of this is exported by the AI Gateway automatically.

Layer 3 — Trace and chain visibility (the story). For agentic and RAG systems, you need distributed tracing that understands AI semantics — not just HTTP spans. Which chunks were retrieved? What was the re-ranker score? Which tool did the agent call and what did it return? How many LLM hops did this user request take? Tools like LangSmith, Arize Phoenix, and Helicone provide this AI-native tracing.

Layer 4 — Output quality evaluation (the truth). This is the hardest layer and the most important. You need to continuously evaluate whether outputs are correct — does the answer match the facts? Faithful — does the response stick to retrieved context? Relevant — does it address what the user asked? Safe — does it avoid harmful or policy-violating content? Consistent — does similar input produce consistent output? Evaluation can be done offline against a test set, online by sampling live traffic for human review, or using LLM-as-Judge — a separate model evaluating the output of your production model.

Layer 5 — Drift detection (the warning system). Models drift silently. Input distribution drift — users asking different kinds of questions than the model was built for. Concept drift — the real world changes but the model does not know. Performance drift — accuracy degrades gradually until someone notices. Automated drift detection combined with evaluation baselines gives you early warning before users notice the problem.

Evaluation-driven development

The best LLMOps teams close the loop. Every production failure, every flagged evaluation, every user complaint feeds back into the evaluation dataset. Model changes are tested against this growing evaluation suite before deployment. Prompt changes go through the same evaluation gate as model changes.

This is the closest thing to test-driven development that the LLM world has produced — and it is what separates teams that manage their AI systems from teams that merely hope they are working.

7. Platform decision — Azure AI Foundry vs. Vertex AI vs. Amazon Bedrock

All three platforms are excellent. All three are evolving rapidly. The right choice depends heavily on your existing cloud footprint and organisational context — and most large enterprises end up using more than one.

Platform choice follows cloud footprint. Most enterprises use two or all three. The AI Gateway makes that practical.

Azure AI Foundry

Microsoft's unified enterprise AI platform, tightly integrated with Azure OpenAI Service, Azure ML, and the broader Microsoft ecosystem — Teams, Copilot Studio, Microsoft Fabric.

Strengths: Best-in-class access to OpenAI models under enterprise SLAs. Deep Microsoft 365 and Copilot integration for knowledge worker scenarios. Mature responsible AI tooling including content filtering and a responsible AI dashboard. Strong hybrid story via Azure Arc for on-premises deployments. Prompt Flow for orchestration, Azure AI Search for enterprise RAG.

Best for: Enterprises with significant Microsoft and Azure investment, Copilot-driven productivity scenarios, or teams that need the OpenAI model family under enterprise agreement terms.

Considerations: Most natural in Microsoft-centric environments. Tighter coupling to Microsoft's model selection, though third-party models are available via the model catalogue. Prompt Flow is maturing rapidly but still catching up to LangGraph for complex agentic workflows.

Google Vertex AI

Google Cloud's unified ML and generative AI platform, providing access to the Gemini model family alongside an increasingly capable agent builder toolchain.

Strengths: Gemini 2.x is highly competitive, particularly for long-context and multimodal tasks. Grounding with Google Search is a genuine differentiator for real-time knowledge tasks. Deep data platform integration with BigQuery and Looker. Agent Builder and Agent Garden for agentic deployments. TPU infrastructure for custom model training at scale.

Best for: Enterprises with heavy Google Cloud data infrastructure, long-context and multimodal use cases, or scenarios where Google Search grounding provides a meaningful accuracy advantage.

Considerations: Google's enterprise support and sales motion have historically been less mature than Microsoft's or AWS's. Model availability varies by region. Agent Builder is newer and less battle-tested than Azure's Copilot Studio ecosystem.

Amazon Bedrock

AWS's managed foundation model service, providing API access to a curated selection of models from Anthropic (Claude), Meta (Llama), Mistral, AI21, Cohere, Stability AI, and Amazon's own Nova and Titan families.

Strengths: Unmatched model diversity — switch between Claude, Llama, Mistral, and Amazon Nova in the same application. Bedrock Agents for agentic workflows with native AWS Lambda and S3 integration. Knowledge Bases for enterprise RAG with Amazon OpenSearch. Guardrails for content filtering. Strong security posture: VPC isolation, PrivateLink, data never leaves the AWS boundary. AWS enterprise agreement integration familiar to existing customers.

Best for: Enterprises already deeply invested in AWS, teams needing maximum model optionality, security-sensitive industries that benefit from AWS's compliance posture, and organizations wanting Anthropic Claude specifically under enterprise terms.

Considerations: Bedrock Agents is less expressive than LangGraph or Azure Prompt Flow for complex orchestration. The model catalogue approach means some constraints on bringing arbitrary self-hosted models. Cost management across model families requires explicit attention.

The honest answer about platform choice

The AI Gateway and AI Mesh make the platform decision less critical than it appears. When your applications call a unified gateway rather than cloud provider APIs directly, switching or adding a platform is a configuration change rather than an engineering crisis.

Design for portability. Treat all three platforms as interchangeable providers behind your gateway. Then make your primary platform choice based on where your data already lives, what your existing cloud agreements look like, and which model families are most relevant to your use cases.

8. Real-world application — financial services & retail use cases

Financial services — intelligent risk and compliance

A global bank wants an AI-assisted credit analysis system. Analysts submit loan applications, the system retrieves relevant financial data, researches the applicant's industry, analyses regulatory exposure, and produces a structured credit memorandum — with the analyst reviewing and approving before any recommendation is finalized.

Architecture decisions:

Platform: Azure AI Foundry — existing Azure enterprise agreement, Microsoft 365 integration for analyst workflows, and strong data residency guarantees required by the regulator
AI Gateway: Centralized gateway with strict PII guardrails on all prompts and responses, per-analyst token budgets, full audit logging for regulatory compliance
AI Services: RAG pipeline over internal credit policy documents, regulatory guidance, and historical loan performance data; fine-tuned summarization model for financial document analysis
Model Registry: All models require compliance team approval before production deployment, with full data lineage documented for regulatory examination
LLMOps: Output faithfulness evaluation on every credit memo (does the recommendation follow from the retrieved evidence?), human review sampling of 10% of outputs weekly, automated drift detection on input distribution
Human-in-the-loop: Every credit recommendation requires analyst sign-off before progressing — non-negotiable given the regulatory environment

Retail — intelligent inventory and supplier management

A large retailer wants an agent that monitors inventory levels, identifies stockout risks, drafts replenishment orders, communicates with suppliers, and updates the inventory management system — with human approval before any order is placed.