Microsoft Foundry - Enterprise AI

00 · Orientation

Something for everyone

DEVELOPERS Builders

Choose and compare models: catalog, leaderboard, side by side
Prototype fast in model and agent playgrounds
Build agents with tools, MCP, memory, human in the loop, SDKs
Ground on Foundry IQ; evaluate, trace and fine-tune

IT ADMINS Platform owners

One Foundry resource, many projects: 1:1 and 1:N patterns
Hub and spoke routing through an APIM gateway
RBAC roles, regions, quota and tokens-per-minute limits
Cost overview and tagging; provision with Bicep and Terraform

SECURITY Risk & compliance

Guardrails: Prompt Shields, content filters, PII blocklists
Red teaming: AI Red Teaming Agent on PyRIT, attack success rate
Governance: Azure Policy deny rules on model deployments
Control plane: Entra Agent ID, Defender, Purview, continuous evals

1 What it is› 2 Portal› 3 Deploy› 4 Inference› 5 Agents› 6 Foundry IQ› 7 Control plane› 8 Wrap

Build for business value outcomes, then govern it at scale.

01 · What it is

One platform

◆ Microsoft Foundry

Observability

Tracing · Monitoring Evaluation · Experimentation

Frameworks & SDKs

Foundry SDKAgent Framework LangGraphLangChain LlamaIndexCrewAI GitHub Copilot SDK

Foundry Agent Service the runtime

Managed orchestrationConversations + state Network isolationOBO auth

Models

Microsoft & partners Fine-tuned Bring your own

Knowledge & Tools

Azure AI SearchBing SharePointFabric FunctionsLogic Apps OpenAPIMCP

Agent integrations

Foundry SDKREST Responses APIA2A protocol

Guardrails, security & governance

Prompt ShieldsContent filters PII detectionCustom blocklists Protected materialPolicy & compliance Cost managementEntra · Defender · Purview

BUILD Multi-agent · 1,400+ tools · Agent memory · Foundry IQ

OPERATE Observability · Agent 365 identity · Fleet ops

Unified breadth: build with agents, tools, memory and knowledge, then operate them with identity, observability and fleet governance, all behind one resource.

This is the whole platform on one slide, and it is a faithful redraw of Microsoft's own architecture diagram with two corrections baked in. Foundry unifies three jobs, running AI in production, building models, and building apps, under a single Azure resource-provider namespace, so RBAC, networking and policy are shared rather than stitched together. Reading the stack: observability and your choice of framework on top; the Foundry Agent Service runtime in the middle; models, knowledge and tools, and integration protocols beneath; guardrails, security and governance as the base. Two corrections from the original Microsoft slide: the integration surface is the Responses API now, not the Assistants API, which is retiring, and Semantic Kernel plus AutoGen have merged into the Microsoft Agent Framework. The ribbon under the diagram is the why-it-matters: the headline capabilities split into build (multi-agent orchestration, a catalogue of over fourteen hundred tools and agent memory, both in preview, and Foundry IQ for grounded knowledge) and operate (real-time observability, an Agent 365 Entra identity per agent, and fleet governance in the Operate hub). The deck follows that shape: build first, then govern. We unpack each layer over the next two hours.

02 · Portal

The portal is organised around jobs to be done

◆ Microsoft Foundry / proj-foundry-core Search with AI (Ctrl + K) New Foundry HomeDiscoverBuildOperateDocs ☼ ✉ ☉

Welcome back

Start building ›

Project endpoint proj-foundry-core.services.ai.azure.com Region Sweden Central API key auth disabled

HomeLand, resume recent work, quick start.

DiscoverNetflix-style catalog of models and tools.

BuildCreate agents, apps and workflows.

OperateAdmin and fleet view across projects.

DocsDocumentation, in context, never leave.

Flip on New Foundry at ai.azure.com and you get five tabs by job. Everything sits inside a Foundry resource (a subscription + resource group: the billing and governance boundary).

Sign in at ai.azure.com and toggle on New Foundry. The portal is deliberately organised around jobs to be done, not Azure resource types. Home is your landing and recent work. Discover is the catalog. Build is where developers create agents, apps and workflows. Operate is the admin and fleet view across subscriptions and projects, that is the control plane we deep-dive later. Docs is integrated so you stay in context. Top left is the current project, click to switch; the search bar is backed by Ask AI, which is itself several agents. The one structural thing to remember: a project always lives inside a Foundry resource, provisioned into a subscription and resource group, which is the billing, management and governance boundary. Note the new portal only shows assets created in the new experience, not classic ones.

03 · Discover

A model catalog you can compare, then deploy

◆ Microsoft Foundry / proj-foundry-core Search with AI (Ctrl + K) New Foundry HomeDiscoverBuildOperateDocs ☼ ✉ ☉

Discover what's possible1,900+ models · explore by provider, collection, leaderboard

Azure OpenAIAnthropicMicrosoftMetaMistral AIxAIDeepSeek

Model leaderboard	Quality	Safety	Throughput	Est. cost
gpt-5.2-codex ★	0.93	0.18%	32 t/s	$4.81
gpt-5.2	0.93	1.87%	60 t/s	$4.81
claude-opus-4-6	0.93	2.41%	43 t/s	$10.00
claude-sonnet-4-6	0.92	2.19%	61 t/s	$6.00
grok-4	0.91	1.41%	48 t/s	$5.20

Tools view: Foundry Tools catalog: remote & local MCP servers, OpenAPI and A2A. Configure once, add to any agent or workflow.

Optimise the choice across quality, safety, throughput and cost, compare models side by side, then quick-deploy (global standard) straight from the card.

04 · Build

Where developers create and manage every asset

◆ Microsoft Foundry / project-admin-c2676f Search with AI (Ctrl + K) New Foundry HomeDiscoverBuildOperateDocs ☼ ✉ ☉

Agents Deployments Fine-tune Tools Knowledge Memory Data Evaluations Guardrails

New Foundry supports V2 agents only. Classic agents and the Assistants API are not carried over: save them as new agents.

Browse templates Code agent Create agent

Name	Version	Type	Description
contoso-bank-agent	1	prompt	Customer-service agent
contoso-pmo-agent	1	prompt	PMO knowledge-base agent
aria-rm-briefing-agent	1	prompt	Private-banking RM briefing

Build is where developers manage every asset they deploy or create, and it is the home of the model and agent playgrounds.

05 · Operate

And one tab to govern it all

◆ Microsoft Foundry / fleet Search with AI (Ctrl + K) New Foundry HomeDiscoverBuildOperateDocs ☼ ✉ ☉

PILLAR 1ControlsRuntime guardrails on inputs, outputs and tool calls.

PILLAR 2ObservabilityTracing, continuous evaluation, cost.

PILLAR 3SecurityEntra Agent ID, Defender, Purview.

PILLAR 4Fleet opsOne view across every project and cloud.

Operate is the Foundry Control to see, govern and act on every agent across the fleet.

06 · Pattern

One resource, many projects, two ways to slice it

Subscription› Resource group› Foundry resource› Project(s)

1 : 1 · dedicated one account, one team: a hard infrastructure boundary

◆ aif-spoke-alpha

project-alpha

For strict compliance or cost isolation, and to separate dev / test / prod.

1 : N · shared one account, many isolated project workspaces

◆ aif-spoke-multi

project-betaproject-deltaproject-gamma iqobscu

For teams on the same cost centre: cost-efficient, shared RBAC, per-project isolation.

The Foundry resource is the billing, networking and quota boundary.
The project is the isolated workspace (its own agents, data and connections).

The new resource model is two levels: a single Foundry resource, the account, contains one or more projects. The resource is the boundary for billing, networking, Azure Policy and model-deployment quota; the project is the isolated workspace, with its own agents, data and connections. So the first design decision is how you map teams to resources. One to one gives each team its own account, a hard infrastructure boundary, which you want for strict compliance or cost isolation and for separating dev, test and prod. One to many puts several teams as separate projects under one shared account, which is cost-efficient and the right default for teams on the same cost centre. The payoff, which the whole repo demonstrates, is that you absorb new capabilities, Foundry IQ, observability, content understanding, by adding projects to the shared account rather than spinning up new accounts.

07 · Architecture

Centralise inference behind one AI gateway

Models live only in the 'hub' Foundry accounts*

◆ aif-core

East US 2 · general

gpt-4.1-miniembeddings

◆ aif-research

Norway East · reasoning

o3-deep-research

◆ aif-oss

West US 3 · open-weights

Phi-4

↑ managed identity · routed by URL (most specific wins)

⇉ APIM gateway · apim-foundry

swaps client key → managed-identity token · per-team rate limits & quota · routes by URL to hubs

↑ each project: its own connection core-{team} + gateway key

Spoke 1 : 1 no models · deny policy

◆ aif-spoke-alpha

project-alpha

reaches models via the core-alpha connection

Spoke 1 : N no models · deny policy

◆ aif-spoke-multi

betadeltagammaiqobscu

each project: own key, own quota

* 'Hub' here means the central model-hosting Foundry accounts in this topology, not the legacy Foundry v1 'hub' resource type.

One Azure API Management gateway fronts every model. Spokes hold zero deployments: they call the hub through a per-team key, so cost, content filters and observability stay unified.

This is the reference architecture for running Foundry at scale: a centralised AI gateway. One Azure API Management instance is the single entry point for all model traffic, and it routes by URL to the right regional hub account, general-purpose in East US 2, a reasoning model in Norway East, open-weights in West US 3. The crucial rule: model deployments live only in the hubs. Every spoke, whether a one-to-one dedicated account or the shared one-to-many account, holds zero local deployments and reaches models only through the gateway, using its own APIM subscription key for independent rate limiting and usage tracking. The gateway also swaps the client key for a managed-identity token, so the hub accounts have key auth disabled and only the gateway can call them. Net effect: one place for cost, content filtering, quota and observability, instead of fragmentation across teams.

08 · Governance

Make the pattern impossible to break

// Azure Policy: deny-model-deployments
{
  "if": {
    "field": "type",
    "equals": "Microsoft.CognitiveServices/accounts/deployments"
  },
  "then": { "effect": "deny" }
}

rg-foundry-coreEXEMPT

rg-foundry-spoke-alphaDENIED

rg-foundry-multiDENIED

Deploy a model into a spoke and Azure returns
RequestDisallowedByPolicy. Spokes can still use models through the gateway, just never deploy their own.

A one-rule Azure Policy denies Microsoft.CognitiveServices/accounts/deployments in spoke resource groups. The hub stays exempt, so the architecture cannot drift.

Architecture diagrams do not enforce themselves, so this is the governance that does. A custom Azure Policy, deny-model-deployments, blocks the creation of any deployment resource in the spoke resource groups; the hub resource group is exempt because that is where the platform team deploys the shared models. The rule is genuinely this small: if the resource type is a Cognitive Services account deployment, deny. Try to deploy a model into a spoke and you get RequestDisallowedByPolicy. Spokes can still use the models through the gateway, they just cannot deploy their own, which is exactly what keeps cost, content filters and observability from fragmenting. This is the kind of guardrail an IT or platform team sets once at the management-group or subscription level.

09 · Deploy

Deploying a model: pick a type, get an endpoint

Standard pay per token

Elastic, no commitment, billed per token
The default: global standard, one click
Best for spiky or early workloads

Provisioned / PTU reserved capacity

Predictable latency and throughput
One PTU pool can be shared across different provisioned models
Best for consistently high utilisation, latency-sensitive production

Global

max throughput, data may leave region

Data Zone

stays within a geography (EU / US)

Regional

pinned to one region for residency

Defaults vs custom: accept global standard + default quota, or customise SKU, quota (TPM) and guardrails. Partner models (Llama, Claude) need an Azure Marketplace subscription; models sold directly by Azure do not.

A deployment is a model + a deployment type. The type sets the cost model (per-token vs reserved) and the data-residency and throughput guarantees; quota is tracked per region and subscription, as PayGo (standard) or PTU (provisioned).

Deploying a model is choosing the model plus a deployment type, and the type is the decision that matters. Standard is pay-per-token and elastic, the default being global standard with one click, ideal for spiky or early workloads. Provisioned, or PTU, is reserved capacity for predictable latency and throughput, and PTU is fungible, you can move that capacity across provisioned deployments. Orthogonal to that is scope: global maximises throughput but data may leave the region; data-zone keeps it within a geography like the EU; regional pins it to one region for residency. You can take defaults or customise the SKU, quota and guardrails. Two gotchas for admins: quota is tracked as tokens per minute per region and subscription, and partner models such as Llama or Claude require an Azure Marketplace subscription, whereas models sold directly by Azure do not.

10 · Endpoints

One resource, three endpoint surfaces

OpenAI SDK

*.openai.azure.com/openai/v1

Full OpenAI API surface: chat completions, embeddings, Responses, fine-tuning. No agents or evaluations.

key or token

Foundry SDK

*.services.ai.azure.com/api/projects/*

Foundry-native: agents, evaluations, connections, tracing. Responses API on its /openai route.

token only · Entra

Foundry Tools SDKs

*.cognitiveservices.azure.com

The other AI services: Speech, Vision, Language, Content Safety (formerly Azure AI Services).

key or token

One Foundry resource exposes all three · an Azure OpenAI resource has only /openai/v1

A Foundry resource is multi-surface: the OpenAI endpoint for raw inference, the project endpoint for Foundry-native agents and evals (token-only via Entra), and Cognitive Services for the other AI tools. Match the SDK to the endpoint.

A quick reference that resolves a common point of confusion. A single Foundry resource exposes three endpoint surfaces, and you pick the SDK to match. The OpenAI host, resource.openai.azure.com/openai/v1, is the pure OpenAI-compatible surface: chat completions, embeddings, Responses and fine-tuning, with maximum client compatibility, but no Foundry-native features. The project host, resource.services.ai.azure.com/api/projects, is the Foundry SDK surface for agents, evaluations, connections and tracing, and it requires token auth via Entra, no keys. The Cognitive Services host, resource.cognitiveservices.azure.com, is the older multi-service surface, now called Foundry Tools, for Speech, Vision, Language and Content Safety. The key point: an Azure OpenAI resource only gives you the openai endpoint, whereas a Foundry resource gives you all three. Source: the Foundry API and SDKs reference, 04-03.

11 · Inference

Calling models through the gateway

Direct client gateway key

AzureOpenAI(base_url=gateway, api_key=key)

Chat completionsEmbeddingsDeep research

Full Azure OpenAI surface; the team holds the gateway key.

Foundry project client keyless

responses.create(model="core-alpha/gpt-4.1-mini")

Responses APIMulti-turnStreaming

Keyless via Entra; routes through the project's core-{team} connection, which speaks the Responses API only.

↓ both paths reach the models through one APIM gateway · managed identity onward ↓

Pick by surface and auth: the direct client (gateway key) for the full OpenAI surface, or the Foundry project client (keyless via Entra) whose core-{team} connection speaks the Responses API. Both reach the same models through the one gateway.

Once models are deployed centrally, teams call them two ways, both through the one APIM gateway. The direct Azure OpenAI client points at the gateway with a subscription key and gives you the classic surfaces: chat completions, embeddings and the deep-research model. The Foundry project client uses DefaultAzureCredential, so no keys, and because the project's gateway connection is an API Management connection it speaks the Responses API specifically: the path for agents, server-side multi-turn via previous_response_id, and a simpler input parameter. Both are keyless to the models themselves; key-based access to the hub accounts is disabled by policy. Multi-turn keeps state server-side, so you pass previous_response_id rather than resending history, though every prior input token is billed on each call. Streaming reads output_text.delta events for incremental tokens.

12 · Agents

An agent is a model, instructions and tools

Input

User messages

System events

Agent messages

→

Agent

LLM

Instructions

Tools

→

Output

Agent messages

Structured output

↓ tool call ↑ result

Tool calls

Retrieval

Actions

Memory

It takes unstructured input, reasons with the model under your instructions, calls tools mid-flight to retrieve or act, and returns a message or structured output.

13 · Lifecycle

Foundry is an assembly line for agents

Six stages, secure and testable end to end: models → customization → knowledge and tools → orchestration → observability → trust.

Microsoft frames Foundry as an assembly line for intelligent agents, six stages that take you from idea to production-ready. One, models, the raw intelligence. Two, customizability, fine-tuning, domain prompts, your own data. Three, knowledge and tools, enterprise knowledge like Bing, SharePoint and Azure AI Search plus actions like Logic Apps and Functions. Four, orchestration, the workflow layer that manages tool calls, conversation state, retries and logging, that is the magenta node because it is the part that ties everything together. Five, observability, logs, traces and evaluations. Six, trust, identity, RBAC, content filters, encryption and network isolation. The deck has already touched models and tools; this act is mostly stages three and four, and the control plane act is stages five and six.

14 · Runtime

The Agent Service runs the loop, not your code

your code · client.responses.create(model, input, agent_reference)

↓

Foundry Agent Service · on the Responses API

1Load the agent version: system prompt, tools, model binding

2Persist the conversation: resume via previous_response_id

3Call the model through the gateway, RBAC-scoped, no keys

4Dispatch tools: Code Interpreter, File Search, MCP server-side

5Stream and trace: output deltas plus OpenTelemetry spans

6Content safety on input and output

↓

grounded result + citations · FunctionTool calls handed back to you for human-in-the-loop

For a prompt agent there is nothing to deploy: you submit a request and the service owns thread state, tool dispatch, retries and content safety server-side.

A common misconception is that you run the agent loop. You do not. The Foundry Agent Service is the runtime. You submit a request via the Responses API against an agent reference, and the service does the rest: loads the agent version with its prompt, tools and model binding; persists the conversation so you resume with previous_response_id rather than resending history; calls the bound model through the project's gateway, RBAC-scoped with no admin keys; dispatches tools, where Code Interpreter, File Search and MCP run server-side while your FunctionTool calls are handed back to your code, which is the basis of human-in-the-loop; streams output deltas while emitting OpenTelemetry spans; and applies content safety on input and output. For a prompt agent there is no container, no pod, nothing to deploy. Which raises the obvious question: what if you DO want to bring your own runtime?

15 · Knowledge & tools

What an agent can know and do

Knowledge (grounding)

Azure AI Search

File search

Web / Bing grounding

Foundry IQ knowledge base

Work IQ (Microsoft 365)

Microsoft Fabric (Fabric IQ)

SharePoint

Tools & actions

Code Interpreter

Function calling

Azure Functions

Logic Apps connectors

Image generation

Browser / computer use

Custom & protocols

Model Context Protocol

OpenAPI tools

Agent-to-Agent (A2A)

Toolbox (curated set of tools)

Tool catalog

Skills (reusable instructions)

Agent memory

Grounding sources and actions plug in by configuration, not custom plumbing. MCP and OpenAPI let an agent call almost any tool server or API, and the IQ family (Foundry, Work, Fabric) grounds it in your data.

What an agent can know and do, configured rather than coded. Knowledge for grounding: Azure AI Search, uploaded files, the web via Bing, SharePoint, and your enterprise data through the IQ family - Foundry IQ over your Azure and web sources, Work IQ over live Microsoft 365 data like email, chat, meetings and files through Microsoft Graph with permissions enforced, and Fabric IQ over your Fabric lakehouse and semantic models. Tools and actions: a code interpreter, your own functions, Azure Functions and Logic Apps connectors, image generation, and browser or computer use. Extensibility: the Model Context Protocol and OpenAPI let an agent call almost any tool server or external API, agent-to-agent handles multi-agent calls, the Toolbox and tool catalog package and govern tools, skills add reusable behavioral guidance as versioned files, and agents have built-in memory. The headline: knowledge and tools are configuration, not custom plumbing. These capabilities move between preview and GA quickly, so the status labels are intentionally left off - check the docs for the current state.

16 · Hosted

When you want to bring your own runtime

Prompt agent declarative, runs on the Agent Service

You define model + instructions + tools
Nothing to deploy, no container
The service owns the loop

Hosted agent bring your own code and runtime

Deploy from source (Foundry builds it, no Docker), or ship your own container
Foundry provisions compute and a dedicated endpoint
Per-session VM sandbox, scale to zero, OpenTelemetry
Gets its own Microsoft Entra agent identity

Bring your framework Microsoft Agent Framework LangGraph OpenAI Agents SDK Anthropic Agent SDK GitHub Copilot SDK your own code

Most agents need no container. For your own runtime, hosted agents run any framework with a managed identity and a sandbox: deploy straight from source (no Docker), or bring your own container.

17 · Hosted SDK

A whole agent harness inside one container

Foundry hosted runtime · your container

main.py + InvocationAgentServerHost · outer loop

POST /invocations→one user turn→stream events out as SSE

spawned Copilot CLI subprocess · inner agentic loop

model call→tool?→run shell / python→feed result back↻idle

inference → ◆ your Foundry gpt-5.x via managed identity · /openai/v1/responses · no secrets in the container

main.py is a thin invocations shell; the real reason-act-observe loop runs in a spawned CLI subprocess. Inference is bring-your-own-key to your own Foundry model, so no secrets ship in the image.

Here is the shape of a serious hosted agent: the GitHub Copilot SDK, the engine behind Copilot CLI, running inside a Foundry container. There are two nested loops. The outer loop is main.py with the invocations server host: Foundry calls POST /invocations once per user turn, and main.py just bridges events out as a stream. The inner loop, the actual agentic reason-act-observe cycle, runs inside a Copilot CLI subprocess that the SDK spawns: model call, decide if a tool is needed, run shell or Python, feed the result back, repeat until idle. So main.py is a thin protocol shell, not the brains. And inference is bring-your-own-key: the SDK points at your own Foundry model over the Responses API using the container's managed identity, so no GitHub or OpenAI keys ever ship in the image. Sovereign, observable, and the same demo you would run locally.

18 · Foundry IQ

Grounded knowledge, as a managed layer

Knowledge sources

Azure Blob · OneLake

SharePoint

Existing search indexes

Web (Grounding with Bing)

→

◆ Knowledge base

One endpoint, shareable across many agents. Permission-aware: honours ACLs and Purview sensitivity labels under the caller's identity.

engine: Azure AI Search agentic retrieval

→

Agents attach via MCP

prompt agent

multi-agent system

Foundry IQ · enterprise knowledge Fabric IQ · analytics Work IQ · M365

Foundry IQ is a managed knowledge layer over Azure AI Search. You build a knowledge base from your sources once and any agent grounds on it, with citations and permissions enforced.

Foundry IQ, announced at Ignite, is Microsoft's managed knowledge layer for agents, and underneath it is Azure AI Search's agentic retrieval. The core object is a knowledge base: a single, shareable endpoint composed of one or more knowledge sources, indexed ones like Blob, OneLake, SharePoint and existing search indexes, and remote ones like web via Grounding with Bing. Agents attach it as a tool over MCP. Two things to stress for this room: it is permission-aware, it enforces ACLs and honours Purview sensitivity labels under the caller's Entra identity, so retrieval respects who is asking; and Microsoft's own benchmark puts agentic retrieval about thirty-six percent above single-shot RAG on response quality. Foundry IQ is the enterprise-knowledge sibling of Fabric IQ for analytics and Work IQ for Microsoft 365. Next, how the LLM actually lives inside the retrieval path.

19 · Retrieval

The model lives inside the search, not just after it

Build the index

documents3k arxiv-nlp abstracts → embedtext-embedding-3-large · 3072d → indexHNSW vector + semantic + ACL trim

◆ Knowledge source registers the arxiv-nlp index

kb-fast

minimal effort · no LLM

direct index-driven retrieval

fastest, no model quota

kb (standard)

low effort · one LLM pass

gpt-4.1-mini (APIM) plans sub-queries

better relevance, multi-turn

Both return raw chunks (EXTRACTIVE_DATA) for the agent's own LLM; each KB auto-exposes an MCP endpoint. Effort: minimal → low → medium → high.

Two knowledge bases over one index: kb-fast (minimal effort, no LLM) for speed, kb (standard) (low effort, gpt-4.1-mini) for relevance. The effort knob puts the model inside the search, trading depth for latency and cost.

This is the concrete Foundry IQ setup the demo builds. First the index, notebooks 10-01 and 10-02: three thousand arxiv NLP abstracts are embedded by text-embedding-3-large at 3072 dimensions through the gateway and written into an Azure AI Search index with HNSW vectors, a semantic configuration and a group-ids field for ACL security trimming. Then in 10-03 the index is registered as a knowledge source and exposed as two knowledge bases that differ on one knob, retrieval reasoning effort. kb-fast uses minimal effort: no LLM at all, direct index-driven retrieval, fastest and no model quota. kb standard uses low effort: one LLM planning pass with gpt-4.1-mini through the gateway that decomposes the question into sub-queries, for better relevance on complex or multi-turn questions. Both return raw chunks, extractive data, for the agent's own model to reason over, rather than the knowledge base synthesizing the answer itself. Each knowledge base auto-exposes an MCP endpoint the agent connects to. The effort knob runs minimal, low, medium, high: that is the model living inside the search.

20 · Control plane

One agent is easy. A fleet is the hard part.

Data plane

Agents in action

Chatting, calling tools, retrieving data, generating responses. The work running.

Control plane

See, govern, act

One surface for identity, policies, security, observability and cost, across every project and cloud.

Risk: Prompt injection

Untrusted content in a tool result hijacks the agent's instructions.

Risk: Task drift

The agent quietly does something other than what it was asked.

Risk: Data leakage

Access plus a confused instruction plus an outbound channel equals exfiltration.

Agents add failure modes apps never had, and they compound as you add tools and data. Their underlying intelligence is a probabilistic, non-deterministic LLM, not deterministic code, so the same input can behave differently each time.

Here is the pivot of the whole talk: from building one agent to running a fleet. In cloud terms, the data plane is the work running, agents chatting, calling tools, retrieving data. The control plane is the management surface above it: see, govern, act, one place for identity, policy, security, observability and cost, across every project and every cloud. Why you need it: agents add failure modes traditional apps never had. Prompt injection, where untrusted content in a tool result hijacks the instructions. Task drift, where the agent quietly does the wrong thing. Sensitive-data leakage, where access plus a confused instruction plus an outbound channel equals exfiltration. And these compound as you add tools, data and multi-agent workflows. The canonical story: a company had about ten agents in production, one retrieved a document carrying a hidden instruction, and it wove customer names and transactions into its answers for three days before anyone noticed. No single team owns this, which is why it needs a dedicated surface.

21 · The pillars

What the control plane brings together

CONTROLSGuardrailsPrompt Shields, content filters and blocklists, validated by red teaming.

OBSERVABILITYSee insideTracing, continuous evaluation on live traffic, per-agent cost.

SECURITYIdentity & dataEntra Agent ID, Microsoft Defender, Microsoft Purview.

FLEET OPSAt scaleOne inventory and to-do list across projects, frameworks and clouds.

The control plane brings four essentials into one surface: runtime guardrails, observability, agent security and fleet operations, across every project, framework and cloud.

22 · Controls

Guardrails on inputs, outputs and tool traffic

User input

↓

INPUT Prompt Shields (jailbreak + indirect) · content filters · custom blocklists

↓

Agent + model

↓

TOOL CALL / RESPONSE task adherence on the call · indirect injection on the response

↓

OUTPUT content filters · protected material · groundedness · sensitive data / PII · custom blocklists

↓

Response

A guardrail is a set of controls at four points: input, tool call, tool response, and output. The tool-call and tool-response checks are the agent-specific part, catching indirect prompt injection before the agent acts.

Controls, the first pillar. A guardrail is a named set of controls; each control is a risk, an intervention point and an action. Guardrails fire at four points in a turn, in order. On the user input: Prompt Shields for jailbreak and indirect attacks, content filters, and custom blocklists. At the tool call, before the tool runs: task adherence, checking the planned call and its arguments match the user's intent. At the tool response, when data comes back: Prompt Shields for indirect, cross-prompt injection, the genuinely new defence, malicious instructions hidden in a document, web page or tool result, caught before the agent acts on them. And on the output: content filters, protected material, groundedness against the retrieved context, and sensitive data or PII detection. Note that groundedness is currently model-level and not yet enabled for agents. In the demo we paste an injection and a fake SSN into the bank agent; the input shield blocks the injection before the model ever sees it, at zero token cost.

23 · Observability

Trace it, evaluate it, cost it

Tracing OpenTelemetry traces, prompt to model to tool, into Application Insights.

Walk back any run, step by step
Auto-traced for Agent Framework, LangChain, LangGraph

Continuous evaluation Score live production traffic, not just a pre-ship test suite.

Set a threshold, alert when quality drops
Groundedness, task adherence, tool-call accuracy

Cost Per-agent token spend across the fleet, near real time.

Agents burn tokens fast: watch them
Sort the fleet by cost or error rate

You cannot human-review every step, so you evaluate live traffic against a threshold and trace what crosses it, while watching cost per agent.

Observability, the second pillar. You cannot put a human on every agent step, that defeats the purpose, so the model is: trace everything, evaluate continuously, and watch cost. Tracing is OpenTelemetry, prompt to model inference to the specific tool call, landing in your Application Insights, and it is automatic for the Microsoft Agent Framework, LangChain and LangGraph. Continuous evaluation runs against real production traffic, not just a pre-ship suite: set a threshold and get alerted when groundedness or task adherence drops, with agent-specific evaluators like tool-call accuracy. And cost: per-agent token spend across the fleet, because agents burn tokens fast. In the demo we open a code-interpreter trace and an evaluation run in the portal.

24 · Security

Every agent gets a real identity

Entra Agent ID Publish an agent and it gets a Microsoft Entra identity, automatically.

Access control like any principal
Ownership: know who to call at 2am
Lineage across its lifecycle

Microsoft Defender AI security posture and threat detection extended to agents.

Attack-path analysis, gap recommendations
A jailbreak surfaces as a security alert

Microsoft Purview Agent interactions available for audit and compliance.

Org-wide content-safety policies
Sensitivity labels honoured

Controls = each developer configures

Opt in or out on your own agent, e.g. Prompt Shields, content filters, a PII blocklist.

Policies = the organisation mandates

"Every agent must have indirect-injection protection on", scanned continuously.

Publishing an agent issues an Entra identity (app + object id) for access, ownership and lineage.
Policies turn a developer's optional control into an org-wide mandate.

Security, the third pillar, and the headline is identity. The moment you publish an agent it gets a Microsoft Entra identity, an application id and an object id, automatically, with no extra steps. That buys three things: access control, you govern it like any user or service principal; ownership, it is tied to a named team so when it breaks at 2am you know who to call; and lineage, the identity follows the agent through its lifecycle. Defender extends to agents with posture management, attack-path analysis and threat detection, so a jailbreak shows up as a security alert with full context. Purview makes agent interactions auditable and honours sensitivity labels. And the key distinction: controls are what each developer configures on their own agent, opt in or out, whereas policies are an organisation-level mandate, every agent in this subscription must have indirect-injection protection on, scanned continuously across every agent in scope.

25 · Red teaming

Attack your own agents, on purpose

Find the holes before someone else does

Probe: automated attacks, every category and strategy
Score: an Attack Success Rate baseline, by technique
Harden: add or tighten guardrails where attacks land
Re-scan: the same suite, and watch the ASR fall
Gate: re-run on every release, in CI/CD

Attack strategies (PyRIT)

Base64ROT13Unicode confusablesCrescendomulti-language

Risk categories

HateViolenceSexualSelf-harm

The AI Red Teaming Agent (built on PyRIT, preview) runs automated adversarial scans and scores an Attack Success Rate, so you can prove a guardrail policy actually moved the needle.

Guardrails are defence; red teaming is offence, and you want both. The AI Red Teaming Agent, in preview, is built on Microsoft's open-source PyRIT. It runs automated adversarial scans against a model or agent endpoint, scores each attack-response pair, and produces a scorecard with Attack Success Rate as the headline metric, broken down by attack technique and risk category. Strategies include obfuscation like Base64, ROT13 and Unicode confusables, the multi-turn Crescendo attack, and multi-language prompts; categories are the four content-safety harms: hate, violence, sexual and self-harm. The powerful move, shown in the demo, is running the same scan before and after attaching a guardrail policy: the attack-success-rate drop is your evidence the control works. The bars here are illustrative; your numbers depend on your agent and corpus.

26 · Fleet ops

One to-do list for every agent, any cloud

Fleet to-do

Jailbreak attempt blocked · contoso-bank-agent

Eval below threshold · aria-rm-briefing-agent

Policy gap: indirect injection off · 2 agents

Cost spike · contoso-pmo-agent, +180% today

Error rate 100% · contoso-bank-agent

2 agents in Unknown state · github-copilot

Assets · Agents

Name	Source	Project	Status	Errors	Tokens	Runs
github-copilot	Foundry	project-copilot-sdk	Unknown	0.00%	-	18
aria-rm-briefing-agent	Foundry	project-admin-c2676f	Running	1.92%	153.8K	52
contoso-bank-agent	Foundry	project-admin-c2676f	Running	100.00%	-	40
contoso-pmo-agent	Foundry	project-admin-c2676f	Running	0.00%	15.3K	4
SpaceExpert	Foundry	project-alpha-mesh	Running	0.00%	860	3
arxiv-nlp-agent	Foundry	iq-project	Running	0.00%	27K	5
team-beta-agent	Foundry	project-beta-c2676f	Running	0.00%	56	1
team-delta-agent	Foundry	project-delta-c2676f	Running	0.00%	56	1
claims-triage	AI Gateway	LangGraph	Running	0.00%	4.1K	12
legacy-helpdesk	AI Gateway	AWS	Blocked	-	-	0

Operate is a to-do list for the fleet: blocked jailbreaks, evals below threshold, policy gaps. External agents (e.g., AWS) join the same view by routing through the AI Gateway.

Fleet-wide operations, the fourth pillar. The Operate view is best thought of as a to-do list for the whole fleet: blocked jailbreak attempts, evaluation scores below threshold, policy-compliance gaps, cost spikes, sortable by error rate or token cost across every project. The assets inventory lists every agent with its source, status, cost and risk, and from an alert you jump straight into that agent's build experience to fix it, no context switching. The big one for this room: non-Foundry agents, a LangGraph app, something on AWS, join the exact same view by routing their traffic through the AI Gateway, which is Azure API Management, and you can block or unblock them at the gateway even though Foundry does not run their infrastructure.

27 · Provisioning

The admin essentials

Four Foundry roles

Foundry User · build & call: models, agents, evals, data plane

Project Manager · manage projects, assign User; full data plane

Account Owner · deploy, connections, quota; no data plane

Owner · unrestricted: full control + data plane

tip: assign to groups, not people

Two-layer RBAC

Layer 1 Foundry plane · portal, projects, build

Layer 2 Cognitive Services plane · content filters, quota, keys

gotcha: the second layer is commonly missed

Regions & quota

Pick by model availability + residency

Quota is TPM per region + subscription

Cost & SDK

Platform is free; pay per deployment

No hard spend cutoffs · budget alerts

SDK: azure-ai-projects + OpenAI client

For platform owners: four roles (assigned to groups), a two-layer RBAC model, TPM quota by region and subscription, and a free platform you meter at the deployment level.

One reference card for the platform owners in the room. Four built-in Foundry roles, recently renamed: Foundry User to build and call, Project Manager to manage projects, Account Owner for deployment and the control plane, and Owner for unrestricted access, and you assign them to Entra groups, not individuals. RBAC is two layers that are configured separately: the Foundry plane for portal and build, and the underlying Cognitive Services plane for content filters, quota and keys, and the second layer is the one people forget. Regions are chosen by model availability and data residency, quota is tokens per minute per region and subscription. Cost: the platform itself is free, you pay at the deployment level, there are no hard spending cutoffs so you wire up budget alerts, and the SDK story is the unified azure-ai-projects package plus the standard OpenAI client.

28 · Why Foundry

One platform, two jobs done well

Build

Deploy any model behind one governed gateway
Agents on the Responses API and one SDK
Knowledge and tools by config, MCP-native
Hosted runtimes in the framework you choose
Grounded, cited answers with Foundry IQ

Operate

A Microsoft Entra identity for every agent
Guardrails on inputs, outputs and tool traffic
Tracing and continuous evaluation on live traffic
Red teaming that measures attack success
One fleet view across projects and clouds

Build with rich primitives and govern them from the same place.

Pulling it together. The reason Foundry is worth your attention is not any single feature, it is the seam between build and operate. On the build side: deploy any model behind one governed gateway; build agents on the Responses API with one SDK; add knowledge and tools by configuration, MCP-native; host your own runtime in whatever framework you like; and ground answers with Foundry IQ. On the operate side: every agent gets a real Entra identity; guardrails sit on inputs, outputs and tool traffic; you trace and continuously evaluate live traffic; you red-team to measure attack success; and you see the whole fleet, including non-Foundry agents, in one place. Most platforms give you one half. Foundry's bet is that you need both, in the same flow.