Your AI application is slow. Was it waiting for the model, running a tool, or stuck inside a nested agent execution?
Microsoft Fabric Eventhouse gives us somewhere to investigate that execution, compare it with other requests, and track tool failures and token usage in dashboards. The same stored events support both the individual investigation and the broader view.
I’ve built @saguiitay/eventhouse-ai-observability to get AI application activity into Eventhouse without changing how the application runs its agents. Once the data arrives, we can use KQL to answer those questions ourselves.
Let’s follow one small example: an agent asks a tool to calculate a square, then makes another model request to explain the result. The arithmetic isn’t interesting. Figuring out where the time went is.
Contents
Why logs aren’t enough
A log saying “tool started” is useful. A provider summary saying “800 input tokens” is useful too. Neither necessarily tells us which model request belonged to which execution.
An AI session can contain several agent turns. A turn can involve multiple model requests and tool calls. Tools can overlap, and an invoking tool can start a subagent. A session’s lifetime is therefore different from the duration of an individual execution.
For our square example, I want to know whether the tool ran, how long it took, and whether the request after it failed. If another execution overlaps, I also want to keep their timings separate.
We need operation IDs, timestamps, and parent relationships in the stored events. That lets us query one execution in Eventhouse without mixing it with another. The instrumentation must preserve relationships the SDK actually reports, rather than guess them from arrival order.
Why Eventhouse
Eventhouse manages KQL databases for streaming data in Microsoft Fabric. In this walkthrough, those databases hold model requests, agent turns, and tool activity.
What appeals to me is being able to move between questions without moving the data. Start with “what happened in this session?”, then query tool failure rates across the last day or compare token usage by model. Put the useful queries on a native dashboard so the next investigation doesn’t start from scratch.
The path looks like this:
Existing AI application
-> SDK adapter
-> private OpenTelemetry provider
-> privacy filtering and OTel batching
-> Fabric exporter using Event Hubs AMQP
-> Eventstream Custom endpoint source
-> Eventhouse table
-> deduplicated KQL queries and native dashboards
Eventstream handles the incoming events; Eventhouse is where we store and query them. The package includes the table schema, queries, and dashboard definition, so there’s a concrete starting point rather than just a stream of JSON to figure out.

If your team already uses Fabric, this also avoids introducing a separate analytics service just for AI activity. It still needs an Eventstream route, database permissions, and a retention policy. Capacity and ingestion cost money; the walkthrough doesn’t remove those operating costs.
What the package observes
There is one npm package, with /copilot, /openai, and /vercel entry points. SDK dependencies are optional, so an OpenAI application doesn’t need the Copilot SDK installed.
Underneath, the adapters use OpenTelemetry spans: timed operations with IDs and parent relationships. Its provider is private, so it doesn’t replace the application’s global tracing configuration. The shared engine handles privacy, batching, and shutdown.
The package doesn’t run your agents. It doesn’t evaluate whether an answer is correct, decide which tool to use, or calculate model prices. Token counts are useful measurements, but a bill needs provider-specific pricing information.
This walkthrough uses version 0.2.0, whose npm publication and registry availability have been verified. It includes the SDK adapters, Fabric setup ZIP, and installed ingestion validator.
Getting spans into Eventhouse
When an operation ends, the engine filters and batches its metadata. The Fabric exporter sends canonical JSON events using the official Azure Event Hubs client over AMQP.
Eventstream receives them through its Custom endpoint source, then routes them to the Eventhouse table using the configured JSON mapping. They are JSON span objects, not OTLP, OpenTelemetry’s export protocol.
Here’s a shortened synthetic model-request event (other fields are omitted for space):
{
"schemaVersion": "1.0",
"timestamp": "2026-11-15T10:00:01.250Z",
"traceId": "00000000000000000000000000000001",
"spanId": "0000000000000004",
"parentSpanId": "0000000000000001",
"spanName": "copilot.model.request",
"startTime": "2026-11-15T10:00:01.000Z",
"endTime": "2026-11-15T10:00:01.250Z",
"durationMs": 250,
"status": 0,
"serviceName": "copilot-square-demo",
"spanAttributes": {
"gen_ai.operation.name": "chat",
"gen_ai.provider.name": "github.copilot",
"gen_ai.request.model": "example-model",
"copilot.span.type": "model_request",
"gen_ai.usage.input_tokens": 120,
"gen_ai.usage.output_tokens": 14,
"gen_ai.usage.cache_read.input_tokens": 80
}
}
The trace ID groups related work. Each span has its own ID, and parentSpanId identifies its reported parent. timestamp is completion time, not export time. Use startTime and endTime when examining overlap.
The cached 80 tokens are a breakdown of the 120 input tokens, not another 80 to add. Reasoning token counts are likewise output breakdowns. Missing counts stay missing; the package doesn’t estimate them.
For Copilot, the trace can look like this:
copilot.session observation interval
|-- copilot.model.request initial request
|-- copilot.agent.turn
| `-- copilot.tool [square]
`-- copilot.model.request response after the tool
That is an illustration, not a captured trace. Copilot usage events don’t provide a verified turn ID, so model requests normally belong to the session unless explicit invoking-tool metadata identifies another parent. A tool with a verified turn ID belongs to that turn.
I’d rather have a slightly less tidy tree than a convincing but invented one. Nested-agent relationships follow explicit invoking-tool identifiers too.
Adding tracing to Copilot
We’ll use Node.js 22.15 or later, an authenticated Copilot SDK setup, and a Fabric workspace with suitable capacity and permissions.
First, create an Eventhouse and KQL database. Install the supplied table schema and JSON mapping. In Eventstream, add a Custom endpoint source, then connect its output to that database and AIObservabilityTraces.
Use the source’s Event Hub producer credentials. An outward Eventstream destination is not the endpoint our application should send to.
The package includes a tenant-neutral setup ZIP, so readers don’t need access to the private source repository:
npm install @saguiitay/eventhouse-ai-observability@0.2.0 @github/copilot-sdk@1.0.31
npm install --save-dev tsx typescript @types/node
$packageRoot = Join-Path (npm root).Trim() '@saguiitay\eventhouse-ai-observability'
Expand-Archive -LiteralPath (Join-Path $packageRoot 'fabric-setup.zip') -DestinationPath '.\fabric-setup'
Follow the extracted README for the schema, mapping, and routing steps. The archive contains this project’s generic setup files and synthetic samples, not Microsoft-private files, existing tenant bindings, or credentials. Configuration details are in the package’s documentation files, including docs/fabric-setup.md.
Set FABRIC_EVENTSTREAM_CONNECTION_STRING through your secret provider. If it lacks EntityPath, also set FABRIC_EVENTSTREAM_ENTITY_NAME. For a local experiment, use an ignored .env; never commit the connection string.
Here’s a complete tracing-demo.ts:
import {
CopilotClient,
defineTool,
type CopilotSession,
} from "@github/copilot-sdk";
import { observeCopilot } from "@saguiitay/eventhouse-ai-observability/copilot";
const square = defineTool<{ value: number }>("square", {
description: "Square a finite number. Pure arithmetic; no external access.",
parameters: {
type: "object",
properties: { value: { type: "number" } },
required: ["value"],
additionalProperties: false,
},
skipPermission: true,
handler: ({ value }) => {
if (!Number.isFinite(value) || Math.abs(value) > 1e6)
throw new Error("Number outside demo range");
return String(value * value);
},
});
const client = new CopilotClient();
let session: CopilotSession | undefined;
let observer: ReturnType<typeof observeCopilot> | undefined;
try {
await client.start();
session = await client.createSession({
model: process.env.COPILOT_MODEL ?? "auto",
enableConfigDiscovery: false,
includedBuiltinAgents: [],
tools: [square],
availableTools: ["square"],
onPermissionRequest: () => ({
kind: "denied-no-approval-rule-and-could-not-request-from-user",
}),
});
observer = observeCopilot(session, {
serviceName: "copilot-square-demo",
captureContent: false,
});
const answer = await session.sendAndWait({
prompt: "Use square(value=12), then explain the result.",
});
console.log(answer?.data.content);
} finally {
try {
await observer?.shutdown();
} finally {
try {
if (observer) console.info(observer.getDiagnostics());
} finally {
try {
await session?.disconnect();
} finally {
await client.stop();
}
}
}
}
Run it with npx tsx --env-file=.env tracing-demo.ts. Model calls consume Copilot quota.
The observer attaches before execution. It owns its telemetry engine, but the application still owns the session and client. The nested cleanup looks a bit tedious (nothing too fancy), but a telemetry cleanup failure shouldn’t leave the client running.
I’ve disabled configuration discovery and denied other permission requests for this isolated demo. skipPermission applies only to the constrained arithmetic tool. Don’t copy that setting to a tool that reads files, runs commands, or calls an external service. In an existing application, retain your normal session configuration and permission policy.
The prompt requests a tool call; the model still decides what to do. Check the recorded spans rather than assume the requested execution happened.
Using OpenAI and Vercel
For the official OpenAI Node SDK, use the returned instrumented client:
import OpenAI from "openai";
import { observeOpenAI } from "@saguiitay/eventhouse-ai-observability/openai";
const observer = observeOpenAI(new OpenAI(), {
serviceName: "openai-demo",
});
try {
await observer.client.responses.create({
model: "gpt-4.1-mini",
input: "Explain why 12 squared is 144.",
});
} finally {
await observer.shutdown();
}
The compatibility target is OpenAI SDK 7.53.0. Direct Chat Completions and Responses create() calls are observed, including streams. Consume a stream before shutting down if you want its complete telemetry. Observation doesn’t eagerly consume it or abort it on shutdown.
For streaming Chat Completions, request stream_options: { include_usage: true } if you need the provider’s final usage counts. No reported usage means no recorded token total. These snippets use the same Fabric environment configuration; OpenAI authentication comes separately from OPENAI_API_KEY.
This adapter observes model requests, not agent turns or execution of tools described in a model response.
Vercel AI SDK 7 uses its native telemetry integration:
import { generateText } from "ai";
import { createOpenAI } from "@ai-sdk/openai";
import { OpenTelemetry } from "@ai-sdk/otel";
import { observeVercelAI } from "@saguiitay/eventhouse-ai-observability/vercel";
const provider = createOpenAI();
const observer = observeVercelAI({
integration: (tracer) => new OpenTelemetry({ tracer }),
});
try {
await generateText({
model: provider("gpt-4.1-mini"),
prompt: "Explain why 12 squared is 144.",
telemetry: observer.telemetry,
});
} finally {
await observer.shutdown();
}
The tested combination is ai@7.0.165 and @ai-sdk/otel@1.0.165. Its native telemetry produces model, step, and tool spans. Pass the configuration on every call.
Don’t instrument the same underlying provider request through both adapters. Otherwise, two model spans can describe one request and double your token totals.
Investigating in Eventhouse
Let’s return to the original question. Did our tool take the time, or did the request after it fail?
The setup installs AIObservabilitySpans(start, stop). It filters completed spans and deduplicates (traceId, spanId) with arg_max(timestamp, *) before deriving model, operation, and token columns.
Use that function before aggregating. Retries and uncertain delivery can produce duplicate rows; counting raw table rows isn’t counting logical operations.
Token usage by model:
AIObservabilitySpans(ago(24h), now())
| where serviceName == "copilot-square-demo"
| where operation in ("chat", "text_completion", "generate_content")
| summarize requests=count(),
inputTokens=sum(inputTokens), outputTokens=sum(outputTokens) by model
Slow or failing tools:
AIObservabilitySpans(ago(24h), now())
| where serviceName == "copilot-square-demo"
| where operation == "execute_tool"
| summarize invocations=count(), failures=countif(status == 2),
p95Ms=percentile(durationMs, 95) by toolName
| order by p95Ms desc
Here’s a complete investigation query for the latest completed demo session:
let selectedTrace = toscalar(
AIObservabilitySpans(ago(1h), now())
| where serviceName == "copilot-square-demo"
| where spanType == "session"
| top 1 by endTime desc
| project traceId
);
AIObservabilitySpans(ago(1h), now())
| where traceId == selectedTrace
| project startTime, endTime, spanName, operation,
spanId, parentSpanId, toolName, model, durationMs,
status, errorType, inputTokens, outputTokens
| order by startTime asc
After observer shutdown, this shows the completed session and its operations. Follow parent IDs and inspect the timestamps. A long tool span suggests tool latency; an error model-request span points elsewhere. Missing SDK durations are marked rather than reconstructed, so inspect lifecycle attributes before treating a zero duration as a fast request.
If the query returns nothing, check the service name and time window, then run the ingestion validator below. An empty query result doesn’t establish that the agent did no work.
Don’t sum parent and child durations into “total latency”: they can overlap. Also, session spans are observation intervals, not execution counts. The helper’s isAgentExecution distinguishes agent executions from session observations and separate subagent lifecycle spans.
The bundled native dashboard uses the same deduplicated data. Use it for the recurring questions, such as which tools are failing or which models account for token usage. When something looks wrong, take the relevant trace ID back into the investigation query.
That’s the Eventhouse workflow I want here: a dashboard to spot a problem, then KQL over the underlying events to inspect it. We don’t need a separate export to investigate the rows behind an aggregate.
Privacy and bounded delivery
I’ve made metadata-only capture the default. Prompts, responses, tool arguments, and tool results require explicit content capture; Vercel also requires opt-in recording flags. Known secret patterns are redacted, and application-specific redaction is supported.
That is not automatic compliance or a complete PII detector. Model names and tool identifiers are still data, and your application can log content independently (our console.log does).
The engine has sampling, active-span limits, bounded queues, and export deadlines. At capacity, observations can be skipped or finalized as incomplete. Queue overflow, oversized events, and failed or uncertain exports can drop spans. There is no disk spool or exactly-once guarantee.
Use getDiagnostics() or onDiagnostic for safe failure categories. Export counters report producer acknowledgements, not database rows. Flush exports ended work; shutdown completes observation and attempts bounded cleanup. Neither is a promise that every event survived a process crash.
Verifying the complete pipeline
A successful send proves less than I’d like it to. The producer can acknowledge a batch while routing or mapping still A successful send isn’t the same as useful data in Eventhouse. The producer can acknowledge a batch while routing or mapping still prevents the rows we need.
Before investigating an agent, check the delivery path with the installed validator:
npx --no-install eventhouse-ai-observability-validate
Alongside the producer connection string, supply FABRIC_KQL_QUERY_ENDPOINT, FABRIC_KQL_DATABASE, and FABRIC_KQL_ACCESS_TOKEN through your secret provider. The query token needs the correct Kusto audience and database read permission; a Fabric management token isn’t a substitute.
The validator sends two metadata-only spans and checks three things separately: producer acknowledgement, both rows appearing in AIObservabilityTraces, and preservation of their shared trace ID and parent-child relationship.
It makes no AI calls, but it does write telemetry and can incur normal Fabric ingestion and query costs. Use --help for an offline check of the CLI instructions.
Automated tests cover adapter behavior, packaging, and privacy controls. They don’t validate your workspace configuration. A successful validator run checks the synthetic delivery path, not whether every operation in your application was observed correctly.
Next, run the square example and use the investigation query above to find its session in Eventhouse. Check which operations actually ran, inspect their timings, and follow the reported parent IDs. That connects the pipeline check to the execution question we started with.
Once that works, add the queries you use most to the dashboard. Start with one application session, verify its stored events, then expand the instrumentation. You’ll have an Eventhouse workflow you can use for the next slow request, rather than just an exporter reporting success.