# Investigating AI agents and model calls with Microsoft Fabric Eventhouse

- Link: https://saguiitay.com/investigating-ai-agents-and-model-calls-with-microsoft-fabric-eventhouse/
- Published: 2026-11-15T20:35:18+02:00
- Author: Itay

Your AI application is slow. Was it waiting for the model, running a tool, or stuck
inside a nested agent execution?

Microsoft Fabric Eventhouse gives us somewhere to investigate that execution, compare
it with other requests, and track tool failures and token usage in dashboards. The
same stored events support both the individual investigation and the broader view.

I’ve built `@saguiitay/eventhouse-ai-observability` to get AI application activity
into Eventhouse without changing how the application runs its agents. Once the data
arrives, we can use KQL to answer those questions ourselves.

Let’s follow one small example: an agent asks a tool to calculate a square, then
makes another model request to explain the result. The arithmetic isn’t interesting.
Figuring out where the time went is.

Contents

 1.  [Why logs aren't enough](https://saguiitay.com/investigating-ai-agents-and-model-calls-with-microsoft-fabric-eventhouse/#why-logs-arent-enough)
 2.  [Why Eventhouse](https://saguiitay.com/investigating-ai-agents-and-model-calls-with-microsoft-fabric-eventhouse/#why-eventhouse)
 3.  [What the package observes](https://saguiitay.com/investigating-ai-agents-and-model-calls-with-microsoft-fabric-eventhouse/#what-the-package-observes)
 4.  [Getting spans into Eventhouse](https://saguiitay.com/investigating-ai-agents-and-model-calls-with-microsoft-fabric-eventhouse/#getting-spans-into-eventhouse)
 5.  [Adding tracing to Copilot](https://saguiitay.com/investigating-ai-agents-and-model-calls-with-microsoft-fabric-eventhouse/#adding-tracing-to-copilot)
 6.  [Using OpenAI and Vercel](https://saguiitay.com/investigating-ai-agents-and-model-calls-with-microsoft-fabric-eventhouse/#using-openai-and-vercel)
 7.  [Investigating in Eventhouse](https://saguiitay.com/investigating-ai-agents-and-model-calls-with-microsoft-fabric-eventhouse/#investigating-in-eventhouse)
 8.  [Privacy and bounded delivery](https://saguiitay.com/investigating-ai-agents-and-model-calls-with-microsoft-fabric-eventhouse/#privacy-and-bounded-delivery)
 9.  [Verifying the complete pipeline](https://saguiitay.com/investigating-ai-agents-and-model-calls-with-microsoft-fabric-eventhouse/#verifying-the-complete-pipeline)

## Why logs aren’t enough

A log saying “tool started” is useful. A provider summary saying “800 input tokens”
is useful too. Neither necessarily tells us which model request belonged to which
execution.

An AI session can contain several agent turns. A turn can involve multiple model
requests and tool calls. Tools can overlap, and an invoking tool can start a subagent.
A session’s lifetime is therefore different from the duration of an individual execution.

For our square example, I want to know whether the tool ran, how long it took, and
whether the request after it failed. If another execution overlaps, I also want 
to keep their timings separate.

We need operation IDs, timestamps, and parent relationships in the stored events.
That lets us query one execution in Eventhouse without mixing it with another. The
instrumentation must preserve relationships the SDK actually reports, rather than
guess them from arrival order.

## Why Eventhouse

[Eventhouse](https://learn.microsoft.com/en-us/fabric/real-time-intelligence/eventhouse)
manages KQL databases for streaming data in Microsoft Fabric. In this walkthrough,
those databases hold model requests, agent turns, and tool activity.

What appeals to me is being able to move between questions without moving the data.
Start with “what happened in this session?”, then query tool failure rates across
the last day or compare token usage by model. Put the useful queries on a native
dashboard so the next investigation doesn’t start from scratch.

The path looks like this:

    ```notranslate
    Existing AI application
       -> SDK adapter
       -> private OpenTelemetry provider
       -> privacy filtering and OTel batching
       -> Fabric exporter using Event Hubs AMQP
       -> Eventstream Custom endpoint source
       -> Eventhouse table
       -> deduplicated KQL queries and native dashboards
    ```

Eventstream handles the incoming events; Eventhouse is where we store and query 
them. The package includes the table schema, queries, and dashboard definition, 
so there’s a concrete starting point rather than just a stream of JSON to figure
out.

![Microsoft Fabric Eventhouse dashboard displaying AI application telemetry.](https://
i0.wp.com/saguiitay.com/wp-content/uploads/image-17.png?resize=825%2C402&ssl=1 "
Investigating AI agents and model calls with Microsoft Fabric Eventhouse 1")

If your team already uses Fabric, this also avoids introducing a separate analytics
service just for AI activity. It still needs an Eventstream route, database permissions,
and a retention policy. Capacity and ingestion cost money; the walkthrough doesn’t
remove those operating costs.

## What the package observes

There is one [npm package](https://www.npmjs.com/package/@saguiitay/eventhouse-ai-observability),
with `/copilot`, `/openai`, and `/vercel` entry points. SDK dependencies are optional,
so an OpenAI application doesn’t need the Copilot SDK installed.

Underneath, the adapters use [OpenTelemetry](https://opentelemetry.io/docs/concepts/signals/traces/)
spans: timed operations with IDs and parent relationships. Its provider is private,
so it doesn’t replace the application’s global tracing configuration. The shared
engine handles privacy, batching, and shutdown.

The package doesn’t run your agents. It doesn’t evaluate whether an answer is correct,
decide which tool to use, or calculate model prices. Token counts are useful measurements,
but a bill needs provider-specific pricing information.

This walkthrough uses version `0.2.0`, whose npm publication and registry availability
have been verified. It includes the SDK adapters, Fabric setup ZIP, and installed
ingestion validator.

## Getting spans into Eventhouse

When an operation ends, the engine filters and batches its metadata. The Fabric 
exporter sends canonical JSON events using the official Azure Event Hubs client 
over AMQP.

Eventstream receives them through its Custom endpoint source, then routes them to
the Eventhouse table using the configured JSON mapping. They are JSON span objects,
not OTLP, OpenTelemetry’s export protocol.

Here’s a shortened synthetic model-request event (other fields are omitted for space):

    ```notranslate
    {
      "schemaVersion": "1.0",
      "timestamp": "2026-11-15T10:00:01.250Z",
      "traceId": "00000000000000000000000000000001",
      "spanId": "0000000000000004",
      "parentSpanId": "0000000000000001",
      "spanName": "copilot.model.request",
      "startTime": "2026-11-15T10:00:01.000Z",
      "endTime": "2026-11-15T10:00:01.250Z",
      "durationMs": 250,
      "status": 0,
      "serviceName": "copilot-square-demo",
      "spanAttributes": {
        "gen_ai.operation.name": "chat",
        "gen_ai.provider.name": "github.copilot",
        "gen_ai.request.model": "example-model",
        "copilot.span.type": "model_request",
        "gen_ai.usage.input_tokens": 120,
        "gen_ai.usage.output_tokens": 14,
        "gen_ai.usage.cache_read.input_tokens": 80
      }
    }
    ```

The trace ID groups related work. Each span has its own ID, and `parentSpanId` identifies
its reported parent. `timestamp` is completion time, not export time. Use `startTime`
and `endTime` when examining overlap.

The cached 80 tokens are a breakdown of the 120 input tokens, not another 80 to 
add. Reasoning token counts are likewise output breakdowns. Missing counts stay 
missing; the package doesn’t estimate them.

For Copilot, the trace can look like this:

    ```notranslate
    copilot.session                      observation interval
    |-- copilot.model.request            initial request
    |-- copilot.agent.turn
    |   `-- copilot.tool [square]
    `-- copilot.model.request            response after the tool
    ```

That is an illustration, not a captured trace. Copilot usage events don’t provide
a verified turn ID, so model requests normally belong to the session unless explicit
invoking-tool metadata identifies another parent. A tool with a verified turn ID
belongs to that turn.

I’d rather have a slightly less tidy tree than a convincing but invented one. Nested-
agent relationships follow explicit invoking-tool identifiers too.

## Adding tracing to Copilot

We’ll use Node.js 22.15 or later, an authenticated [Copilot SDK](https://github.com/github/copilot-sdk)
setup, and a Fabric workspace with suitable capacity and permissions.

First, create an Eventhouse and KQL database. Install the supplied table schema 
and JSON mapping. In Eventstream, add a [Custom endpoint source](https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-streams/add-source-custom-app),
then connect its output to that database and `AIObservabilityTraces`.

Use the source’s Event Hub producer credentials. An outward Eventstream destination
is not the endpoint our application should send to.

The package includes a tenant-neutral setup ZIP, so readers don’t need access to
the private source repository:

    ```notranslate
    npm install @saguiitay/eventhouse-ai-observability@0.2.0 @github/copilot-sdk@1.0.31
    npm install --save-dev tsx typescript @types/node

    $packageRoot = Join-Path (npm root).Trim() '@saguiitay\eventhouse-ai-observability'
    Expand-Archive -LiteralPath (Join-Path $packageRoot 'fabric-setup.zip') -DestinationPath '.\fabric-setup'
    ```

Follow the extracted README for the schema, mapping, and routing steps. The archive
contains this project’s generic setup files and synthetic samples, not Microsoft-
private files, existing tenant bindings, or credentials. Configuration details are
in the [package’s documentation files](https://www.npmjs.com/package/@saguiitay/eventhouse-ai-observability?activeTab=code),
including `docs/fabric-setup.md`.

Set `FABRIC_EVENTSTREAM_CONNECTION_STRING` through your secret provider. If it lacks`
EntityPath`, also set `FABRIC_EVENTSTREAM_ENTITY_NAME`. For a local experiment, 
use an ignored `.env`; never commit the connection string.

Here’s a complete `tracing-demo.ts`:

    ```notranslate
    import {
      CopilotClient,
      defineTool,
      type CopilotSession,
    } from "@github/copilot-sdk";
    import { observeCopilot } from "@saguiitay/eventhouse-ai-observability/copilot";

    const square = defineTool<{ value: number }>("square", {
      description: "Square a finite number. Pure arithmetic; no external access.",
      parameters: {
        type: "object",
        properties: { value: { type: "number" } },
        required: ["value"],
        additionalProperties: false,
      },
      skipPermission: true,
      handler: ({ value }) => {
        if (!Number.isFinite(value) || Math.abs(value) > 1e6)
          throw new Error("Number outside demo range");
        return String(value * value);
      },
    });

    const client = new CopilotClient();
    let session: CopilotSession | undefined;
    let observer: ReturnType<typeof observeCopilot> | undefined;

    try {
      await client.start();
      session = await client.createSession({
        model: process.env.COPILOT_MODEL ?? "auto",
        enableConfigDiscovery: false,
        includedBuiltinAgents: [],
        tools: [square],
        availableTools: ["square"],
        onPermissionRequest: () => ({
          kind: "denied-no-approval-rule-and-could-not-request-from-user",
        }),
      });
      observer = observeCopilot(session, {
        serviceName: "copilot-square-demo",
        captureContent: false,
      });
      const answer = await session.sendAndWait({
        prompt: "Use square(value=12), then explain the result.",
      });
      console.log(answer?.data.content);
    } finally {
      try {
        await observer?.shutdown();
      } finally {
        try {
          if (observer) console.info(observer.getDiagnostics());
        } finally {
          try {
            await session?.disconnect();
          } finally {
            await client.stop();
          }
        }
      }
    }
    ```

Run it with `npx tsx --env-file=.env tracing-demo.ts`. Model calls consume Copilot
quota.

The observer attaches before execution. It owns its telemetry engine, but the application
still owns the session and client. The nested cleanup looks a bit tedious (nothing
too fancy), but a telemetry cleanup failure shouldn’t leave the client running.

I’ve disabled configuration discovery and denied other permission requests for this
isolated demo. `skipPermission` applies only to the constrained arithmetic tool.
Don’t copy that setting to a tool that reads files, runs commands, or calls an external
service. In an existing application, retain your normal session configuration and
permission policy.

The prompt requests a tool call; the model still decides what to do. Check the recorded
spans rather than assume the requested execution happened.

## Using OpenAI and Vercel

For the official OpenAI Node SDK, use the returned instrumented client:

    ```notranslate
    import OpenAI from "openai";
    import { observeOpenAI } from "@saguiitay/eventhouse-ai-observability/openai";

    const observer = observeOpenAI(new OpenAI(), {
      serviceName: "openai-demo",
    });
    try {
      await observer.client.responses.create({
        model: "gpt-4.1-mini",
        input: "Explain why 12 squared is 144.",
      });
    } finally {
      await observer.shutdown();
    }
    ```

The compatibility target is OpenAI SDK 7.53.0. Direct Chat Completions and Responses`
create()` calls are observed, including streams. Consume a stream before shutting
down if you want its complete telemetry. Observation doesn’t eagerly consume it 
or abort it on shutdown.

For streaming Chat Completions, request `stream_options: { include_usage: true }`
if you need the provider’s final usage counts. No reported usage means no recorded
token total. These snippets use the same Fabric environment configuration; OpenAI
authentication comes separately from `OPENAI_API_KEY`.

This adapter observes model requests, not agent turns or execution of tools described
in a model response.

Vercel AI SDK 7 uses its native telemetry integration:

    ```notranslate
    import { generateText } from "ai";
    import { createOpenAI } from "@ai-sdk/openai";
    import { OpenTelemetry } from "@ai-sdk/otel";
    import { observeVercelAI } from "@saguiitay/eventhouse-ai-observability/vercel";

    const provider = createOpenAI();
    const observer = observeVercelAI({
      integration: (tracer) => new OpenTelemetry({ tracer }),
    });
    try {
      await generateText({
        model: provider("gpt-4.1-mini"),
        prompt: "Explain why 12 squared is 144.",
        telemetry: observer.telemetry,
      });
    } finally {
      await observer.shutdown();
    }
    ```

The tested combination is `ai@7.0.165` and `@ai-sdk/otel@1.0.165`. Its [native telemetry](https://ai-sdk.dev/docs/ai-sdk-core/telemetry)
produces model, step, and tool spans. Pass the configuration on every call.

Don’t instrument the same underlying provider request through both adapters. Otherwise,
two model spans can describe one request and double your token totals.

## Investigating in Eventhouse

Let’s return to the original question. Did our tool take the time, or did the request
after it fail?

The setup installs `AIObservabilitySpans(start, stop)`. It filters completed spans
and deduplicates `(traceId, spanId)` with `arg_max(timestamp, *)` before deriving
model, operation, and token columns.

Use that function before aggregating. Retries and uncertain delivery can produce
duplicate rows; counting raw table rows isn’t counting logical operations.

Token usage by model:

    ```notranslate
    AIObservabilitySpans(ago(24h), now())
    | where serviceName == "copilot-square-demo"
    | where operation in ("chat", "text_completion", "generate_content")
    | summarize requests=count(),
        inputTokens=sum(inputTokens), outputTokens=sum(outputTokens) by model
    ```

Slow or failing tools:

    ```notranslate
    AIObservabilitySpans(ago(24h), now())
    | where serviceName == "copilot-square-demo"
    | where operation == "execute_tool"
    | summarize invocations=count(), failures=countif(status == 2),
        p95Ms=percentile(durationMs, 95) by toolName
    | order by p95Ms desc
    ```

Here’s a complete investigation query for the latest completed demo session:

    ```notranslate
    let selectedTrace = toscalar(
        AIObservabilitySpans(ago(1h), now())
        | where serviceName == "copilot-square-demo"
        | where spanType == "session"
        | top 1 by endTime desc
        | project traceId
    );
    AIObservabilitySpans(ago(1h), now())
    | where traceId == selectedTrace
    | project startTime, endTime, spanName, operation,
        spanId, parentSpanId, toolName, model, durationMs,
        status, errorType, inputTokens, outputTokens
    | order by startTime asc
    ```

After observer shutdown, this shows the completed session and its operations. Follow
parent IDs and inspect the timestamps. A long tool span suggests tool latency; an
error model-request span points elsewhere. Missing SDK durations are marked rather
than reconstructed, so inspect lifecycle attributes before treating a zero duration
as a fast request.

If the query returns nothing, check the service name and time window, then run the
ingestion validator below. An empty query result doesn’t establish that the agent
did no work.

Don’t sum parent and child durations into “total latency”: they can overlap. Also,
session spans are observation intervals, not execution counts. The helper’s `isAgentExecution`
distinguishes agent executions from session observations and separate subagent lifecycle
spans.

The bundled native dashboard uses the same deduplicated data. Use it for the recurring
questions, such as which tools are failing or which models account for token usage.
When something looks wrong, take the relevant trace ID back into the investigation
query.

That’s the Eventhouse workflow I want here: a dashboard to spot a problem, then 
KQL over the underlying events to inspect it. We don’t need a separate export to
investigate the rows behind an aggregate.

## Privacy and bounded delivery

I’ve made metadata-only capture the default. Prompts, responses, tool arguments,
and tool results require explicit content capture; Vercel also requires opt-in recording
flags. Known secret patterns are redacted, and application-specific redaction is
supported.

That is not automatic compliance or a complete PII detector. Model names and tool
identifiers are still data, and your application can log content independently (
our `console.log` does).

The engine has sampling, active-span limits, bounded queues, and export deadlines.
At capacity, observations can be skipped or finalized as incomplete. Queue overflow,
oversized events, and failed or uncertain exports can drop spans. There is no disk
spool or exactly-once guarantee.

Use `getDiagnostics()` or `onDiagnostic` for safe failure categories. Export counters
report producer acknowledgements, not database rows. Flush exports ended work; shutdown
completes observation and attempts bounded cleanup. Neither is a promise that every
event survived a process crash.

## Verifying the complete pipeline

A successful send proves less than I’d like it to. The producer can acknowledge 
a batch while routing or mapping still A successful send isn’t the same as useful
data in Eventhouse. The producer can acknowledge a batch while routing or mapping
still prevents the rows we need.

Before investigating an agent, check the delivery path with the installed validator:

    ```notranslate
    npx --no-install eventhouse-ai-observability-validate
    ```

Alongside the producer connection string, supply `FABRIC_KQL_QUERY_ENDPOINT`, `FABRIC_KQL_DATABASE`,
and `FABRIC_KQL_ACCESS_TOKEN` through your secret provider. The query token needs
the correct Kusto audience and database read permission; a Fabric management token
isn’t a substitute.

The validator sends two metadata-only spans and checks three things separately: 
producer acknowledgement, both rows appearing in `AIObservabilityTraces`, and preservation
of their shared trace ID and parent-child relationship.

It makes no AI calls, but it does write telemetry and can incur normal Fabric ingestion
and query costs. Use `--help` for an offline check of the CLI instructions.

Automated tests cover adapter behavior, packaging, and privacy controls. They don’t
validate your workspace configuration. A successful validator run checks the synthetic
delivery path, not whether every operation in your application was observed correctly.

Next, run the square example and use the investigation query above to find its session
in Eventhouse. Check which operations actually ran, inspect their timings, and follow
the reported parent IDs. That connects the pipeline check to the execution question
we started with.

Once that works, add the queries you use most to the dashboard. Start with one application
session, verify its stored events, then expand the instrumentation. You’ll have 
an Eventhouse workflow you can use for the next slow request, rather than just an
exporter reporting success.

### _Related_
