Diego Souza · AI Engineer

All projects

Agent observability

See what an agent did without storing what users said.

service gate verified, production telemetry pendingLast verified: 2026-10-29 · Commit afd9b49

TL;DR

Problem
An agent can repeat, retry and spend across services, and teams should not need to collect prompts to see it.
My role
I built the instrumentation, the metrics and the loop detector.
Stack
Python, OpenTelemetry, Prometheus, Grafana, Jaeger
Result
The full fake service gate passed: 70 tests, service traces, a dashboard and four firing alerts.

OpenTelemetry traces, correlated logs, Prometheus metrics and deterministic loop detection. Content capture is off by default, and tests prove that span status stays content-free. In the live demo, this is what you are looking at.

See it run

The live demo runs this system against a real model and shows every step.

Live demo

How it works

A trace with no visitor textFlow: API request, worker.run span, agent.triage span, Model call, Loop detectorAPI requestworker.run spanagent.triage spanModel callLoop detectorA trace with no visitor textFlow: API request, worker.run span, agent.triage span, Model call, Loop detectorAPI requestworker.run spanagent.triage spanModel callLoop detector

The whole flow. Press play or next to walk through it.

Text version of this diagram
  1. API request P05. The request starts the trace.
    • then worker.run span
  2. worker.run span P05. The worker continues the same trace from the job's context.
    • then agent.triage span
  3. agent.triage span P05. One span around the four agents.
    • then Model call
  4. Model call P03. Tokens, cost and latency are recorded. Content is not.
    • then Loop detector
  5. Loop detector P03. A deterministic detector flags repeated actions.

Decisions

Content capture is off by default

Spans carry timing, status and token counts. A test checks that span status never carries content.

Code: supportops_obs/instrumentation.py

Rejected: Capturing prompts and trimming them later.

Loop detection is deterministic

A repeated-action detector flags loops from the action sequence, with no model in the loop.

Code: supportops_obs/loopdetect.py

Rejected: Asking a model whether the agent is stuck.

Results

Measured

Measured
MetricValueModeSource
Full fake service gate70 tests, service traces, a dashboard and four firing alertsservices, fake model.planning/phases/025-accounted-local-live/VERIFY.mdCommit 83fe067 · 2026-10-29 · Exit code 0

Not measured, so not claimed

  • Production telemetry volume and cost.
  • Privacy for every integration, only for the paths the tests cover.

Failure gallery

What happened
The first Docker gate exceeded its wait for the synthetic alerts.
Cause
A cold start took longer than the configured wait. All alerts fired later.
What changed
The wait is configurable and was raised without changing any threshold.

Validate it yourself

git clone https://github.com/DiegoVSouza/supportops-observability.git
cd supportops-observability
uv sync --locked --extra dev && uv run --locked python scripts/check_full.py

Expected: The unit tests pass. The service part needs Docker.

What I would do differently

I would add the loop detector's thresholds to the trace as attributes from the start, so tuning them is visible.

Next project

Durable async workflows