Diego Souza · AI Engineer

All projects

Foundation and live evals

The contracts every project shares, and the evaluation that told me my agent was not ready.

failed provisionally, reportedLast verified: 2026-10-30 · Commit 0725025

TL;DR

Problem
Independent services need one vocabulary for tickets, invoices and tools, and an honest way to test a model against it.
My role
I wrote the contracts, the testkit and the campaign runner, and ran the evaluation.
Stack
Python, Pydantic, uv, pytest
Result
27 of 36 real-model trials passed, and 9 of 12 unique cases passed all three trials (confidence interval 0.47 to 0.91 (Wilson, 95%)). Verdict `FAIL_PROVISIONAL`.

Domain models, a seeded dataset, a testkit and one budget ledger across fake, replay and live. Then the honest part: a real-model campaign, reported with its pass rate, its confidence interval and the families that failed. The verdict was `FAIL_PROVISIONAL`, and I kept the held-out budget untouched instead of tuning until the number looked good.

See it run

The live demo runs this system against a real model and shows every step.

Live demo

How it works

From dataset to verdictFlow: Seeded dataset, One client factory, Fake, Replay, Live with budget, Campaign, VerdictSeeded datasetOne client factoryFakeReplayLive with budgetCampaignVerdictFrom dataset to verdictFlow: Seeded dataset, One client factory, Fake, Replay, Live with budget, Campaign, VerdictSeeded datasetOne client factoryFakeReplayLive with budgetCampaignVerdict

The whole flow. Press play or next to walk through it.

Text version of this diagram
  1. Seeded dataset P00. Deterministic data with a digest, so a change is visible.
    • then One client factory
  2. One client factory P00. One interface selects fake, replay or live.
    • then Fake
    • then Replay
    • then Live with budget
  3. Fake P00. Deterministic, the default in tests and CI.
  4. Replay P00. Reads recorded cassettes and fails on a mismatch.
  5. Live with budget P00. Budget is reserved before transport.
    • then Campaign
  6. Campaign P00. Chunked trials that never replay a completed or uncertain one.
    • then Verdict
  7. Verdict P00. The report states the pass rate, the interval and the failing families.

Decisions

Unknown fields are errors

Models reject unknown fields and require timezones, so a contract change shows up in a consumer's tests before it shows up in production.

Code: supportops_foundation/domain/models.py

Rejected: Loose dictionaries passed between services.

One interface for fake, replay and live

A factory picks the mode, and the live mode reserves budget before any transport. Control logic is testable with no provider.

Code: supportops_foundation/llm/factory.py

Rejected: A separate code path for evaluation.

Report the failure, protect the held-out set

When the campaign failed, I did not rerun it against the protected budget. The report states the failing families and stops. 84 held-out calls and 60 confirmation calls stayed reserved.

Code: scripts/run_live_campaign.py

Rejected: Tuning the prompt against the held-out cases until the number passes.

Results

Measured

Measured
MetricValueModeSource
Real-model trials passed (development set) Development set, not held out. Status FAIL_PROVISIONAL; the exit code follows that status, as the P00 README defines it.27 of 36live modelartifacts/evals/live/free-dev-c2r1/report.jsonCommit 09d8c1e · 2026-10-28 · Exit code 1
Unique cases that passed all three trials Same campaign, same status.9 of 12live modelartifacts/evals/live/free-dev-c2r1/report.jsonCommit 09d8c1e · 2026-10-28 · Exit code 1
Wilson 95% interval for the all-three pass rate Rounded to two decimals from the report.0.47 to 0.91 (Wilson, 95%)live modelartifacts/evals/live/free-dev-c2r1/report.jsonCommit 09d8c1e · 2026-10-28 · Exit code 1
Failures by family in the development campaign Counts of errors, not of failed trials; one trial can carry more than one.classification 3, routing 6, tool selection 3live modelartifacts/evals/live/free-dev-c2r1/report.jsonCommit 09d8c1e · 2026-10-28 · Exit code 1
Protected evaluation budget84 held-out calls and 60 confirmation calls stayed reservedlive modeldocs/harness/progress.mdCommit 6b84f9a · 2026-10-30 · Exit code 0

Not measured, so not claimed

  • Held-out and confirmation performance. Those budgets were not spent.
  • Human review of the answers. It is pending.
  • Provider latency under load.

Failure gallery

What happened
A resumed campaign could start a provider client after an earlier provider error.
Cause
An existing completed error outcome did not block initialization on resume.
What changed
Resume now refuses before any client is created and requires a review.
What happened
Two development cases disagreed with the generic prompt.
Cause
The prompt encourages a lookup for concrete issues, while one case expects a clarifying question first.
What changed
Recorded as a contract tension for a person to resolve. The dataset was not edited to pass.

Validate it yourself

git clone https://github.com/DiegoVSouza/supportops-foundation.git
cd supportops-foundation
uv sync --locked && uv run --locked python scripts/check_fast.py

Expected: The fast gate passes. Live commands need a funded campaign and are not part of it.

What I would do differently

I would split the prompt contract before the campaign, not after it failed, so the two disagreeing cases never counted against the model.

Next project

Multi-agent orchestration