Diego Souza · AI Engineer

All projects

Production AI toolkit

The boring safety layer that keeps an agent from hurting production.

fixtures verified, quality not claimedLast verified: 2026-10-29 · Commit 22eb0c2

TL;DR

Problem
A tool call needs a schema, a bound and a refusal path before a model can act on it.
My role
I designed the library and its adversarial suite.
Stack
Python (standard library only)
Result
The locked fake gate passes: 82 unit tests plus fixture and adversarial checks.

Tool schemas, safe retries, step and time limits, code-first evals, an adversarial suite and cost helpers, in the Python standard library. It separates what is proven from what is not: fixtures test control logic, and they are never presented as model quality.

See it run

The live demo runs this system against a real model and shows every step.

Live demo

How it works

A tool call under controlFlow: Tool call, Guardrails, Step and time limits, Code-first evals, Cost helpersTool callGuardrailsStep and time limitsCode-first evalsCost helpersA tool call under controlFlow: Tool call, Guardrails, Step and time limits, Code-first evals, Cost helpersTool callGuardrailsStep and time limitsCode-first evalsCost helpers

The whole flow. Press play or next to walk through it.

Text version of this diagram
  1. Tool call P02. A call arrives with arguments from a model.
    • then Guardrails
  2. Guardrails P02. Input is screened before anything runs.
    • then Step and time limits
  3. Step and time limits P02. A budget of steps, tokens and seconds is consumed and enforced.
    • then Code-first evals
  4. Code-first evals P02. Checks run as code, against fixtures and an adversarial suite.
    • then Cost helpers
  5. Cost helpers P02. Routing, caching and streaming helpers for cost.

Decisions

No dependencies

The library uses only the standard library, so adopting it never drags a transitive dependency into a service.

Code: supportops_toolkit/__init__.py

Rejected: Wrapping a popular validation library.

Limits are consumed, not advised

Steps, tokens and seconds are consumed from one budget, and an exhausted budget raises.

Code: supportops_toolkit/guardrails.py

Rejected: Logging a warning when a limit is passed.

Results

Measured

Measured
MetricValueModeSource
Locked fake gate Fixtures test control logic, not model quality.82 unit tests plus fixture and adversarial checksfake: control logic.planning/phases/016-accounted-local-operation/VERIFY.mdCommit 5b91e73 · 2026-10-29 · Exit code 0

Not measured, so not claimed

  • Model quality. The evals run on fixtures.
  • Adversarial coverage against a real model.

Failure gallery

What happened
The first real completion returned a bad-request error and was not retried.
Cause
The exact cause was not established, because the response body was deliberately not retained.
What changed
The request now sets the completion-token cap and a low reasoning effort, a regression test checks both, and a corrective real evaluation passed.
What happened
A lock test read a held lease file on Windows and failed.
Cause
Windows locks prevent reading a file another process owns.
What changed
The test reads sanitized metadata after the lease is released, and still proves concurrent acquisition is blocked.

Validate it yourself

git clone https://github.com/DiegoVSouza/supportops-ai-toolkit.git
cd supportops-ai-toolkit
uv sync --locked && uv run --locked python scripts/check_full.py

Expected: The full gate passes with fixtures and the adversarial suite.

What I would do differently

I would add a provider-neutral retry classification table with a test per row, so each retry rule is a visible decision.

Next project

Agent observability