Diego Souza · AI Engineer

All projects

Durable async workflows

Agent work that survives a worker dying mid-task.

isolated gate verified, no cloud runLast verified: 2026-10-30 · Commit ceaeedc

TL;DR

Problem
Work continues after the HTTP response, and brokers deliver a message more than once.
My role
I designed the task states, the lease model and the effect record.
Stack
Python, FastAPI, Redis Streams, PostgreSQL, SQS-compatible service, Terraform
Result
The isolated full fake gate passed: 127 tests, locked service scenarios, image build and Terraform plan.

An API and workers for long-running AI jobs, with leases, retries, a dead-letter path and crash recovery, tested against real Redis Streams and an SQS-compatible service. A stuck task is a state I designed for, not an incident.

See it run

The live demo runs this system against a real model and shows every step.

Live demo

How it works

A task that survives its workerFlow: Queued, Leased, Running, Retry, Dead letter, DoneQueuedLeasedRunningRetryDead letterDoneA task that survives its workerFlow: Queued, Leased, Running, Retry, Dead letter, DoneQueuedLeasedRunningRetryDead letterDone

The whole flow. Press play or next to walk through it.

Text version of this diagram
  1. Queued P04. The job is stored before the HTTP response returns.
    • then Leased
  2. Leased P04. A worker holds a time-limited lease.
    • then Running
  3. Running P04. The effect is recorded so a redelivery cannot repeat it.
    • then Done
    • then Retry (transient failure)
    • then Dead letter (permanent failure)
  4. Retry P04. A transient failure goes back with a delay.
    • then Queued
  5. Dead letter P04. A permanent failure stops here, visible and replayable.
  6. Done P04. The result is stored once.

Decisions

Results

Measured

Measured
MetricValueModeSource
Isolated full fake gate Terraform was planned, never applied.127 tests, locked service scenarios, image build and Terraform planservices, fake model.planning/phases/043-root-snapshot-gate/VERIFY.mdCommit d3e694d · 2026-10-30 · Exit code 0

Not measured, so not claimed

  • Behavior in a cloud deployment of this service alone. Terraform was planned, never applied.
  • Throughput.

Failure gallery

What happened
An outbox event failed with an `UnboundLocalError`.
Cause
A function-local import shadowed the module import.
What changed
The local imports were removed and the suite passes.
What happened
An offline install could not resolve a pinned dependency.
Cause
The ad hoc install ignored the lock file.
What changed
Installs go through the lock file, so the same set of packages is reproduced.

Validate it yourself

git clone https://github.com/DiegoVSouza/supportops-async.git
cd supportops-async
uv sync --locked

Expected: The environment installs from the lock. The service gate needs Docker and the compose files named in the workflow.

What I would do differently

I would write the crash-recovery scenario as the first test, before the happy path.

Next project

This site and its design system