- Home
- Case studies
- An agent-operated task engine
An agent-operated task engine
Giving autonomous AI agents a safe shared task board: claim work once, recover abandoned leases, and escalate repeated failures to humans without a background worker.
Published Jun 30, 2026Reviewed Jul 2, 2026
The short version
- Problem
- AI agents needed to work from the same queue as humans without double-claiming tasks or hiding failures.
- Constraint
- The system had to stay simple: no always-on worker, no fragile polling loop, and a full audit trail for every state transition.
- System built
- A contract-first task API with Postgres row locks, renewable leases, lazy expiry, server-sent events, and append-only events.
- Result
- Agents can claim, heartbeat, complete, fail, and escalate work safely while humans watch the same board update in real time.
- Where this applies
- Useful when a team wants AI agents in production but needs ownership, review, retry, and failure boundaries before autonomy expands.
- Stack
- Drizzle · Postgres LISTEN/NOTIFY · SSE · contract-first API
Why this matters
If your team is planning a system with the same failure modes, this is the level of thinking a first call starts at.


Overview
The constraint above is the entire brief: a shared task board that both humans and autonomous agents can safely claim work from, with no always-on worker to deploy and babysit, no polling loop trading responsiveness against load, and a full audit trail of every claim, heartbeat, completion, failure, and escalation so the board's history never has to be reconstructed after the fact. Two agents can watch the same board at the same instant, decide to claim the same task, and issue their claims microseconds apart — a crashed agent can vanish mid-task and leave its claim behind — and a misbehaving agent can fail the same task over and over without anything forcing a human to look at it. Everything below is built out of primitives Postgres already provides — row locks, a lease with a TTL, and an append-only log — rather than a second system bolted on to compensate for what the database could already do natively.
Claiming without contention
A naive claim is a SELECT to find a pending task followed by a separate UPDATE to mark it claimed, and the gap between those two queries is exactly where two agents can both select the same row and both believe they claimed it.
The fix isn't a distributed lock service running alongside Postgres — it's a locking clause Postgres already ships:
SELECT id FROM tasks
WHERE status = 'pending'
ORDER BY priority DESC, created_at ASC
FOR UPDATE SKIP LOCKED
LIMIT 1;
Claim query — concurrent agents each land on a different row
FOR UPDATE takes a row lock on the candidate task inside the claiming transaction; SKIP LOCKED tells Postgres that if a row is already locked by someone else's in-flight claim, don't wait for it and don't error, just move on to the next row.
Two agents running this query at the same instant don't block each other and don't collide — each one walks past whatever the other has already locked and locks the next available task instead.
The claim needs no retry loop and no compensating "undo the claim" step, because mutual exclusion is enforced by the database's own row-level locking rather than approximated in application code.
Why not a polling loop or a lock service
A polling loop — agents periodically asking "anything pending?" and racing to update whatever a query returns — still has this exact race unless the update itself is atomic, so polling alone is a scheduling strategy, not a concurrency control.
A distributed lock service solves the race but adds a second system to deploy, monitor, and reason about — including what happens to a claim if the lock service itself becomes unreachable — for a guarantee Postgres already gives on the same table the task lives in.
SKIP LOCKED keeps the concurrency control inside the same transaction boundary as the data it protects, which is one fewer moving part to operate.
Renewable leases and lazy expiry
A claim is a lease, not a permanent handoff: claiming a task stamps it with a leased_until timestamp a fixed TTL in the future, and the claiming agent is expected to heartbeat before that deadline, pushing leased_until forward again.
"Claimed" and "still being worked on" aren't the same fact — an agent can crash, lose its connection, or be killed mid-task, and a task it claimed shouldn't stay claimed forever just because no explicit release step ever ran.
UPDATE tasks
SET leased_until = now() + interval '2 minutes'
WHERE id = $1 AND leased_by = $2 AND leased_until > now()
RETURNING id;
Heartbeat — renew a lease the agent still holds
The leased_until > now() guard matters as much as the interval itself: if the lease already expired, the heartbeat matches no row, and the agent that assumed it still owned the task finds out immediately instead of continuing work against a task someone else may have already reclaimed.
Why not a background reaper
The obvious design for expiring a lease is a scheduled reaper — a job that scans the table every few seconds and resets rows whose leased_until has passed back to pending.
That's a process with its own lifecycle: it has to be deployed, it has to be monitored for whether it's still running, and if it silently stops, expired leases quietly stop being reclaimed until someone notices a backlog of stuck tasks.
It is exactly the always-on worker the constraint rules out.
The alternative is lazy expiry: nothing reclaims an expired lease proactively. Instead, the claim query itself treats an expired lease as claimable, alongside a task that was never claimed at all:
SELECT id FROM tasks
WHERE status = 'pending'
OR (status = 'in_progress' AND leased_until < now())
ORDER BY priority DESC, created_at ASC
FOR UPDATE SKIP LOCKED
LIMIT 1;
Claim query extended to reclaim expired leases inline
An expired lease gets reclaimed the moment another agent goes looking for work, which on a board that's actually busy is seconds later rather than on some fixed sweep interval. If the board goes quiet, an abandoned task simply waits, which is the correct behavior when there's no one around to hand it to anyway. There is no separate process to keep alive for any of this — the reclaim logic lives entirely inside the same query that claims fresh work.
Diagram: the task-claim lifecycle — an agent claims a task with FOR UPDATE SKIP LOCKED, holds a renewable lease with periodic heartbeats, either completes or fails the task, and on repeated failure the three-strike counter escalates it to a human; an expired, un-renewed lease is reclaimed lazily by the next claimant instead of a background sweeper.
Escalation and the audit trail
Not every failure is transient.
An agent can fail a task because of a bad prompt, an unsolvable task, or malformed input, and retrying with the same agent under the same conditions will often just fail again, potentially forever, unless something intervenes.
Every task carries a failure counter that increments on each fail transition; the third failure escalates the task to a human queue instead of allowing a fourth claim.
That threshold draws the line the whole design depends on: one bad attempt is noise, three is a pattern a human should look at.
Every transition — claimed, heartbeated, completed, failed, escalated — is written as a new row in an append-only event log rather than overwritten onto a single mutable status column.
INSERT INTO task_events (task_id, type, actor, payload, created_at)
VALUES ($1, 'failed', $2, $3, now());
Recording a transition — the events table is insert-only
A mutable status column can only ever answer "what is this task's status right now"; an append-only log can answer "what happened to this task, and in what order." Status-only tracking would tell you a task failed three times but not what each attempt looked like, which agent ran it, or how a heartbeat that may or may not have landed relates to the eventual failure — that history either lives in logs that may have already rotated away, or it's simply gone. The event log makes the board's history a queryable fact instead of something reconstructed after an incident, which matters more, not less, once the client claiming and failing tasks is autonomous and can't be asked afterward what it thought was happening.
Real-time fanout
Humans and agents need to watch the same board without either one hammering it with requests.
Client-side polling — every open tab issuing a fresh query every few seconds — scales badly, since every additional viewer multiplies load linearly, and it's inherently laggy, since average staleness is half the poll interval at best.
The alternative is to let Postgres announce changes itself: every state transition issues a NOTIFY in the same transaction that made the change, and a single long-lived process holds a LISTEN subscription and fans each notification out to connected browsers as a server-sent event.
NOTIFY task_events, '{"taskId": "...", "type": "claimed"}';
Publishing a change alongside the transaction that made it
SSE rather than WebSockets is the right amount of complexity here: the board only needs a one-way stream from server to browser, and SSE gets that over plain HTTP, with reconnection handled by the browser's own EventSource API rather than hand-rolled.
One LISTEN connection fans out to every open tab; the database's notification system is the single source of truth for "something changed," and the UI never has to poll to find out.
Tradeoffs and limitations
- Lazy expiry means a task whose lease has expired isn't actually reclaimed until some other claimant goes looking for work — on a quiet board, a stranded task can sit unreclaimed well past its TTL, since nothing forces the issue.
FOR UPDATE SKIP LOCKEDstill needs the underlying table watched under high churn — heavy update and delete volume needs normal Postgres autovacuum attention, or lock contention and dead-tuple bloat show up under load regardless of the claim pattern.LISTEN/NOTIFYdoesn't survive a dropped connection or a restart, and payloads are capped — a fanout process re-subscribing after a disconnect can miss whatever happened while it was down, so it's a best-effort real-time layer, not a durable one; the table and the event log stay the source of truth, not the notification stream.- A lease's TTL bounds how long a well-behaved agent can hold a task, but nothing stops a misbehaving agent from heartbeating on schedule while making no real progress — the lease protects against a crashed or disconnected agent, not a live one spinning uselessly inside its own window.
- Contract-first API discipline is enforced by review, not by tooling — nothing technical stops a future endpoint from being added that breaks the claim/heartbeat/complete/fail contract an autonomous client already depends on.
- The three-strike threshold is fixed rather than tuned per task type, so a task that's simply harder than average escalates on the same schedule as one that's genuinely broken.
References and further reading
The primitives here are all well-documented, general-purpose building blocks rather than anything bespoke:
Related case studies
Systems that share a constraint with this one — retries, isolation, cost, recovery.
A durable job queue that replaced a wall of no-code automations
Failures became queryable rows, retries became safe by construction, and new automations could be tested against live traffic before acting.
Go · Postgres leases · gocron · bearer + HMAC authA multi-provider AI gateway with built-in cost accounting
Model choice became configuration, new providers inherited common controls, and spend became visible while requests were still running.
Go adapters · gRPC / protobuf · persisted in-memory queue