Overview

The brief was narrower than it sounds. Not "replace no-code with better no-code," and not "stand up a new platform" — the same production behavior had to keep running, only now in a form where a failure is explainable instead of a shrug, calls to external APIs stay inside whatever rate limit each provider actually enforces, every entry point into the system is authenticated rather than a bare webhook URL anyone with the link can hit, and a job that dies partway through can be picked back up instead of silently lost. All of that had to happen without a rewrite of the surrounding application — the queue had to slot in as one more piece of infrastructure, not a six-month platform migration that put every other roadmap item on hold. That framing ruled out the obvious move of just adding better dashboards to the existing no-code tools: a dashboard describes a black box, it doesn't turn it into a queryable one. The fix had to live at the level of how work is claimed, retried, and rate-limited — not at the level of how it's displayed.

Lease-based job claiming with heartbeats

Every unit of work in the queue is a row in a jobs table, and the row itself is the only thing that needs to know whether it's currently being worked. Three columns carry that: a leased_until timestamp, a claim_count integer, and a heartbeat_at timestamp updated while the job is in flight. A worker claims a job by atomically setting leased_until to a few minutes in the future — not by taking an application-level lock that vanishes if the process dies while holding it, but by writing an expiry into the row itself, in the same transaction that reads it.

UPDATE jobs
SET leased_until = now() + interval '5 minutes',
    claim_count = claim_count + 1,
    heartbeat_at = now()
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'pending' AND leased_until < now()
  ORDER BY priority DESC, created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING *;

Claiming the next available job, with SKIP LOCKED so concurrent workers never block each other on the same row

While the job runs, the worker periodically bumps heartbeat_at and extends leased_until — proof of life, not just proof of claim. If the process dies mid-job — an OOM kill, a deploy that recycled the container, a panic that took the goroutine down without a chance to clean up — nothing has to notice and react. The lease simply expires. The next worker that polls for pending work sees a row whose leased_until has passed and claims it exactly the same way it would claim a job nobody has touched yet. No process is watching for dead workers, no supervisor is reaping them; the row's own timestamp is the entire liveness check. claim_count records how many times that's happened, which turns "this job keeps failing" from an anecdote into a WHERE claim_count > 3 query.

Why not keep the no-code tool and bolt on monitoring

The instinct when a no-code workflow starts misbehaving is to point something at it — an uptime check on the trigger, an alert on the error webhook, a status page scraped from the vendor's run history. None of that touches the actual problem. Monitoring can tell you a workflow failed; it can't tell you where it stopped, because the no-code tool doesn't expose a resumable unit of work in the first place — a run either completed or it didn't, and "partway through" isn't a state its data model has a place for. Bolting on observability adds a second opaque box next to the first one: now there are two systems to reconcile instead of one, and the underlying workflow logic — the actual sequence of steps, the actual retry behavior — is still sitting inside a vendor's UI, where it can't be queried, versioned, or claimed by a lease. Fixing the opacity means moving the workflow logic itself into something that has rows, not adding a dashboard on top of the box it's still locked inside.

Per-provider rate limiting

A single global rate limit is the wrong shape for this problem, because the constraint isn't "how much traffic can this queue generate" — it's "how much traffic will this specific provider tolerate before it starts returning 429s or, worse, silently throttling." Every provider's limit is different: some count per second and some per minute, some share a budget across an entire account and some scope it per API key, and a global limiter generous enough for the least restrictive provider will get the most restrictive one banned.

The queue keeps a rate budget per provider, not per job and not globally. Before a worker dispatches a call, it checks out capacity from that provider's bucket; if the bucket is empty, the job goes back into the pending state with a short delay instead of retrying immediately, and the worker moves on to a job for a different provider that still has budget. That's the other reason a global limiter doesn't work here: one slow or heavily-throttled provider would otherwise starve dispatch to every other provider queued behind it, even though they have nothing to do with each other.

Every outbound call also carries the queue's own authentication rather than trusting whatever the destination happens to accept — a bearer token for calls the queue makes into its own internal control endpoints (enqueue, pause, replay a job by id), and an HMAC signature over the outbound payload for calls where the receiving side needs to verify the request actually came from this queue and hasn't been altered in transit. Neither of those is a courtesy; a queue that can silently be told to enqueue or replay work by anyone who finds the URL is not meaningfully safer than the no-code tool it replaced, just harder to notice being abused.

Child idempotency keys on outbound calls

A job that fails partway through and gets reclaimed by another worker is, from the outside, indistinguishable from a job that's simply slow — nothing about the retry says "this already ran once and failed after the API call went out but before the response came back." That's the dangerous middle case: the charge captured, the email sent, the record created upstream, and then the worker died before it could write that success back to its own row. Retrying naively repeats the side effect.

The fix is a child idempotency key, generated once per job and reused across every retry of that job, not regenerated per attempt. It's derived from the job's own identity — its row id plus a step identifier — so the key is stable no matter how many times the job is reclaimed, but distinct from every other job's key so unrelated work never collides on the same key by accident.

func childIdempotencyKey(jobID uuid.UUID, step string) string {
    h := sha256.Sum256([]byte(jobID.String() + ":" + step))
    return hex.EncodeToString(h[:])
}

Deterministic per-step key, independent of attempt number, so a retry reuses it

That key is sent as a header or field on every outbound call the job makes, and any provider that honors idempotency keys — which is most payment, messaging, and CRM APIs at this point — recognizes a repeat of the same key and returns the original response instead of executing the side effect a second time. The queue doesn't have to know whether a given retry is the first attempt or the fifth; it just has to always send the same key for the same unit of work, and let the provider's own idempotency handling absorb the duplication. Retries stop being a source of risk and become what they should have been all along — a mechanical detail the caller doesn't have to reason about.

Shadow and dry-run execution

Every no-code migration carries the same risk in reverse: the workflow it replaces might behave subtly differently once it's rewritten, and the only way to find out for certain is to run it against real traffic — but running an unproven automation against real traffic is exactly the thing this whole system exists to make safe, not to do carelessly. The queue resolves that by running the same engine, the same jobs, the same trigger conditions, in two modes that stop short of the real thing.

In dry-run mode a job runs its full logic — reads, transforms, decisions — right up to the point of an outbound call, and then logs what it would have sent instead of sending it. That's the cheap check: does the automation make the decisions a human expects it to make, given real inbound data, without the risk of an email or an API call actually going out.

Shadow mode goes a step further, and it's the one that actually earns trust before a switch gets flipped. A shadow job runs alongside the automation it's meant to replace or supplement, triggered by the same live events, and its own outbound calls are still suppressed or routed to a sandboxed endpoint — but its decisions, its branch choices, its computed payloads are recorded and compared against what the system it's replacing actually did. Disagreements between the two are the signal that matters: not "did the new automation crash," but "did it decide something different than the old one would have, on this specific real input." A new automation only earns the right to act — to leave dry-run and shadow behind and actually fire its outbound calls — once it's been run this way against enough real traffic that its disagreements with the old behavior are understood, not just absent.

That's a materially different bar than what a no-code tool's built-in test mode usually offers, which is a single synthetic example clicked through in the editor. Shadow mode's test data is whatever production is actually sending that day, which is the only data that reliably contains the edge cases a synthetic example misses.

Tradeoffs and limitations

• Migrating a no-code workflow into the queue is a one-by-one rewrite, not a bulk import — each automation's logic has to be re-expressed as queue jobs by hand, so the payoff arrives gradually as workflows migrate, not on day one.
• Lease and heartbeat timing is a tuning problem, not a solved one: a short lease detects a dead worker fast but risks reclaiming a job that's merely slow, running it twice; a long lease avoids that false positive at the cost of a stuck job sitting idle longer before anyone notices.
• Per-provider rate limits are a snapshot of terms that change. A provider tightening its limits, or moving to a different accounting window, means the budget configuration has to be revisited — it isn't something the queue infers on its own from response codes alone.
• Child idempotency keys only work if the receiving API actually honors them. Most modern payment and messaging APIs do; a legacy or internal API that ignores the idempotency header offers no such protection, and a retry against it is back to being a genuine risk.
• Shadow mode only surfaces disagreements that show up in the traffic it actually sees. A rare trigger condition that hasn't occurred yet during the shadow window is still untested when the automation goes live.
• None of this replaces judgment about which automations are worth the migration effort. A workflow that runs twice a year with no external side effects gets little from being rewritten into a durable queue; the ones worth moving are the ones already causing the pain described above.

References and further reading

The individual pieces here are standard patterns, not novel inventions — the value was in combining them into one boring, queryable system instead of a dozen opaque ones:

• River — a Postgres-backed job queue for Go
• Crunchy Data — Devious SQL: message queuing using native PostgreSQL
• Stripe — Idempotent requests
• Stripe — Rate limits
• gocron — job scheduling for Go