Overview

The constraint above is the entire brief: a serverless GPU worker is cheap precisely because it is disposable, and a system that treats its disappearance — or the loss of whatever socket happens to be open to it — as an exceptional event has already lost the argument. A single inference job here can run for minutes, produce a large binary artifact, and cross a mobile network handoff, a laptop sleep cycle, or a load balancer's idle timeout somewhere along the way. None of those are edge cases; they are the median outcome of running anything long enough on rented, ephemeral hardware. The orchestration layer sits between the client and RunPod-hosted ComfyUI workers with a single job: make disconnection, worker churn, and abandoned jobs look like routine operating conditions instead of incidents a user has to report.

Enqueue and poll instead of trusting the socket

A job doesn't start by opening a socket and hoping. It starts by being enqueued: the client submits the job, gets back a job ID, and that ID — not any particular connection — is what the rest of the system tracks. Progress lives server-side, keyed by job ID, and is queried rather than delivered as though a push were the only source of truth. A WebSocket is layered on top purely as a low-latency notification channel: when it's open and healthy, progress updates arrive quickly; when it isn't, the client falls back to polling the same job ID on a fixed interval and sees exactly the same state a live socket would have delivered, just a little later.

That separation is what makes everything downstream possible. Because a job's status is a row the server can answer questions about, "is this job still running" is a query, not a property of whether a particular browser tab happens to still be open. A client can close, reopen, refresh, or hand off to a different device entirely, and the answer to "where is my job" doesn't change.

Why not rely purely on the WebSocket connection

Treating the socket as authoritative — the job exists exactly as long as the socket does — ties a multi-minute GPU job to the least reliable part of the whole stack. Sockets close for reasons that have nothing to do with the job's health: a laptop lid closes, a mobile carrier hands the device to a new tower, a corporate proxy or load balancer enforces its own idle timeout regardless of what's still in flight. If the job lived only in that connection's memory, every one of those routine events would silently kill a job the user has no reason to think has failed. Worse, the worker itself would have no way to distinguish "the client is still watching, just disconnected" from "the client is gone, tear this down" — precisely the distinction the whole system depends on getting right. Making enqueue-then-poll the source of truth, and the socket a convenience layered on top of it, turns a connection drop into a UX inconvenience — the progress indicator freezes for a beat — instead of a lost job.

Resumable, checkpointed state

Enqueue-and-poll solves finding a job again; it doesn't by itself solve not losing the work already done. A long inference job — multi-step diffusion, frame-by-frame video processing, anything measured in minutes rather than seconds — has real, expensive progress sitting in GPU memory partway through, and restarting from zero on every disconnect wastes exactly the compute this whole exercise exists to avoid wasting.

The fix is to make progress durable at each meaningful step, rather than only in the worker's live memory. As the job advances, the worker writes a checkpoint — which step it's on, what intermediate state or partial artifact exists, what's left to do — to storage that survives independently of any socket, any single worker process, or any client connection. A client that reconnects, whether after a two-second blip or a five-minute gap, doesn't ask "did my job survive"; it asks "what's the current checkpoint," and resumes its display — and, where the job itself supports it, its computation — from there instead of from scratch.

checkpoint = {
  job_id: "job_8f2c",
  step: 34,
  total_steps: 50,
  last_artifact_ref: "s3://inference-out/job_8f2c/step_34.png",
  updated_at: "2026-06-30T14:22:07Z",
}

Representative shape of a durable checkpoint record

Nothing here requires the checkpoint format to be exotic — a step counter, a pointer to the last durable artifact, and a timestamp is usually enough to answer "where do I resume" correctly. What matters is that the record lives outside the failure domain of the thing most likely to fail: the live connection.

Bounded exponential backoff on reconnection

A dropped connection still needs to come back, and how it comes back matters almost as much as the fact that it does. Reconnecting immediately in a tight loop is the wrong default: a worker recovering from its own blip, or a network path that's degraded rather than dead, gets hammered by retry traffic at the exact moment it's least able to absorb it. Waiting indefinitely is equally wrong in the other direction — a client that never gives up on a truly gone worker just sits there, and a user watching it has no signal that anything is actually wrong.

The middle path is bounded exponential backoff: each failed reconnect attempt waits longer than the last, up to a ceiling, so the very first retry is fast — a real blip recovers almost invisibly — while a sustained outage backs off to a sane interval instead of continuing to hammer the system. A maximum attempt count or maximum elapsed time bounds the whole thing, so a truly dead job eventually and deliberately surfaces as a failure rather than retrying forever in the background.

delay = min(base_delay * 2^attempt, max_delay)
delay = delay * random(0.5, 1.0)   # jitter
if elapsed > backoff_ceiling: surface_as_failed(job_id)

Reconnect backoff: exponential growth, a ceiling, and jitter to avoid synchronized retries

Diagram: inference job recovery — a job is enqueued and progress is polled rather than trusted to a single socket; a mid-job WebSocket drop triggers bounded exponential-backoff reconnection; the client resumes from its last durable checkpoint instead of restarting; large binary output is offloaded to S3 as it completes; and the worker is explicitly torn down on completion or abandonment to free GPU/VRAM.

Jitter is easy to skip and expensive to skip: without it, a network blip that affects many concurrent jobs at once produces synchronized retry waves that hit the backend in lockstep instead of smoothly spreading out.

S3 offload for large binary output

By the time a job finishes, the connection that has been the least reliable part of this whole system is the last place that should be responsible for delivering the largest payload. A generated image, and especially a generated video or a batch of frames, can run from megabytes into the hundreds of megabytes — exactly the kind of payload that turns a marginal connection from "will probably recover" into "will time out partway through a multi-minute transfer."

Instead of streaming the finished binary back over the same channel used for progress updates, the worker uploads it directly to object storage as it's produced, and the job's checkpoint record is updated with a reference to that object rather than the bytes themselves. The client's job is reduced to fetching a URL over a plain, resumable HTTP request once the job reports done — a well-understood problem with mature tooling, rather than a custom transfer riding on the same fragile channel that already needed retry logic once.

This also means a job producing output incrementally — frame by frame, for instance — doesn't need to hold the entire result in memory or in a single connection until the very end. Each finished piece can be offloaded and referenced as soon as it exists, so a late failure loses only what hasn't been offloaded yet, not everything produced up to that point.

Explicit worker teardown

Serverless GPU pricing is a direct function of how long a worker stays up, so a worker that isn't explicitly told to stop doesn't stop — it sits there, billed and holding VRAM, until something notices. Left implicit, that "something" tends to be a person checking a bill, which is much too late.

Teardown is triggered explicitly, and from more than one direction. A job that completes tears down its worker once its output is confirmed offloaded. A job that fails tears down its worker immediately rather than leaving it idle "in case." And a job whose client never comes back — no reconnect, no poll, nothing for longer than the same bounded window the reconnection backoff already uses — is treated as abandoned and torn down on a timeout, rather than left running indefinitely on the assumption someone might still be watching.

That last case is the one worth being deliberate about. The abandonment timeout has to be long enough that a client legitimately recovering from a slow network or a backoff cycle isn't torn down out from under itself mid-recovery, but short enough that a genuinely abandoned job doesn't sit pinning an expensive GPU on the off chance someone comes back. Tying it to the same time horizon the reconnection logic already uses keeps the two policies from working against each other.

Tradeoffs and limitations

None of this is free, and it's worth being explicit about what it costs and doesn't solve:

• Polling has a latency floor a live push channel doesn't: even at a short interval, progress updates lag slightly behind what a healthy socket would deliver instantly.
• Checkpointing has a granularity tradeoff — checkpoint too rarely and a drop loses more work; checkpoint too often and the writes themselves become overhead on jobs that are already GPU-bound.
• Bounded backoff means a long enough outage is still, eventually, a failure. It buys time for transient problems to resolve; it doesn't turn a genuinely dead network path or a permanently gone worker into a recoverable one.
• Explicit teardown is only as good as the abandonment detection behind it — get the timeout wrong in one direction and a slow-but-legitimate client gets torn down mid-recovery; get it wrong in the other and a truly abandoned job keeps billing longer than it should.
• S3 offload removes the connection as a bottleneck but adds a new dependency with its own failure mode: an upload can itself fail or stall, and a job can reach "done" before its output is confirmed durably stored unless that ordering is enforced deliberately.
• All of this adds real complexity — queue, checkpoint store, backoff logic, offload target, teardown policy — on top of what would otherwise be a single request and a single response; it's justified here because the alternative is silently losing paid-for GPU compute, not because it's simpler.

References and further reading

• RunPod documentation
• ComfyUI documentation
• AWS Builders' Library — Timeouts, retries, and backoff with jitter
• AWS S3 — Multipart upload overview
• MDN — The WebSocket API