Overview

The constraint above is the entire brief: a growing product surface calling out to several AI model providers, each with its own SDK, its own rate limits, its own failure modes, and its own pricing — and no single place able to answer what a request cost, which provider served it, or what happens if that provider goes down. Some callers just need a response back once a job finishes; others need tokens streamed to a user in real time as they're generated. Whatever sits in front of the providers has to serve both kinds of caller without turning into two separate systems, and it has to keep concurrency limits, failover behavior, and cost accounting identical regardless of which path a given request takes. One gateway, many providers, no per-feature reimplementation of any of it.

The adapter contract: one interface, many providers

Every provider adapter in the gateway implements the same interface: given a request — model name, messages, parameters — it either returns a response or streams one, and it reports back what the request cost and how long it took. Nothing above that interface knows or cares which commercial API actually served a given call; the router, the metering, and the failover logic are all written once, against the interface, never against a specific provider's SDK.

type ProviderAdapter interface {
    Complete(ctx context.Context, req Request) (Response, error)
    Stream(ctx context.Context, req Request) (<-chan Chunk, error)
    HealthCheck(ctx context.Context) error
}

The contract every provider adapter satisfies

Writing a new adapter means implementing these three methods against one provider's actual SDK, translating its request and response shapes, its error codes, and its usage-reporting fields into the gateway's own types. Everything downstream of that — routing, limits, failover, cost aggregation — already exists and doesn't change.

Why not call each provider's SDK directly from feature code

The alternative is what most teams start with: each feature imports whichever provider's SDK it needs and calls it directly. That works until a second provider shows up, and then a third, and every one of those call sites has to separately handle a different rate-limit header format, a different retry policy, a different way of reporting token usage, and a different shape of error. Multiply that by however many features call into a model, and it's the SDK's quirks — not the product's logic — that end up duplicated across the codebase.

Worse, none of it composes. A concurrency limit added at one feature's call site doesn't apply to a different feature calling the same provider from somewhere else. A failover added for one integration doesn't help another one hitting the same outage. And cost, scattered across as many call sites as there are features, can only be reconstructed after the fact — from logs, from provider dashboards, from a script someone writes once that goes stale the moment a new feature starts calling a model directly. Centralizing behind one adapter contract isn't an aesthetic preference; it's the only way any of those concerns gets enforced once instead of reimplemented per caller.

Concurrency limits and the persisted request queue

Every provider caps how much concurrent traffic it will accept for a given model, and that cap is a property of the provider account, not of any single feature calling it. If each feature enforced its own limit independently, the aggregate across all of them could still blow past what the provider allows — the individual limits would each be locally correct and collectively wrong. The router owns this instead: concurrency limits are configured per model, in one place, and enforced against the combined traffic from every caller, not against any one caller's view of the world.

Requests that arrive after a model is already at its concurrency ceiling don't fail outright — they queue. The queue is in-memory for latency but persisted, so a router restart doesn't silently drop work that was already accepted; a request the gateway acknowledged is a request the gateway is responsible for eventually completing or explicitly failing, not one that quietly vanishes because the process holding it in memory happened to restart. As capacity frees up — a slot at the provider opens because an earlier request finished — the router pulls the next queued request and dispatches it, in order, without the caller needing to poll or retry.

This is also where a provider's own rate limits get absorbed instead of propagated outward. A caller that would otherwise see a rate-limit error from the provider instead sees its request wait briefly and then complete — the limit is real, but managing it is the router's job, not something every feature has to catch and retry around on its own.

Provider health and automatic failover

Concurrency limits assume the provider behind them is healthy; they don't do anything about a provider that's up but degraded — elevated error rates, latency creeping past what a streaming caller can tolerate, or an outright outage. The router tracks health per provider from the outcomes of the requests already flowing through it: a rising error rate or a run of timeouts marks a provider degraded, and new requests for the models it serves get routed to a healthy alternative instead, with no caller-side code ever deciding to retry against a different provider.

Diagram: provider routing — a request enters the gateway through either the async REST or streaming gRPC path, the router applies per-model concurrency limits, checks provider health and routes to a healthy provider (failing over automatically if the primary is degraded), the response streams or returns to the caller, and the request is metered against session, daily and total cost counters throughout.

Failover here only works because of the adapter contract underneath it: because every provider adapter exposes the same Complete/Stream methods, the router can substitute one provider for another without the caller — or the rest of the gateway — needing to know a substitution happened. That substitution is invisible by design; the caller asked for a model, not a specific upstream account, and getting an answer from a healthy provider is what it actually wanted.

Dual transport: async REST and streaming gRPC

Not every caller wants the same thing back. A batch job scoring a few thousand records overnight wants to submit a request and pick up the result later — it doesn't need a connection held open for the seconds or minutes a large job might take, and it would rather poll for a result than hold a socket open. An interactive feature streaming a response into a chat UI wants the opposite: the first token as soon as it exists, not the whole response buffered and returned at once. Forcing either caller onto the other's transport is a bad trade in one direction or the other — polling for a result that could have streamed adds latency the user feels, and holding an open stream for a batch job that doesn't need one wastes a connection for no benefit.

The gateway exposes both: an async REST endpoint for batch and fire-and-forget callers, and a streaming gRPC endpoint, defined over protobuf, for interactive callers that want tokens as they're generated.

service Gateway {
  rpc Complete(CompletionRequest) returns (CompletionResponse);
  rpc Stream(CompletionRequest) returns (stream CompletionChunk);
}

One schema, two ways to consume the same request

Both transports terminate in the same router, the same concurrency limits, the same failover logic, and the same metering — the transport a caller picks changes how a response is delivered, not which provider serves it, what it costs, or what happens if that provider turns out to be unhealthy. A gateway that instead ran REST and gRPC as separate deployments, or worse, separate codebases, would have to keep two implementations of every one of those concerns in sync; here there's exactly one.

Live cost metering

Every provider adapter reports back what a completed request actually cost — the token counts and the pricing that applied to them — and the router aggregates that figure the moment the request finishes, not on a schedule. That figure rolls up into three counters at once: the session the request belongs to, the calendar day it happened on, and a running total. None of those are computed after the fact from logs; they're incremented as part of the same code path that returns the response, so the number a caller or a dashboard reads is never more than one completed request behind reality.

Making metering a router-level concern rather than a per-feature one is what makes the aggregation trustworthy in the first place. If every feature reported its own spend, that report would only be as complete as the discipline of every engineer who ever added a call site — one omitted call, and the total silently undercounts. Because every request already has to pass through the router to reach a provider at all, cost accounting for it is not optional and not something a new feature could accidentally skip.

Tradeoffs and limitations

None of this is free, and being honest about what it doesn't solve is part of the design:

• A uniform adapter contract can only expose what every provider has in common; a provider-specific capability that doesn't fit the shared interface either gets left out or forces the contract itself to grow, at which point every other adapter has to at least tolerate the new method.
• Failover changes which provider actually serves a request, and different providers can produce different output characteristics — different phrasing, different latency, different behavior at the edges of a prompt — even when both are "healthy" by the router's own definition; a caller sensitive to that difference has no visibility into when a failover happened mid-session.
• Concurrency limits are only as accurate as their configuration, and a provider's real limits and pricing shift on the provider's schedule, not the gateway's — someone has to keep the router's per-model configuration current as those terms change, or the limits either throttle unnecessarily or stop protecting the account at all.
• Centralized metering means the cost figure the business trusts has exactly one source; a bug in that one place — a misreported token count from a provider, an aggregation error — is wrong everywhere at once, with no independent per-feature figure left to cross-check it against.
• The persisted queue smooths over a provider being temporarily at capacity, but it doesn't manufacture capacity that doesn't exist; a queue that keeps growing because every provider for a model is simultaneously constrained just delays the failure instead of preventing it.
• A genuinely new provider capability — not a quirk, a capability none of today's providers has — still means extending the adapter contract itself before any caller can use it; writing an adapter only gets a team as far as the interface already goes.

References and further reading

The patterns here aren't original — they're well-established, and worth reading in their original form:

• gRPC documentation
• Protocol Buffers documentation
• microservices.io — API Gateway pattern
• Refactoring.Guru — Adapter pattern
• Martin Fowler — CircuitBreaker