LLM systems
The machinery around the model call: gateways across providers, GPU inference orchestration, prompt optimization, and the fallbacks for the day an API goes down.
For teams past the prompt stage, now facing service boundaries, fallback paths, observability, cost control and evaluation.
Case studies behind this
A multi-provider AI gateway with built-in cost accounting
Direct model-provider calls spread SDK quirks, rate limits, failures, and pricing logic across feature code.
Go adapters · gRPC / protobuf · persisted in-memory queueOrchestrating long-running inference on serverless GPUs
Long-running image-generation jobs could lose work or leak GPU memory when a WebSocket dropped or an ephemeral worker disappeared.
RunPod · ComfyUI · WebSockets · S3 offloadEvolutionary prompt optimization
Prompt tweaks changed output quality unpredictably, and manual testing could not explore enough variants to find better prompts reliably.
BLEURT · LLM APIs · evolutionary operators · NumPyA local-first dictation pipeline for macOS
Cloud dictation was not acceptable for sensitive work, while context-free local transcription produced text disconnected from the current screen.
Swift / SwiftUI · whisper.cpp · Accessibility & CoreGraphics APIs
Questions worth asking first
How will latency, cost and fallback behavior be visible in production?
Which provider or model boundary should the product depend on?
Do we need RAG, fine-tuning, workflow design — or just a simpler prompt boundary?
Related services
LLM productionization
Start here when the prototype is convincing but nobody can say what it costs at scale, why it fails, or whether yesterday's prompt change made it worse.
Open serviceRAG systems and knowledge workflows
Start here when knowledge lives in documents, tickets or transcripts, and answers need grounding, citations and boundaries.
Open service