Self-built · Open source
One endpoint in front of every LLM provider. Self-hosted, bring-your-own-keys, built for the hot path.
Every project I run talks to LLMs, and every one of them used to carry its own pile of glue code — key rotation, retries on 429, per-provider quirks, no idea what anything cost. Relay is that glue extracted into a service: a gateway in Go that speaks OpenAI- and Anthropic-compatible APIs, pools provider keys behind policies and rate limits, and meters every request. It fronts the LLM traffic for everything else I build.
Features
Projects authenticate with disposable relay keys — rate-limited, usage-tracked, revocable on their own. The real provider credentials stay inside the gateway; a leaked key caps the damage at whatever limits it was minted with.
Swap a model — or an entire provider — and the codebase doesn't move. The endpoints are OpenAI- and Anthropic-compatible, translation between wire shapes is built in, and routing lives in the model string, not in application logic.
The inference plane is stateless and cluster-shaped by design: every pod holds the same in-memory catalog snapshot, config writes propagate fleet-wide in about a second, and scaling is a matter of adding pods.
Model definitions come from an open catalog and can be overridden at every level — catalog, overlays, policies. Customizing a model is a sparse patch that survives catalog upgrades, not a fork.
Apache-2.0 across the board. The engine, the model catalog, the admin UI, and the docs are separate public repos: relay, relay-catalog, relay-ui, relay-docs.
Benchmarks
All measured in my cluster under sustained load, not a synthetic bench.
Challenges
Staying compatible with the upstream schemas — OpenAI Chat Completions and Responses, Anthropic Messages — while translating between them means chasing specs that providers keep extending. Every new field, streaming event, or tool-call variant has to survive the round trip through the canonical protocol.
Access rules had to be flexible enough to express real setups — this key gets that model on this host, under these limits — and enforced by one generic evaluation path. No special cases in code: if a rule can't be expressed as configuration, the policy model is wrong, not the code.
War story
Partway through a production-environment campaign of 120,000 varied test requests, things went wrong in the way that actually teaches you something: requests started dropping, and the frontend was catching errors almost constantly. The proxy's own numbers looked fine — overhead steady, no saturation — which made it worse. Something outside the code I was testing was eating requests.
The trail led to Valkey. Operations were stalling at random, blowing past their two-second budget for no reason the metrics could explain — and every stalled operation turned straight into a dead request.
The real culprit was three layers down, in a resolv.conf I had never read. The Valkey address in config was a short service name, not the FQDN — and Kubernetes ships resolv.conf with ndots:5, so every resolution of that short name fanned out through the whole search-domain list, several DNS queries per lookup. It happened per connection, uncached. A handful of concurrent requests multiplied into enough queries to overload resolution, connection setup stalled, and the two-second budget did the rest.
The fix came in three layers. Config now speaks the full FQDN — the namespace.svc form — so a lookup is one query, no fan-out. A node-local DNS cache keeps repeat lookups from ever leaving the node. And the Valkey pool is fully static and pre-dialed at boot (commit 3013bff), so no request ever pays a dial — DNS or TCP — inside its own budget. Reran the campaign: zero drops.
The proxy was never the bottleneck; the platform under it was. Load testing the code tests the code — testing in production tests everything you forgot you were depending on.
Highlights
Going in, the goal was to keep proxy overhead under 10 ms. Sustained load testing at 5k req/s — long-running and streaming requests included — landed at about 1 ms, most of it a single Valkey roundtrip, with zero drops and CPU and memory costs that barely register at that scale. The decision path itself — auth, policy, routing — measures 30–50 µs.
One small DSL, provider/model@host, expresses a huge combination space for free: bare model, model@host, provider/model, provider/model@host, plus wildcard aliases. Pin one host, spread one model across every host that serves it, or anything in between — and the string drops into any OpenAI or Anthropic SDK's model field unchanged and routes directly.
The common algorithms — token bucket, sliding window, fixed window, leaky bucket, session window — implemented as Lua scripts executed on Valkey. Each one is atomic and faithful to its algorithm's semantics, and the server has no per-algorithm code paths: picking a strategy is configuration, not a branch.
Get started
The whole thing — API, admin UI, database, and a pre-seeded model catalog — starts with a single Docker command. Docs, quickstart, and the source are public.