Reproducible, honest benchmarks for the AI tooling you actually run.
Every entrant is measured the same way, on neutral hardware: same box, same mock upstream, same load, same CPU pin.
Every number regenerates from committed JSON. Fully open source, re-runnable by anyone.
Built and operated by the Busbar team, who are also one of the entrants. Same harness for everyone. A failure is our bug until proven the gateway's.
Benchmark repository on GitHub · How we measure
Every gateway on the board, in alphabetical order. Language, the exact version measured (hover for the full build/image digest), when it was last benchmarked, project age, stars (snapshot), and each project's own self-description. The other tabs measure how they perform.
Click a column to sort, a row for full detail. A muted cell was not measured. Every view is shareable via its URL.
Protocol support matrix (6x6)
- probe passed (translation verified)
- not configured (probe failed; evidence on hover)
- not served (legacy pre-probe-first result)
- not verified
- untestable (mock limit)
- unprobed (auth)
- same-dialect passthrough
Probe-first: every ingress→upstream pairing of every gateway is attempted with a correctness-checked round trip (response validated in the ingress protocol's shape, upstream leg verified against the recording mock). A green cell passed; a grey cell's probe failed and its hover shows the probe's own evidence. No gateway is graded red for a pairing it does not serve.
Protocol matrix results are coming soon: no results/matrix JSONs are committed yet.
Method
Who runs this. Built and operated by the Busbar team; busbar is one of the entrants. Same shared harness for every gateway, no per-gateway special-casing, every number from committed JSON, fully open source. Read the code and re-run it yourself. Gateway creators are welcome to PR and own their own gateway.sh.
Every gateway is measured the same way, on neutral hardware:
- One rig. One box, one instant Rust mock upstream, one Go load generator, fixed CPU pinning (gateway on one core set, mock and loadgen on the others).
- Added latency is gateway-minus-direct. The same load runs straight against the mock and that baseline is subtracted, so the figure is what the gateway adds, not the network or the mock.
- Throughput is each gateway's own qualifying ceiling, not a subtraction. A guardrail flags any run approaching the mock's own ceiling, so the mock is never silently the bottleneck.
- Every result is stamped with hardware, architecture, build version and timestamp.
- A gateway is its own directory. A one-file manifest declares how to build, launch and probe it; add or remove a directory and it appears or disappears everywhere.
perf/run.sh Latency and throughput
- Added latency (p50/p99, microseconds): at concurrency 1, gateway p99 minus direct-to-mock p99, small payloads.
- Max proxy RPS: highest requests per second against an instant mock, found by a peak search — concurrency ramps (bidirectionally) until RPS stops rising, then a unimodal refinement locates the true peak — subject to p99 < 1000 ms and error rate < 0.1%. This replaces a fixed concurrency grid, which could miss a peak sitting between rungs and understate the fastest gateways. The winning concurrency is recorded alongside the RPS figure.
- Sustained RPS at 20 ms: the same peak-search method and the same gates (p99 < 1000 ms, < 0.1% errors), but the mock sleeps 20 ms per request to model real LLM latency, so the ceiling is concurrent-in-flight capacity — the real production bottleneck. At a 20 ms mock delay, Little's law bounds throughput (RPS ≤ concurrency ÷ 0.020 s), so the peak typically sits at much higher concurrency than the instant-mock sweep; the bidirectional ramp finds it regardless of which direction it lies from the search start. The winning concurrency is shown in the cell tooltip.
- File-descriptor fairness. Containerised gateways run with the Docker daemon's
nofile limit raised to 1 048 576, matching the limit native gateways and the load generator already have. Without this fix, a fast containerised gateway would hit the kernel's default socket limit (~1024 open files) mid-sweep and collapse — understating its true throughput. This is applied uniformly; no gateway is singled out.
- The Peak tab shows these metrics on each gateway's best same-dialect path (pure forwarding, no translation); the Tested-on pill names the dialect (OpenAI when served, else the gateway's fastest native one). Best-of, so every gateway appears on its strongest path and none is filtered out — the ceiling each gateway can post. For a strict, identical-path comparison, use the Matched tab.
- Idle RSS: process resident set after launch, before load.
- Peak RSS: maximum resident set while sustaining large-payload load at high concurrency.
- The mock answers stream:true with a paced SSE stream; the suite measures what the gateway adds on top of that pace.
- The mock's first token is instant (it then paces later tokens), so the first-token figure deliberately includes the gateway's first-write network behaviour, not just its compute. Against a real model, whose own first token takes hundreds of milliseconds, that component is largely hidden; here it is exposed on purpose. A benchmark whose mock waits before its first token, or that times the first SSE event of any kind rather than the first content token, will report a near-zero first-token overhead by construction.
- Added wait for the first token (p99), also called TTFT (time to first token): gateway first-content-frame time minus direct-to-mock, at concurrency 1.
- Added gap between tokens (p99): gateway content-frame gap minus direct-to-mock gap.
- Streams sustained: max concurrent streams where at least 99.9 percent of expected frames deliver, no stream stalls beyond twice the pacing interval, and the stream error rate stays under 0.1 percent.
- A gateway that answers 200 but buffers the whole response is recorded stream_served=false: measured, not hidden.
- The client speaks Anthropic (POST /v1/messages) while the upstream mock speaks OpenAI, so the gateway must translate both directions. The mock is untouched; that is the point.
- Added latency (p99) and sustained RPS at 20 ms on the translation path, same gates as perf. The direct baseline is the OpenAI shape straight to the mock, so the added-latency figure deliberately includes the translation work.
- The Matched tab pins both ends with a pair of dropdowns (client dialect and upstream dialect), so every row is the identical path and the ranking is apples-to-apples; only gateways that serve the chosen pair appear. Pick the same dialect on both sides and it becomes that dialect's strict passthrough.
- A gateway that cannot serve Anthropic ingress against an OpenAI upstream records xlate_served=false with the probe status and body snippet as evidence.
- Headline translation numbers (table, drawer, compare, charts) all come from one canonical record per gateway: the matrix per-cell sweep's OpenAI-in cell. This suite's Anthropic-to-OpenAI numbers are the fallback for a gateway with no measured matrix translation cell, and every surface labels the direction it shows.
Why we measure /v1, not passthrough endpoints
- Some gateways expose a separate passthrough route — a provider-specific endpoint that forwards your request to one named upstream and skips the gateway's protocol translation. It is faster, but it is a different thing than what most integrations use.
- We measure the unified OpenAI-compatible route — the endpoint a stock SDK points at — for every gateway, on the same terms.
- It is the path people actually take. Using a passthrough route means changing your code and hard-coding which provider you are calling. The unified endpoint is the "point your SDK here and go" experience these gateways are built to sell.
- Not every gateway has one. A passthrough column would compare a special route on some gateways against the normal route on others — apples to oranges.
- It's pinned to one provider. A passthrough route is named after a single provider — the provider is part of the URL path — so calls on it go to that one provider and bypass the cross-provider routing and failover the gateway offers on its unified endpoint. Measuring it would rank a single-provider forward against the multi-provider route people actually deploy.
- A capability suite first: one probe per (ingress, upstream) cell across the full 6x6 (OpenAI, OpenAI Responses, Anthropic, Gemini, Cohere, Bedrock Converse), relaunching the gateway per upstream dialect.
- Envelope validation per cell, a passthrough guard that rejects the mock's canned bodies as untranslated proxying, an upstream round-trip check (the mock records which endpoint the gateway actually called), and an unprobed_auth state for ingresses that demand signatures the harness does not forge.
- Per-cell perf: every passing (green) cell also gets the perf suite's exact sweep on that path (added latency at concurrency 1, sustained RPS at 20 ms, max proxy RPS). The Peak and Matched tabs read specific cells from this sweep; hovering any green cell shows that path's numbers and its RPS delta vs the gateway's reference cell (named, not called "best").
Charts
Generated by the repo's chart script from the committed result JSONs. Bars are coloured by implementation language (a neutral property), not by rank or brand, so the colour never favours any gateway.
