Reproducible gateway overhead measurement on neutral hardware. Same box, same mock upstream, same load, same CPU pin for every gateway; every number regenerates from committed JSON.
Reproducible, honest benchmarks for the AI tooling you actually run.
Every entrant is measured the same way, on neutral hardware: same box, same mock upstream, same load, same CPU pin.
Every number regenerates from committed JSON. Fully open source, re-runnable by anyone.
Built and operated by the Busbar team, who are also one of the entrants. Same harness for everyone. A failure is our bug until proven the gateway's.
Every gateway on the board, in alphabetical order. Language, the exact version measured (hover for the full build/image digest), when it was last benchmarked, project age, stars (snapshot), and each project's own self-description. The other tabs measure how they perform.
Gateway
Language
Version
Last benchmarked
Project Age
Stars
Class
Click a column to sort, a row for full detail. A muted cell was not measured. Every view is shareable via its URL.
Recovery across the field
Protocol support matrix (6x6)
probe passed (translation verified)
not configured (probe failed; evidence on hover)
not verified (harness could not get it serving)
untestable (mock limit)
same-dialect passthrough
Probe-first: every ingress→upstream pairing of every gateway is attempted with a correctness-checked round trip (response validated in the ingress protocol's shape, upstream leg verified against the recording mock). A green cell passed; a grey cell's probe failed and its hover shows the probe's own evidence. No gateway is graded red for a pairing it does not serve.
Protocol matrix results are coming soon: no results/matrix JSONs are committed yet.
Method
Who runs this. Built and operated by the Busbar team; busbar is one of the entrants. Same shared harness for every gateway, no per-gateway special-casing, every number from committed JSON, fully open source. Read the code and re-run it yourself. Gateway creators are welcome to PR and own their own gateway.sh.
We run the 6×6 protocol matrix once. Every view is a pick of that one run.
One run, one source. For each gateway the harness sweeps the full 6×6 protocol matrix — every ingress dialect the client can speak against every upstream dialect the mock can speak — a single time. Latency, throughput and streaming are all read out of that one run; there is no separate perf/stream suite to disagree with it. Memory is measured in a dedicated window after that same run, on each gateway's peak cell under an identical fixed load (see the Memory card) — one producer, matrix/run.sh, still the single source for every number.
One rig. One box, one instant Rust mock upstream, one Go load generator, fixed CPU pinning (gateway on one core set, mock and loadgen on the others).
The instrument records itself. The mock and load generator are prebuilt binaries fetched from a release tag that CI rebuilds whenever they change — so two runs of a byte-identical harness can legitimately disagree because the instrument moved between them, not because a gateway did. Each run therefore stamps the SHA-256 of the exact mock and loadgen binaries it executed (plus the release asset's timestamp) into its own result file, and the short mock digest is shown in the footer. If two runs are ever compared, whether the measuring instrument changed is a fact on the page rather than an investigation.
Added latency is gateway-minus-direct. The same load runs straight against the mock and that baseline is subtracted, so the figure is what the gateway adds, not the network or the mock.
Throughput is each gateway's own qualifying ceiling, not a subtraction. A guardrail flags any run approaching the mock's own ceiling, so the mock is never silently the bottleneck; a rig-limited number reads n/a rather than a fabricated bar.
Every result is stamped with hardware, architecture, build version and timestamp. A gateway is its own directory: a one-file manifest declares how to build, launch and probe it; add or remove a directory and it appears or disappears everywhere.
The one run: the 6×6 protocol matrix
A capability probe on every cell, then a per-cell latency/throughput/streaming sweep on every cell that passes. This is the single producer every tab reads.
Probe first. One correctness-checked round trip per (ingress, upstream) cell across the full 6×6 — OpenAI, OpenAI Responses, Anthropic, Gemini, Cohere, Bedrock Converse — relaunching the gateway per upstream dialect. The response is validated in the ingress protocol's shape and the upstream leg is verified against the recording mock (which records the endpoint the gateway actually called). A failed probe is not configured with its evidence, never a red.
Then sweep. Every passing (green) cell gets the same perf sweep on that path: added latency p50/p99 at concurrency 1, sustained RPS at 20 ms, and max proxy RPS. Each ceiling is found by a bidirectional peak search then a unimodal refinement (a fixed grid would miss a peak between rungs); the winning concurrency is recorded with it. At 20 ms Little's law bounds throughput (RPS ≤ concurrency ÷ 0.020 s), so that peak sits at much higher concurrency — the real in-flight capacity bottleneck.
Streaming on the same cell. The paced-SSE measurement runs on the swept cell too, so the streaming figures come from the same run as the latency/throughput.
Fairness. Containerised gateways run with nofile raised to 1 048 576, matching native gateways and the load generator, so a fast container can't collapse on the kernel's ~1024-socket default mid-sweep. Applied uniformly.
Every view is a pick of that run
Peak — each gateway on its OWN best same-dialect diagonal (best-of, pure forwarding, no translation). The Tested-on pill names the dialect (OpenAI when served, else the gateway's fastest native one), so every gateway appears on its strongest path and the dialect is never hidden.
Same — one chosen dialect's diagonal (X→X) for every gateway, an identical-path apples-to-apples comparison. A gateway that does not serve that dialect reads n/a.
Custom — any ingress→egress cell for every gateway, including a translation cell (client speaks one dialect, upstream another, the gateway translates both ways). In→out is pinned by two dropdowns; picking the same on both sides is that dialect's strict passthrough.
Streaming uses the SAME three picks for the SSE columns (added TTFT p50/p99, added gap p50/p99, streams sustained, CPU-bound fps).
One value, everywhere. The table, the matrix hover popup, the drawer and the charts all read the same per-cell record through the same accessors, so a value shows identically on every surface (a build-time guard fails the deploy if they ever diverge). The Protocol matrix tab is the visual face of Custom: hover a cell for its gated numbers plus its Δ versus the gateway's own peak; click it to open that cell in Custom.
Memory is a dedicated window that runs after the full 6×6, per gateway. First the 6×6 runs; then the harness picks each gateway's peak cell — its served cell with the highest max-proxy throughput (there is no universal cell, since some gateways do not serve OpenAI→OpenAI) — and runs an identical fixed load on it. Every gateway gets the same load recipe, so the memory comparison is apples-to-apples; only the cell differs (shown under Tested on). The recipe (concurrency × payload bytes × duration) travels in each gateway's record.
Fresh cold restart. The gateway is killed and cold-restarted for the peak cell's egress dialect, on a brand-new process — so idle is a true cold-start baseline, not a warm post-6×6 process (which would measure recovery, not idle).
Idle. RSS sampled for 60 s with no traffic on the fresh process; reported as the median of that cold-idle window.
Peak. The MAX RSS sampled continuously while the fixed load runs on the peak cell — the real working-set peak under a fair, identical load, not a synthetic burst.
Recovered @60 s. RSS at the end of a 60 s window after the load stops — whether the gateway releases what it took or stays pinned near its peak. This, not raw peak, is the memory story: a gateway that peaks near 1 GB and settles to tens of MiB behaves very differently from one that holds it.
Recovery curve. RSS sampled at a steady cadence across the whole window on one process lifecycle — cold idle → fixed load → recovery — drawn as an idle→peak→recovery sparkline on the Memory tab. A gateway with no served cell reads n/a, never a fabricated line.
Honesty rules on every surface
Sole-source and null-safe. A value a record lacks renders n/a, never 0 and never a fabricated number. A measured 0 (served, but no tested load held p99 < 1 s at < 0.1% errors) is shown as 0 with its note, distinct from a rig-ceiling.
Rig-limited numbers are suppressed. A throughput or frames-per-second the harness could not certify as gateway-limited (the sweep saturated near the mock's own ceiling) reads n/a and draws no bar — it never ranks or wins a comparison.
No gateway is graded red for a pairing it does not serve. Grey cells carry their own probe evidence; a boot/build failure reads “did not run”, a measured streaming refusal reads “did not stream” — measured, not hidden.
Every number regenerates from committed JSON, and the per-gateway Download results is the full 6×6 record the board is built from.
Why we measure /v1
The unified OpenAI-compatible route every SDK points at, on the same terms for every gateway.
Why we measure /v1, not passthrough endpoints
We measure the unified OpenAI-compatible route — the endpoint a stock SDK points at — for every gateway, on the same terms.
Some gateways also expose a separate passthrough route: a provider-specific endpoint that forwards to one named upstream and skips the gateway's protocol translation. It is faster, but it is a different thing than what most integrations use.
It is the path people actually take. A passthrough route means changing your code and hard-coding which provider you call. The unified endpoint is the "point your SDK here and go" experience these gateways are built to sell.
Not every gateway has one. A passthrough column would pit a special route on some gateways against the normal route on others — apples to oranges.
It is pinned to one provider. The provider is part of the URL path, so calls on it bypass the cross-provider routing and failover the gateway offers on its unified endpoint. Measuring it would rank a single-provider forward against the multi-provider route people actually deploy.
Charts
Generated by the repo's chart script from the committed result JSONs. Bars are coloured by implementation language (a neutral property), not by rank or brand, so the colour never favours any gateway.
