Method
Who runs this. Built and operated by the Busbar team; busbar is one of the entrants. Same shared harness for every gateway, no per-gateway special-casing, every number from committed JSON, fully open source. Read the code and re-run it yourself. Gateway creators are welcome to PR and own their own gateway.sh.
Every gateway gets one fresh box, one fixed config, and one measurement run. We probe every protocol pairing before we time anything, we subtract the rig's own overhead from every latency number, and we would rather publish "n/a" than a number we cannot stand behind. Below is exactly how that works, why we built it that way, and where it can still go wrong.
One gateway, one box, one config
The rig, and the promise it exists to keep.
Each gateway is measured on its own fresh AWS Graviton m7g.4xlarge, sixteen real Graviton3 cores, launched new for that gateway and terminated when its run finishes. Nothing another gateway built, cached, or left in page cache is ever on the box. Four of those cores are pinned to the gateway under test; the mock upstream and the load generator each get six cores of their own, disjoint from the gateway's and from each other, so the thing generating load and recording the result can never steal cycles from the thing being measured.
Every gateway runs from its own as-shipped default config, pointed at the mock, and we publish that config verbatim alongside the numbers. The goal is a simple, checkable claim: install this gateway version, drop in this config, on this class of box, and you should see this shape of result. We treat "here is the config that produced this number" as part of the result, not a footnote, and anyone can file a correction if we got a gateway's out-of-the-box setup wrong.
That config does not get quietly retuned partway through a run, and it does not get retuned loudly either. Every upstream a gateway will be asked to reach is wired into that one config up front, and the rendered bytes are identical for every column of the matrix below. What changes between columns is not the gateway, it is the request: the client asks for a different model name, or sends a different provider header, exactly as a real user would to pick a provider. Some gateways are restarted between columns so each column begins on a fresh process, but they are restarted onto the very same config.
This is a deliberately harder standard than it sounds. It means a gateway has to serve every dialect we probe from a single deployment, rather than being reconfigured into a specialist for each cell. Where a gateway genuinely cannot do that, we do not quietly rewrite its config to help it: the cells it cannot reach from one config are simply not claimed, and you will see them as not offered rather than as a failure.
The mock and load generator are themselves prebuilt binaries pulled from a release tag that CI rebuilds whenever either changes, so two runs of an otherwise byte-identical harness can legitimately disagree because the instrument moved, not because a gateway did. Every run stamps the SHA-256 of the exact mock and loadgen binaries it executed into its own result file, and the short mock digest sits in the footer of this page. If two runs are ever compared, whether the measuring instrument changed between them is a fact you can read off the page, not a question you have to go dig for.
The one run: the 6×6 protocol matrix
A capability probe on every cell, then a per-cell latency, throughput and streaming sweep on every cell that passes. This is the single producer every tab reads.
matrix/run.sh
Probe, then sweep, every cell
- Probe first. Every gateway is attempted against all 36 pairings of six ingress dialects (OpenAI, OpenAI Responses, Anthropic, Gemini, Cohere, Bedrock Converse) and the same six upstream dialects, one correctness-checked round trip per cell. The response has to be shaped like the ingress protocol says it should be, and, separately, the mock has to confirm it actually received a request on the upstream dialect the gateway was supposed to speak. A gateway that answers 200 from its own logic without ever calling the right upstream fails that check, evidence and all; a failed probe reads not configured, never a red.
- Then sweep. Every cell that passes gets the same perf sweep on that exact path: added latency at p50 and p99 at concurrency 1, sustained RPS at a 20 ms latency ceiling, and the maximum proxy RPS the gateway will hold. Each ceiling comes from a bidirectional peak search followed by a unimodal refinement, because a fixed concurrency grid can walk straight past the real peak between two rungs; the winning concurrency is recorded alongside the number. At a 20 ms ceiling, Little's law bounds throughput to concurrency divided by 0.020 s, so that particular peak sits at much higher concurrency than the max-proxy search, which is the real in-flight capacity limit rather than a latency-bounded one.
- Streaming on the same cell. The paced server-sent-events measurement runs on the already-swept cell, so the streaming numbers come from the same run as the latency and throughput, never a separate pass that could disagree with it.
- Fairness. Containerised gateways get their open-file limit raised to 1,048,576, matching native gateways and the load generator, so a fast container cannot collapse on the kernel's roughly-1024-socket default in the middle of a sweep. Applied the same way to every gateway.
Every view is a pick of that one run
- Peak shows each gateway on its own best same-dialect diagonal: best-of, pure forwarding, no translation. The Tested-on pill names the dialect (OpenAI when served, otherwise the gateway's fastest native one), so every gateway appears on its strongest path and that path is never hidden.
- Same fixes one dialect's diagonal for every gateway at once, an identical-path apples-to-apples comparison. A gateway that does not serve that dialect reads n/a rather than being silently dropped from the row.
- Custom lets you pick any ingress-to-egress cell for every gateway, including a real translation cell where the client speaks one dialect and the upstream speaks another. Two dropdowns pin the in and out; picking the same dialect on both is that dialect's strict passthrough.
- Streaming reuses the same three picks for the SSE columns: added time-to-first-token at p50 and p99, added inter-token gap at p50 and p99, sustained concurrent streams, and CPU-bound frames per second.
- One value, everywhere. The table, the matrix hover popup, the drawer and the charts all read the same per-cell record through the same accessors, so a number shows identically on every surface (a build-time guard fails the deploy if any two of them ever disagree). The Protocol matrix tab is simply the visual face of Custom: hover a cell for its gated numbers and its delta against the gateway's own peak, click it to open that exact cell in Custom.
matrix/run.sh (memory)
Memory: a fixed load on each gateway's peak cell
- Memory is a dedicated window that runs after the full 6×6, per gateway. The harness first finds each gateway's peak cell, its served cell with the highest max-proxy throughput (there is no single universal cell, since some gateways do not serve OpenAI-to-OpenAI), and then runs an identical fixed load on it. Every gateway gets the same load recipe, concurrency times payload bytes times duration, so the memory comparison is apples-to-apples and only the cell itself differs, shown under Tested on.
- Fresh cold restart. The gateway is killed and cold-restarted specifically for the peak cell's egress dialect, on a brand-new process, so idle measures a true cold-start baseline rather than a warm process that just finished the 6×6, which would measure recovery, not idle.
- Idle. Resident memory is sampled for 60 seconds with no traffic at all on that fresh process, and reported as the median of that window.
- Peak. The maximum resident memory sampled continuously while the fixed load runs, the real working-set peak under a fair, identical load rather than a synthetic burst.
- Recovered @60 s. Resident memory 60 seconds after the load stops, which is really asking whether the gateway gives back what it took or stays pinned near its peak. That question, more than the raw peak number, is the interesting part of the memory story: a gateway that peaks near a gigabyte and settles back to tens of megabytes behaves very differently from one that just holds onto it.
- Recovery curve. Memory is sampled at a steady cadence across the whole window, cold idle, then fixed load, then recovery, on one continuous process lifecycle, and drawn as a single idle-peak-recovery sparkline on the Memory tab. A gateway with no served cell reads n/a there, never a fabricated line. When RSS genuinely cannot be measured for a gateway, we publish that as null rather than as a zero, since zero would read as "uses no memory," which is never what actually happened.
Honesty rules on every surface
- Sole-source and null-safe. A value a record lacks renders n/a, never 0 and never a fabricated number. A measured 0 (served, but no tested load held p99 under 1 second at under 0.1% errors) is shown as 0 with a note, kept distinct from a value the rig simply could not certify.
- Rig-limited numbers are suppressed. A throughput or frames-per-second figure the harness could not certify as gateway-limited, because the sweep saturated near the mock's own ceiling instead, reads n/a and draws no bar. It never ranks and never wins a comparison.
- No gateway is graded red for a pairing it does not serve. Grey cells carry their own probe evidence: a boot or build failure reads "did not run," a measured streaming refusal reads "did not stream." Measured and shown, not hidden.
- Every number regenerates from committed JSON, and the per-gateway Download results link is the full 6×6 record the board is built from, not a summary of it.
Noisy neighbours: the same box does not always give the same answer
What a cloud instance actually is, and what we do about it before a run starts.
Here is the uncomfortable fact, stated plainly: the same gateway version, with the same config, on the same instance type, can still produce materially different numbers between two runs. An m7g.4xlarge is a slice of shared physical hardware, and a neighbour on that same host can steal CPU cycles that our cores-pinning has no way to see or stop. We measure resource isolation inside our own box; we cannot measure who else is on the machine underneath it.
So before any box is trusted with a multi-hour run, we measure its own floor. Before the gateway is even built, the load generator hits the mock directly, with no gateway anywhere in the path, and records direct_c1_p99_us, the p99 latency of that direct hit. That number describes the box, not the gateway: it is pure loadgen-to-mock overhead on whatever hardware this run happened to land on.
That floor is then compared against a rolling baseline, the median of that same gateway and architecture's last several qualified runs, with a 4% band. A box that comes in more than 4% slower on this cheap, tens-of-seconds probe is terminated and replaced before the real measurement ever starts, rather than spending hours running the full matrix on contaminated hardware and publishing the result as a gateway regression.
The band is deliberately one-sided, and the reasoning behind that is the interesting part. Contention only ever adds latency: a noisy neighbour can slow a box down, but nothing about sharing a host can make it faster than its own clean hardware. So a box cannot randomly beat its own baseline. If a fresh floor comes in faster than history says it should, that is not a red flag, it is the box showing you its true, uncontended speed, and it means the earlier baseline run was probably the noisy one. Only the slow side of the band is treated as a fault, and a second, harder-to-fool check runs after the gateway boots: a replay of that gateway's own last peak cell, compared against its own recorded throughput, which catches problems the cheap floor probe alone can miss.
We should be honest about where that 4% figure came from: it is provisional, calibrated against variance we measured on a harness that had its own known bugs at the time, including a mock-relaunch race condition and a couple of frozen probe paths, all since fixed. Some real share of the spread we used to set that band was almost certainly harness noise rather than genuine box-to-box variance, and we intend to tighten it toward under 2.5% once we have repeat runs on a frozen codebase to calibrate against properly.
A concrete example makes the risk land better than the mechanism alone. In one field run, a single box measured a gateway's peak-cell throughput at 86% below that gateway's own recent baseline, a sevenfold miss, while the rig's own floor on that same box had drifted only about 5%, a small enough number to have looked like ordinary noise on its own. Nothing about the gateway had changed. That pairing, a huge downstream regression sitting on top of a barely-moved floor, is exactly the signature this gate now catches before the multi-hour run begins, instead of after we have already published it as someone else's regression.
Why we measure /v1
The unified OpenAI-compatible route every SDK points at, on the same terms for every gateway.
Why we measure /v1, not passthrough endpoints
- We measure the unified OpenAI-compatible route, the endpoint a stock SDK points at, for every gateway, on the same terms.
- Some gateways also expose a separate passthrough route: a provider-specific endpoint that forwards to one named upstream and skips the gateway's own protocol translation. It is faster, but it is a different thing than what most integrations actually use.
- It is the path people actually take. A passthrough route means changing your code and hard-coding which provider you call. The unified endpoint is the "point your SDK here and go" experience these gateways are built to sell.
- Not every gateway has one. A passthrough column would pit a special route on some gateways against the normal route on others, which is not a fair comparison.
- It is pinned to one provider. The provider is part of the URL path, so calls on it bypass the cross-provider routing and failover the gateway offers on its unified endpoint. Measuring it would rank a single-provider forward against the multi-provider route people actually deploy.
Charts
Generated by the repo's chart script from the committed result JSONs. Bars are coloured by implementation language, a neutral property, not by rank or brand, so the colour never favours any gateway.
