How we measure, and why we think this is the fair way to do it
Who runs this. Built and operated by the Busbar team; busbar is one of the entrants. Same shared harness for every gateway, no per-gateway special-casing, every number from committed JSON, fully open source. Read the code and re-run it yourself. Gateway creators are welcome to PR and own their own gateway.sh.
A benchmark is only worth reading if you can see what it did. Every gateway here gets its own machine, its own dedicated cores, the same local upstream, the same prebuilt measuring instruments and a configuration of the sort a real operator would actually deploy. What follows is how that works, why each choice was made, and where the whole thing can still be wrong.
One gateway, one machine
Each gateway is measured alone. It gets a fresh AWS m7g.4xlarge, sixteen real Graviton3 cores and 64 GB of memory on Ubuntu 24.04, launched for that gateway and terminated when its run finishes. Nothing another gateway built, pulled, cached or left behind in page cache is ever on the box. The field can also be run on an Intel instance of the same size, and every result records which architecture produced it, because a number from one architecture is not a number from the other.
Running the field on one shared machine would be cheaper and faster. It would also be worthless. A field of gateways taking turns on the same host shares a page cache, a disk, a network interface and a thermal budget, and the gateway that happened to run third would carry the residue of the two before it. A dedicated box per gateway is the cost of the comparison meaning anything.
Three groups of cores, which is the whole basis of the comparison
Inside that box the sixteen cores are split three ways and pinned. The gateway under test gets four cores and only those four. The load generator gets six of its own. The mock upstream gets the remaining six. The three sets are disjoint, so the process generating the load and the process answering it can never take a cycle from the process being measured.
This split is not a detail, it is the thing that makes any of the numbers comparable at all. A load generator saturating the same cores as the gateway does not measure the gateway, it measures the fight between them, and the winner of that fight changes with how efficiently each side happens to use a core. Pin them apart and the gateway is running on a known, fixed amount of hardware: four cores, the same four, for every entrant. Four cores is also a real deployment size rather than an artificial one, and it is small enough that a gateway cannot hide inefficiency behind spare capacity.
The corollary is that these numbers describe four cores. A gateway that scales well to thirty two may look different there. We publish the size we measured rather than implying it generalises.
The upstream is a mock, on purpose
Nothing in this benchmark talks to a real model provider. The upstream is a local mock running on the same box, on its own cores, a few microseconds away. That is a deliberate restriction of what we are willing to claim to measure.
Point a gateway at a real provider and most of the resulting latency belongs to somebody else: the region you landed in, the queue in front of the model, the rate limiter, the weather on the public internet that afternoon. Two gateways measured an hour apart would be measured against two different upstreams, and the ranking would move for reasons no gateway author can control or fix. With a local mock, the upstream is close to free and effectively constant, so what is left in the measurement is the gateway: its parsing, its translation, its connection handling, its scheduling.
The mock does more than answer quickly. It records what it received, so a gateway that returns a plausible 200 without ever calling the upstream in the dialect it was supposed to speak fails the correctness check rather than scoring a fast time. The mock also has a ceiling of its own, and when a throughput sweep saturates near that ceiling instead of the gateway's, the number is not published: it reads n/a and draws no bar, because it would be a measurement of our instrument rather than of the entrant.
What this cannot tell you is how a gateway behaves against a slow, flaky, remote provider, where retries, timeouts and connection reuse start to dominate. That is a real and interesting question. It is not this benchmark.
Real configurations, not benchmark configurations
Every gateway runs in a normal, sensible, documented configuration of the kind an operator would actually deploy. We start from the project's own published artifact, preferring its official image at a pinned version, then a binary it publishes, and only building from a pinned commit when the project ships neither. On top of that, the configuration is the bare minimum required to run.
That rule has a precise meaning here, and a lint enforces it. A setting may appear in a gateway's config only if it is needed to boot the process at all, to point an upstream at the test mock instead of a real provider, to expose an ingress path the matrix exercises, or to bind the port and cores the rig requires. Every setting has to name which of those four necessities justifies it, in the manifest itself, and a setting with no claim fails the build. A claim that no longer matches any setting fails too, so the list cannot rot into a record of things we used to do.
We do not turn features on and we do not turn them off. We do not even restate a default, because a line that merely repeats today's default is still a line we wrote, and it silently becomes an override the day the project changes that default. Four of our manifests once carried settings that suppressed outbound telemetry. They are gone. A gateway that reports usage by default is now measured reporting usage by default.
The consequence is that some numbers here are lower than the figures a project publishes itself, and we think that is the honest outcome rather than an embarrassing one. Vendor benchmarks are usually run in a tuned configuration by the people who know exactly which knobs to turn. We are not in a position to tune every entrant to its authors' standard, and if we tried, the board would stop being a comparison of gateways and start being a comparison of how much time we spent on each one.
This has been tested. Some gateway contributors asked us to optimise their configuration by turning off features that would have improved their numbers but would have left a less realistic running gateway. We declined. Turning a feature off because it costs performance is exactly the tuning we are trying not to do, and the configuration we run is the one a real operator would deploy, not the one that wins the benchmark. The offer stands in the other direction: if the configuration we published is not what a competent operator would actually run, that is a bug in our setup and we will fix it. Every gateway's config is published verbatim next to its numbers, with a one-click link to propose a correction, and anyone can file one.
Running defaults finds things tuning would hide. One entrant's default install ships metrics collection enabled with nothing draining it, so its resident memory grows with every request served. That is a real property of the software as shipped, and it only shows up because nobody went in and switched it off first. Softening a measurement because of what it revealed would be backwards, which is why the rule is symmetric: nothing gets turned on either.
The same configuration serves the whole run. It is not quietly retuned between one part of the matrix and another. Every upstream a gateway will be asked to reach is wired into that one config up front, and the rendered bytes are identical for every column. What changes between columns is the request, not the gateway: the client asks for a different model or sends a different provider header, exactly as a user would. That is a harder standard than it sounds, because a gateway has to serve every dialect we probe from a single deployment rather than being reconfigured into a specialist for each cell. Where a gateway genuinely cannot do that, we do not rewrite its config to help it. The cells it cannot reach are simply not claimed.
The same instrument, every time
The measuring apparatus is two programs: the engine, which drives a run and generates the load itself as one binary, and the mock upstream. Both are prebuilt binaries fetched from a pinned release rather than compiled on the box, so every machine in the field runs byte-identical instruments and no box's compiler, toolchain version or build flags leak into anyone's numbers. If that download fails, the run fails. It does not fall back to building from the local tree, because a locally built instrument would produce numbers that look like they came from the pinned one.
Instruments still move between runs. The release tag is rebuilt whenever the mock or the load generator changes, and that has already caused real trouble: a mid-week tightening of the mock's request checking changed cell verdicts across the entire field, and working out that the instrument had moved rather than the gateways took hours. So every run now records the SHA-256 of the exact mock and load generator binaries it executed, and the short mock digest sits in the footer of this page. Every run also records the commit of the harness that produced it, and marks itself as not reproducible if that working tree was dirty. Whether two results are even comparable is now a fact you can read off the page instead of an investigation.
One run, and every view is a slice of it
There is a single producer behind the whole board. Each gateway is attempted on all thirty six pairings of six ingress dialects and the same six upstream dialects, one correctness-checked round trip per cell. The response has to be shaped the way the ingress protocol says it should be, and separately the mock has to confirm it received a request in the upstream dialect the gateway was supposed to speak. Only cells that pass that probe are then timed, so no performance number exists for a path we have not first proven works.
Each passing cell gets the same sweep on that exact path: added latency at p50 and p99 at concurrency one, sustained requests per second under a latency ceiling, and the maximum throughput the gateway will hold. The peak comes from a bidirectional search followed by a refinement rather than a fixed concurrency grid, because a grid walks straight past the real peak between two rungs, and the concurrency that won is recorded next to the number. The streaming measurement runs on the same already-swept cell, so the streaming and latency figures always come from one run and cannot disagree.
Memory is measured on every cell a gateway serves and never averaged across cells. It used to be one number per gateway, which forced the harness to pick a cell to produce it, and it picked each gateway's fastest one. That selected on throughput and reported memory, two quantities with nothing to do with each other, and it meant a gateway measured on a pure passthrough was compared against one paying for a full translation on every request. Memory is the one measurement that gets a freshly restarted process, because an idle reading taken after a throughput sweep is not an idle reading at all: it is whatever the sweep left resident. Throughput and streaming are deliberately not restarted, so no published rps number is measured on a cold start.
Idle is sampled at rest, before a single request is served. Then load runs at a fixed concurrency in repeated windows until the trailing minute of resident memory is flat, rather than for a fixed duration, because a fixed duration reports where memory was when we stopped watching rather than where the gateway settles. A gateway that never flattens inside the cap is published as exactly that, with the rate it was still climbing at, which is a more useful finding than any peak. "We could not tell" is kept as a third, separate answer from "it did not settle", because they are different claims.
Then the load stops and we keep watching for another minute. That last minute is there to answer a question a peak cannot: whether the gateway hands the memory back. Climbing to a high peak and returning to idle is a different proposition from climbing to the same peak and staying there, and both curves are published in full, one reading every tenth of a second, so you can see the shape rather than take our word for the summary.
The tabs are then choices of which cell to show, not separate experiments. The table, the matrix, the drawer and the charts all read the same per-cell record through the same accessors, and a build-time guard fails the deploy if any two of those surfaces ever disagree about a number.
What a run is allowed to publish
A run that measures nothing publishes nothing. If a run produced no fresh result anywhere, nothing derived from it is pushed, which sounds obvious until you have watched a broken harness finish in ninety seconds, report success and regenerate the whole board from the previous run's data. That happened here, and the rule exists because of it.
Below that sits a promote guard on every individual result. A gateway that failed to build or never came up at all is our environment failing, not a measured property of the gateway, and such a non-result is refused rather than allowed to overwrite a good one. A gateway that did boot and then honestly refused a request is a real observation and is published as one, because quietly republishing last week's better number in its place would bias the board.
On the surface, a value we do not have renders n/a, never a zero and never a filled-in guess. A measured zero is shown as a zero with a note, kept distinct from a value the rig could not certify. A cell a gateway does not serve is grey and carries the probe's own evidence, never a red mark, because failing to offer a protocol is not the same as failing at it. Every number on the board regenerates from committed JSON, and the per-gateway download is the complete record rather than a summary of it.
Where this can still be wrong
The uncomfortable part first: the same gateway, same version, same config, same instance type can produce materially different numbers on two different boxes. A cloud instance is a slice of shared physical hardware, and a neighbour on that host can take cycles our core pinning cannot see or prevent. We measure isolation inside our own box. We cannot measure who else is on the machine underneath it.
So every box measures its own floor before it is trusted. Before the gateway is even built, the load generator hits the mock directly with no gateway in the path and records the p99 of that direct hit. That number describes the hardware, not the entrant. It is compared against a rolling baseline built from that same gateway's recent qualified runs, with a band of four percent, and after the gateway boots a second and stronger check replays that gateway's own last peak cell against its recorded throughput. Both verdicts are written into the result file, so the state of the box that produced a number travels with the number.
The band is deliberately one-sided, and the reasoning is the interesting part. Contention only ever adds latency and removes throughput. Nothing about sharing a host makes a box faster than its own clean hardware, so a box cannot randomly beat its own baseline. A floor that comes in faster than history predicts is not a fault, it is the box showing its true uncontended speed, and it suggests the earlier baseline run was the noisy one. Only the slow side is treated as a problem.
We should also be honest about where four percent came from. It is provisional. It was calibrated against variance measured on a harness that had its own bugs at the time, including a mock relaunch race and a couple of frozen probe paths, all since fixed, so some real share of that spread was our own noise rather than genuine box-to-box variance. We expect to tighten it once we have repeat runs on a frozen codebase to calibrate against properly.
A concrete case makes the risk clearer than the mechanism. One box measured a gateway's peak throughput at eighty six percent below that gateway's own recent baseline, a sevenfold miss, while the rig's floor on that same box had drifted only about five percent, small enough to pass for ordinary noise on its own. Nothing about the gateway had changed. That pairing, a large collapse sitting on top of a barely moved floor, is the signature this check exists to catch, and it is why the box's own qualifying measurement is recorded rather than discarded.
Beyond the hardware, the honest limits are these. The upstream is a mock, so nothing here describes behaviour against a slow or failing real provider. The gateway gets four cores, so nothing here describes scaling on a much larger machine. The configuration is a competent default rather than an expert tuning, so a determined operator can very likely beat these numbers. And we build one of the entrants, which is why the harness carries no per-gateway special cases, why the code is public, and why the results are re-runnable by anyone who disagrees with them.
Why we measure the unified route
Every gateway is measured on its unified OpenAI-compatible route, the endpoint a stock SDK points at, on the same terms as every other gateway. Some gateways also expose a passthrough route: a provider-specific endpoint that forwards to one named upstream and skips the gateway's own translation. It is faster, and it is a different product.
We do not measure it, for three reasons. It is not the path most integrations take, since using it means changing your code and hard-coding which provider you call, whereas the unified endpoint is the point-your-SDK-here experience these gateways are built to sell. Not every gateway has one, so a passthrough column would pit a special route on some entrants against the normal route on others. And it is pinned to a single provider by construction, so calls on it bypass the cross-provider routing and failover that are the reason to deploy a gateway in the first place. The code is public if you want to check any of this: a run is driven by run-on-ec2.sh, and every measurement, the memory window included, is taken by the engine.