On the Bench
Reproducible, honest benchmarks for the AI tooling you actually run.
Every entrant is measured the same way, on neutral hardware: same box, same mock upstream, same load, same CPU pin.
Every number regenerates from committed JSON. Fully open source, re-runnable by anyone.
Built and operated by the Busbar team, who are also one of the entrants. Same harness for everyone. A failure is our bug until proven the gateway's.
Benchmark repository on GitHub · How we measure
Every gateway on the board, in alphabetical order. Language, the exact version measured (hover for the full build/image digest), stars (snapshot), and each project's own self-description. The other tabs measure how they perform.
Click a column to sort, a row for full detail. A muted cell was not measured. Every view is shareable via its URL.
Protocol support matrix (6x6)
- supported, passing
- supported, failing
- not declared
- not verified
- untestable (mock limit)
- unprobed (auth)
- same-dialect passthrough
The set of cells each gateway is expected to serve (green vs grey) was drafted by the Busbar team from each project's public docs; it stands in until maintainers confirm their own grid.
Maintainers are welcome to PR their real capability grid.
Protocol matrix results are coming soon: no results/matrix JSONs are committed yet.
Method
Who runs this. Built and operated by the Busbar team; busbar is one of the entrants. Same shared harness for every gateway, no per-gateway special-casing, every number from committed JSON, fully open source. Read the code and re-run it yourself. Gateway creators are welcome to PR and own their own gateway.sh.
Every gateway is measured the same way, on neutral hardware:
- One rig. One box, one instant Rust mock upstream, one Go load generator, fixed CPU pinning (gateway on one core set, mock and loadgen on the others).
- Added latency is gateway-minus-direct. The same load runs straight against the mock and that baseline is subtracted, so the figure is what the gateway adds, not the network or the mock.
- Throughput is each gateway's own qualifying ceiling, not a subtraction. A guardrail flags any run approaching the mock's own ceiling, so the mock is never silently the bottleneck.
- Every result is stamped with hardware, architecture, build version and timestamp.
- A gateway is its own directory. A one-file manifest declares how to build, launch and probe it; add or remove a directory and it appears or disappears everywhere.
perf/run.sh Latency and throughput
- Added latency (p50/p99, microseconds): at concurrency 1, gateway p99 minus direct-to-mock p99, small payloads.
- Max proxy RPS: highest sustained requests per second against an instant mock where p99 stays under 1000 ms and the error rate stays under 0.1 percent (CPU-bound ceiling).
- Sustained RPS at 20 ms: the same gates, but the mock sleeps 20 ms per request to model real LLM latency, so the ceiling is concurrent-in-flight capacity, the real production bottleneck. Same concurrency grid for every gateway.
- The Passthrough tab shows these metrics on each gateway's best same-dialect passthrough (pure forwarding, no translation); the Tested-on pill names the dialect (OpenAI when served, else the gateway's fastest native one). Best-of, so every gateway appears on its strongest passthrough and none is filtered out. For a strict single-dialect comparison, pin the same dialect on both sides in the Translation tab.
- Idle RSS: process resident set after launch, before load.
- Peak RSS: maximum resident set while sustaining large-payload load at high concurrency.
- The mock answers stream:true with a paced SSE stream; the suite measures what the gateway adds on top of that pace.
- The mock's first token is instant (it then paces later tokens), so the first-token figure deliberately includes the gateway's first-write network behaviour, not just its compute. Against a real model, whose own first token takes hundreds of milliseconds, that component is largely hidden; here it is exposed on purpose. A benchmark whose mock waits before its first token, or that times the first SSE event of any kind rather than the first content token, will report a near-zero first-token overhead by construction.
- Added wait for the first token (p99), also called TTFT (time to first token): gateway first-content-frame time minus direct-to-mock, at concurrency 1.
- Added gap between tokens (p99): gateway content-frame gap minus direct-to-mock gap.
- Streams sustained: max concurrent streams where at least 99.9 percent of expected frames deliver, no stream stalls beyond twice the pacing interval, and the stream error rate stays under 0.1 percent.
- A gateway that answers 200 but buffers the whole response is recorded stream_served=false: measured, not hidden.
- The client speaks Anthropic (POST /v1/messages) while the upstream mock speaks OpenAI, so the gateway must translate both directions. The mock is untouched; that is the point.
- Added latency (p99) and sustained RPS at 20 ms on the translation path, same gates as perf. The direct baseline is the OpenAI shape straight to the mock, so the added-latency figure deliberately includes the translation work.
- The Translation tab pins both ends with a pair of dropdowns (client dialect and upstream dialect), so every row is the identical path and the ranking is apples-to-apples; only gateways that serve the chosen pair appear. Pick the same dialect on both sides and it becomes that dialect's strict passthrough.
- A gateway that cannot serve Anthropic ingress against an OpenAI upstream records xlate_served=false with the probe status and body snippet as evidence.
- Headline translation numbers (table, drawer, compare, charts) all come from one canonical record per gateway: the matrix per-cell sweep's OpenAI-in cell. This suite's Anthropic-to-OpenAI numbers are the fallback for a gateway with no measured matrix translation cell, and every surface labels the direction it shows.
- A capability suite first: one probe per (ingress, upstream) cell across the full 6x6 (OpenAI, OpenAI Responses, Anthropic, Gemini, Cohere, Bedrock Converse), relaunching the gateway per upstream dialect.
- Envelope validation per cell, a passthrough guard that rejects the mock's canned bodies as untranslated proxying, an upstream round-trip check (the mock records which endpoint the gateway actually called), and an unprobed_auth state for ingresses that demand signatures the harness does not forge.
- Per-cell perf: every passing (green) cell also gets the perf suite's exact sweep on that path (added latency at concurrency 1, sustained RPS at 20 ms, max proxy RPS). The Passthrough and Translation tabs read specific cells from this sweep; hovering any green cell shows that path's numbers and its RPS delta vs the gateway's reference cell (named, not called "best").
Charts
Generated by the repo's chart script from the committed result JSONs. Bars are coloured by implementation language (a neutral property), not by rank or brand, so the colour never favours any gateway.