Click a column to sort, a row for full detail. A muted cell was not measured; a labeled cell was probed and the gateway does not serve it. Every view is shareable via its URL.
Protocol support matrix (6x6)
- supported, passing
- supported, failing
- not supported
- not verified
- same-dialect passthrough
Support is each gateway's own declared capability. A red cell is a gateway that served and returned a wrong or untranslated body; "not verified" means the harness could not get that gateway serving under this upstream config, so no pass or fail is claimed.
Protocol matrix results are coming soon: no results/matrix JSONs are committed yet.
Charts
Generated by the repo's chart script from the committed result JSONs. The highlight goes to whichever gateway measures best on each metric, in a neutral colour.
Method
Every gateway is measured the same way: one box, one instant Rust mock upstream, one Go load generator, fixed CPU pinning (gateway on one core set, mock and loadgen on the others). A guardrail flags any measurement where the gateway approaches the mock's own ceiling, so the mock is not silently the bottleneck. The added-latency figures are gateway-minus-direct: the same load is run straight against the mock and the direct baseline subtracted, so the figure is what the gateway adds, not what the network or the mock costs; the throughput figures are each gateway's own qualifying ceiling, not a subtraction. Hardware, architecture, build version and timestamp are stamped into every result JSON. A gateway is fully defined by its own directory: a one-file manifest declares how to build, launch and probe it, so adding or removing a directory adds or removes it everywhere.
perf/run.sh Latency and throughput
- Added latency (p50/p99, microseconds): at concurrency 1, gateway p99 minus direct-to-mock p99, small payloads.
- Max proxy RPS: highest sustained requests per second against an instant mock where p99 stays under 1000 ms and the error rate stays under 0.1 percent (CPU-bound ceiling).
- Sustained RPS at 20 ms: the same gates, but the mock sleeps 20 ms per request to model real LLM latency, so the ceiling is concurrent-in-flight capacity, the real production bottleneck. Same concurrency grid for every gateway.
- Idle RSS: process resident set after launch, before load.
- Peak RSS: maximum resident set while sustaining large-payload load at high concurrency.
- The mock answers stream:true with a paced SSE stream; the suite measures what the gateway adds on top of that pace.
- Added TTFT (p99): gateway first-content-frame time minus direct-to-mock TTFT at concurrency 1.
- Added per-token latency (p99): gateway content-frame gap minus direct-to-mock gap.
- Streams sustained: max concurrent streams where at least 99.9 percent of expected frames deliver, no stream stalls beyond twice the pacing interval, and the stream error rate stays under 0.1 percent.
- A gateway that answers 200 but buffers the whole response is recorded stream_served=false: measured, not hidden.
- The client speaks Anthropic (POST /v1/messages) while the upstream mock speaks OpenAI, so the gateway must translate both directions. The mock is untouched; that is the point.
- Added latency (p99) and sustained RPS at 20 ms on the translation path, same gates as perf. The direct baseline is the OpenAI shape straight to the mock, so the added-latency figure deliberately includes the translation work.
- A gateway that cannot serve Anthropic ingress against an OpenAI upstream records xlate_served=false with the probe status and body snippet as evidence.
- A capability suite, not a latency suite: one probe per ingress protocol cell (OpenAI, OpenAI Responses, Anthropic, Gemini, Cohere, Bedrock Converse) against the fixed OpenAI upstream.
- Envelope validation per cell, a passthrough guard that rejects the mock's canned bodies as untranslated proxying, and an unprobed_auth state for ingresses that demand signatures the harness does not forge.