We beat every published Qwen3-TTS benchmark. Here's exactly how we measured it.

By 40 requests a second on one H100, every other Qwen3-TTS engine we tested had broken, failing more than 5% of its requests. Ours was still serving first audio in 33.7 ms at p95, and across every load it failed 0 of 41,337 requests. We ran SGLang-Omni, vLLM-Omni and Nari Labs’ open-source engine through the same benchmark client, with no code changes, from 1 to 40 requests a second. At 20 requests a second ours was 1.9x faster than Nari Labs, 4x faster than SGLang-Omni and 10x faster than tuned vLLM-Omni. Over HTTP with the full text sent up front, none of the published Qwen3-TTS results we found comes in under it.

33.7 ms at 40 requests a second. Zero failures.

Pipeshift’s p95 rises 9 ms from 1 to 40 requests a second. The other engines climb with load, then fail.

Our p95 time to first audio went from 24.6 ms at 1 request a second to 33.7 ms at 40. That’s about 9 ms of extra wait for 40 times the traffic. The median moved even less, from 12.7 ms to 20.2 ms.

At 1 request a second, every other engine started somewhere between 28 ms and 234 ms, then climbed. By 30 requests a second, two of the four were failing more than 5% of their requests. The other two followed at 40.

The only latency your caller hears

In a voice agent, the caller notices the gap between finishing a sentence and hearing the agent start to speak. Text to speech sits right in that gap, so the number that matters is how long it takes for the first sound to come out.

So we time the first audible sound, not the first byte, using the benchmark client’s own onset detector. A server can send bytes that are pure silence, and the caller hears nothing until real speech arrives. More on that trap further down.

We report p95 with the median next to it. If one caller in twenty waits much longer, the product feels broken, however good the average looks.

And failures count. A request that times out, errors, stalls mid-stream or never produces speech is treated as infinitely slow. If more than 5% of requests fail at a load, we don’t report a p95 for it at all and call the load failed.

The engines, as shipped

We compared our engine against the main open-source serving engines for Qwen3-TTS, the text to speech model from Alibaba’s Qwen team (the 1.7B CustomVoice model):

• SGLang-Omni, a build newer than its v0.1.8 release, with its default settings. We also tested the v0.1.8 release with its defaults. The newer build was faster at every load where both served, so it’s the one we show. The release failed the load at 30 requests a second (25.6% of requests) and at 40 (45.5%, against 49.3% for the newer build). Its default settings include SGLang-Omni’s own trim of the silence at the start of the audio, which stays on. That trim only applies while 24 or fewer requests are streaming, so it stops working under heavier load.

• vLLM-Omni 0.30.0, twice. Stock means its built-in deploy config for Qwen3-TTS. Tuned means we set codec_chunk_frames to 32 and codec_chunk_ramp to [3, 3, 16, 32] in that same config. That was the fastest of three settings we tried.

• Nari Labs’ open-source engine, run by us from their published container image, using its two built-in profiles (ttfa and balanced) with everything else at its defaults. At each load we show whichever profile did better. That choice can only help Nari Labs. We included it because, before this post, Nari Labs’ own benchmarks were the fastest published numbers for serving Qwen3-TTS. Their GitHub repo (nari-labs/nari-qwen3-tts) calls it “Ultrafast Qwen3-TTS”, with sub-50 ms p95 time to first audio at 10 requests a second. It’s also open source, so we could run it ourselves.

• Pipeshift, our own proprietary serving engine, built in-house for real-time voice.

No engine had its code changed. Each one got a fresh server for every run.

How it was done

What we time: from sending the request to the first sound a caller can hear.

Client and prompts. Every engine was called over HTTP, with the full text sent up front in one request and audio streamed back. For every engine, ours included, we used Nari Labs’ open benchmark client, tts-bench, unmodified. Its prompt list is the English test set from ByteDance’s Seed-TTS, 1,088 sentences, spoken here in one preset voice. At a given seed the client sends every engine the same sequence of prompts on the same arrival schedule. Arrivals are Poisson, so requests bunch up like real traffic. The timeout is 120 seconds.

Load and runs. We tested 1, 4, 8, 12, 20, 30 and 40 requests a second. Each run had 30 seconds of warmup, then at least 120 seconds measured. Every engine ran 3 times at each load, with seeds 0, 1 and 2. That’s 126 runs, the SGLang-Omni release included.

First audio. The client’s onset detector slices the audio into 20 ms frames with a 10 ms hop. It marks the first point where the loudness (DC-removed RMS) stays at or above -45 dBFS for 2 frames in a row. Time to first audio runs from sending the request to that point. The same detector scores every engine, ours included.

Failures. A request fails if it times out, returns an error, stalls mid-stream (the client’s stall check) or never produces audible speech. A failed request counts as infinitely slow. If more than 5% of a load’s requests fail, that load is marked failed, with no p95. The onset detector, the stall check and the playback-gap check are the client’s own. The 5% rule, the 1% failure limit for capacity, nearest-rank percentiles and the median of 3 seeds are ours, set before the measured runs.

Statistics

• Percentiles are nearest-rank. The p95 (or p50) for a load is the median of the 3 seeds’ values.

• Confidence intervals come from a hierarchical bootstrap, 10,000 resamples: lanes first, then requests within each lane, with requests paired across engines by their position in the schedule.

• We only call a difference a win when the 95% interval and all 3 seeds agree on the direction.

Keeping it fair. The machine is an 8-GPU H100 SXM server. Every run had an H100 to itself, plus its own block of 12 to 32 CPU cores for that run’s server and client. On each GPU the runs went one at a time: the previous engine’s server was shut down and the GPU checked before the next one started. The order of the engines was rotated at each load, so none of them got a better slot. Before each run, a gate checked that no other process was busy on that run’s CPU cores. After it, a run would have been thrown out and re-run if the client itself fell behind, meaning its p99 dispatch lag was over 50 ms. None did. The worst in any run was 9.8 ms, so a slow client never made an engine look slow. All but one run passed on the first try. The one rerun was because another job was loading the same machine while it ran.

Accuracy scoring. Word error rate uses ByteDance’s official Seed-TTS-eval code, unmodified. Its transcriber is OpenAI’s Whisper large-v3 with greedy decoding and English forced. Text is normalized the official way, which strips punctuation except apostrophes and lowercases. The score is the mean of per-sentence word error rates, with no cap. As a check, scoring Seed-TTS’s own ground-truth recordings through the same setup reproduces their published 2.143% exactly. There are two setups:

• Voice cloning. Each engine that supports cloning cloned all 1,088 sentences from their own reference clips with the Qwen3-TTS 1.7B Base model, four requests at a time. Nari Labs’ open-source image serves only the CustomVoice model and rejects reference-audio requests.

• Preset voice. We scored the audio each engine served during the speed runs at 1, 20 and 40 requests a second. For Nari Labs it came from runs of its balanced profile. Requests that returned no complete audio are left out. Stalled requests that still delivered complete audio are scored. A load where more than 5% of requests returned no complete audio isn’t scored. At a load that failed the speed test (more than 5% failures, stalls included), we don’t compare word error rates.

Intervals come from a bootstrap with 10,000 resamples. For voice cloning it resamples reference clips (666 of them), because several sentences share a clip. For the preset voice it resamples sentences.

At 20 requests a second, ours is at least 1.9x faster than every other engine

Ours is the fastest at every load, at both the median and p95.

At this load, the audio each vLLM-Omni setup sends starts with about 90 ms of silence (p95), and SGLang-Omni’s with about 80 ms. That silence is inside the audio stream, and the caller hears it as waiting (see the traps section below).

The median tells the same story as p95: Pipeshift stays lowest at every load.

The full numbers, for anyone who wants to check them. Each cell is p95 time to first audio, with the median in brackets, in milliseconds, and each is the median of the 3 seeds’ values.

Failed means more than 5% of requests failed at that load, so there is no p95 to report. Nari Labs: our run of their open-source engine, best of their two profiles at each load (see The engines, as shipped). Its 40 req/s cell is from its balanced profile, where 26% of all requests got HTTP 429 errors and the other failures were stalls.

Past 20 requests a second, everyone else hits a wall

Ours kept going past 20 requests a second: 30.7 ms p95 at 30 and 33.7 ms at 40, with zero failures. 40 is where our test range ends, not where ours broke. Across all 41,337 requests at every load ours failed none, and just one, at 30 requests a second, had a playback gap, and it lasted 3 ms. The other engines still served their traffic up to 20, just slower and slower. Past that they went quiet for a long time before the first sound, up to several seconds, and then started failing.

Share of requests that failed at each load. Ours stays at 0 the whole way.

• SGLang-Omni reached 925.5 ms p95 at 30 requests a second, then failed 49% of requests at 40.

• Tuned vLLM-Omni still answered every request at 30, but its p95 was 4,394 ms, more than 4 seconds before the first sound. At 40 it failed 14% of requests.

• Stock vLLM-Omni already failed 1.7% at 20, then almost everything (99.6%) at 30.

• Nari Labs failed 46% of requests at 30 with its better profile, and 100% with the other. At 40 requests a second, with its balanced profile, 88% of its requests failed by this post’s rule. 26% of all requests got HTTP 429 errors, and the other failures were stalls.

Another way to read the same data is to ask how much load each engine can carry while keeping p95 under a target.

Highest load held with p95 under 200 ms and under 100 ms, and under 1% failures, in at least 2 of the 3 seeds at that load and every lower one.

Under 200 ms, ours held at least 40 requests a second, the top of our test range. Nari Labs and SGLang-Omni held 20, tuned vLLM-Omni 8, and stock vLLM-Omni didn’t get under 200 ms even at 1 request a second. Under 100 ms, ours still held 40, Nari Labs 20, SGLang-Omni 12, and neither vLLM-Omni setup got there at any load. Ours held 40 at both targets, and no other engine held 30 at either.

A few more numbers from the same runs, at the loads where things change. End to end is the time until the last of the audio arrived. Requests in flight is the most that were open at once, as the client counts it over the whole run including warmup, which grows when an engine falls behind. It’s shown even where a load failed, because it shows how far the engine fell behind. A playback gap means the audio stream ran dry for a moment after it started. The client counts it separately from a stall: a stall fails the request, a playback gap doesn’t. Each value is the median of the 3 seeds, except the playback gap share, which pools all three. Nari Labs’ numbers at 20 and 30 come from its 300 second runs, so its peaks there had more time to build.

The playback gap counts come from the benchmark client, which flags a request when, after its first audio arrives, a later chunk comes in too late to keep playback going. Nari Labs’ 13.6% at 20 requests a second is 2,459 of 18,093 requests with its ttfa profile, counted the same way over its 300 second windows.

At 40 requests a second, the failures break down differently by engine. SGLang-Omni’s were mostly HTTP 503 errors (6,205) plus 934 stalls. Those 503s come from its default admission limits, 64 running and 64 queued requests. A larger setting (128 and 1,024) tied with the default in tuning and lost on the tie-break in our pre-registration, so it wasn’t run on the full ladder. It might fail fewer requests at 30 and 40. Both vLLM-Omni setups failed by stalling: 2,044 requests tuned and 14,342 stock.

Quality holds

We also scored word error rate with the official Seed-TTS-eval code, in two setups described under “How it was done”.

“Lower” means the 95% interval on the difference excludes zero, and “on par” means it doesn’t. The Qwen reference is the official qwen-tts package, run offline with default sampling, sdpa attention, one draw per line. The SGLang-Omni and vLLM-Omni preset-voice audio is from the speed runs, re-scored. Nari Labs’ is from its own runs at the same settings. The tuned vLLM-Omni config in the cloning test differs from the one in the speed results.

The scorer itself checks out. Scoring Seed-TTS’s own ground-truth recordings through the same setup gives their published 2.143% exactly, so these numbers are on the official scale.

For voice cloning, ours scored 1.44%. That’s lower than SGLang-Omni, stock vLLM-Omni and our own run of the Qwen reference, and on par with tuned vLLM-Omni. It’s consistent with the 1.24% in the Qwen3-TTS technical report, which sits inside our 95% interval of 1.19% to 1.71%. Nari Labs’ open-source image serves only the CustomVoice model and rejects reference-audio requests, so it isn’t in that test.

Under load, with the preset voice, ours served 1, 20 and 40 requests a second with zero failures and no detected change in WER. Ours scored 1.50% at 1 request a second and 2.67% at 20. On the same 335 sentences, 40 requests a second came out +0.29 points against 1, with an interval of -0.48 to +0.79. At 40 requests a second every other engine had failed the load by this post’s rule, so we don’t compare word error rates there. At 1 and 20 requests a second we saw no clear difference from vLLM-Omni or SGLang-Omni, and at 20 the SGLang-Omni and Nari Labs averages are slightly below ours. Against Nari Labs, ours minus theirs was -0.17 points at 1 request a second (-0.77 to +0.61) and +0.22 at 20 (-0.54 to +0.80), so no clear difference either way. At 20 requests a second a few clips with very high error rates move every engine’s average. These Nari Labs scores are for its balanced profile. Its ttfa profile, the one behind its speed numbers at 1 to 20 requests a second, was also on par with ours: ours minus theirs was -0.17 points at 1 request a second (-0.77 to +0.61) and -0.23 at 20 (-1.56 to +0.64), and its naturalness score was 4.13 at 1 request a second, the same as its balanced profile.

Naturalness, scored with UTMOS (a model that predicts listener ratings, not a human listening test) on the same schedule of 370 prompts served at 1 request a second, came out on par within noise. SGLang-Omni was highest at 4.14 and ours was 4.11. The difference, 0.03 in SGLang-Omni’s favor, has an interval of -0.02 to +0.08, so it isn’t clear. Nari Labs scored 4.13 with its balanced profile at the same load and settings. The difference from ours, 0.02 in Nari Labs’ favor, has an interval of -0.04 to +0.08, so there’s no clear difference there either. Tuned and stock vLLM-Omni scored 4.07 and 4.08.

Preset voice at 1 request a second, 370 clips per engine. The axis starts at 4.0.

Two traps that make TTS benchmarks lie

Both of these make an engine look better than it is. We avoided both, for every engine including ours.

1. First byte is not first sound. Some servers send silence before the speech starts. Here is first byte against first audible sound at 1 request a second, p95, for every engine shown. The same detector scores all of them. Ours and Nari Labs’ engine send almost no silence before the speech, 10 ms at p95, which is the detector’s step.

vLLM-Omni as shipped starts the audio it sends with about 90 ms of silence (p95) before the speech. Tuned vLLM-Omni’s first bytes arrived at 73 ms (p95, on an H100), close to the 70.6 ms mean (and median) to first packet that vLLM-Omni publishes for one request at a time on 2x H20. The first sound came much later. Stock vLLM-Omni’s first sound came later still. Every one of its requests at 1 request a second also had a gap in its audio stream, and we haven’t broken down the rest of the time between its first byte and its first sound. The two columns are separate percentiles, so the gap between them isn’t an exact measure of the silence.

Nari Labs’ published vLLM-Omni results (56.8 ms at 1 request a second) came from a build patched to trim that silence. Their report says so. That patch isn’t part of vLLM-Omni, and we ran every engine without code changes, so the silence counts here. A benchmark that stops the clock at the first byte would miss all of this, which is why we time the first audible frame. A request that never produces audible speech counts as a failure.

2. “x times faster” against an engine that already failed means nothing. Once an engine fails a large share of its requests, its latency on the ones that did succeed is no longer a fair number to divide by. Neither is a p95 computed after dropping the failures, which looks better the more an engine fails. We don’t quote a ratio at any load where the other engine failed. We mark it failed, and the failure rate is the result.

What’s been published

Ours is lower than every published Qwen3-TTS latency figure we found, at every load they report. The closest anyone gets is Nari Labs’ own published result for their engine at 1 request a second: 26.5 ms against our 24.6, a 1.9 ms margin. All three of our seeds were under 26.5 ms (24.3, 24.6 and 25.2), and Nari Labs’ engine run through our setup measured 28.4 ms at that load. A few of those loads (2, 6, 10 and 16 requests a second) aren’t on our ladder, but at the next higher load on our ladder ours was already below their figure.

Nari Labs’ reports used the same GPU model as us. The rest used other hardware or don’t say what they used, apart from faster-qwen3-tts, which also ran on an H100, one request at a time. The rows from Nari Labs’ reports use the same metric we do, p95 time to first audio, measured with the same tts-bench client. Several of them (SGLang-Omni, vLLM-Omni, VoxServe and M*) used builds with onset trimming or other patches, as those reports say, so they’re context rather than a controlled head-to-head. The other rows measure something easier: a mean or median, the first packet, or one request at a time.

The full list, with the metric and hardware each one uses:

If you know of a published result we missed, tell us and we’ll add it.

What’s next

This post covers what we measured. A later post will go into the technical details of how we built our serving engine, along with more results.