Measured on-device · English technical report

How fast can Small speak?

Single-stream incremental PCM, synchronized batches, and offline inference on one RTX 3090. This is model throughput, not browser latency.

What “on device” means here

CUDA events bracket TTS prefill, autoregressive global/local decoding and the GPU audio codec. The end event is synchronized before reading elapsed time. This is a device event time span: it includes host launch gaps when the GPU waits, not just a sum of kernel-active durations. Prompt packing and reference encoding are completed before timing. Luna, forced alignment, scoring, CPU PCM copies, MP3 encoding, HTTP and playback are excluded. See PyTorch CUDA timing guidance.

Single stream RTF = device seconds / generated audio seconds
Batch aggregate RTF = device seconds / sum(audio seconds across all rows)
Per-stream RTF = device seconds / mean(audio seconds per row)
Throughput = sum(audio seconds) / device seconds = 1 / aggregate RTF

RTF < 1 means faster than real time. A small aggregate RTF does not automatically mean each individual stream is real-time. We therefore report both.

Single-stream: streaming versus offline

Streaming really decodes each newly generated ten-frame block into PCM using a persistent causal codec state: 10 frames / 12.5 Hz = 0.8 seconds per chunk. It does not repeatedly decode the growing prefix, and it does not split an already completed waveform. Offline waits for all tokens, then batch-decodes PCM; it saves frequent codec launches and is faster, but offers no early audio. First-PCM time is not network time or time-to-audible playback.

How small should streaming chunks be?

A separate warmed sweep measures the latency-throughput trade-off, with three trials per setting. More frequent codec calls cost substantial GPU work. Chunk sizes below are actual PCM duration, not model time-to-first-audio.

Parallel streaming: 1 through 10, then beyond

¹ Median across trials of each trial’s maximum gap between successive ready PCM chunks. First-chunk warm-up is separate. B=1–10: three measured trials; B=12–48: one trial each, so no robust high-batch variance estimate.

Longer-cache stress check: 60 seconds per stream

A separate fixed 750-frame run checks sustained decode and memory growth at B=1 and B=12, using the same 800 ms PCM chunk and short prompt recipe. Stop decisions are overridden only for this timing workload. This is not a 60-second spoken-content quality test.

The demo uses batches up to 12 for best-of-N, with a smaller-batch retry on GPU out-of-memory. The UI’s N limit remains 12. Successful short-prefix stress tests do not guarantee memory capacity for arbitrary long prompt prefixes; cold shape compilation can also increase latency.

What was optimized

Single-stream CFG cost after optimization

CFG=1 uses one conditional branch, CFG=1.5 two fused branches. The measured B=1 overhead is small on this GPU after fusion; this does not imply zero CFG cost at larger batches or on other hardware.

• Fuse the two CFG branches into one global/local batch while retaining shared sampled codebook prefixes, conditional-only stop decisions and the exact equation u + g(c − u). Left padding preserves text positions and masks.
• Capture the fixed 12-codebook local step in reusable CUDA graphs. Captures restore the pre-warm RNG state; graph replay advances RNG normally. The graph cache is bounded and keyed by batch, CFG and sampling parameters.
• Compile the growing-cache global decoder with dynamic shapes. This fuses small pointwise operations and reduces launch overhead without changing model weights.
• Use the codec’s supported BF16 encoder/decoder weight setter, keeping its quantizer FP32. This reduces steady GPU memory by roughly 6 GiB; it did not independently improve measured speed. See the codec inference contract.
• Batch the PCM codec instead of decoding each candidate separately. Streaming maintains persistent row-specific states and masks finished rows.

Cold costs are real: the initial compilation/warm-up in the compile experiment took 139.1 s. Local graph capture costs typically 0.25–0.5 s per new configuration. Steady measurements exclude warm-up; live UI timings include first-use work if it occurs inside generation. Compiler disk caching helps subsequent processes, but new shapes may still compile. This is not a cold-start benchmark.

Numerical and quality checks

Fused eager and local CUDA-graph code sequences were exactly equal for batches 1 and 4 in the numerical test, and across three repeated optimization seeds. Streaming versus full codec decode was tested with rows terminating at 30, 19, 7 and 25 frames, including mid-chunk termination. All output lengths were correct and PCM finite.

Fusing CFG changes BF16 GEMM batch shape, and compiling global operations changes rounding. Stochastic trajectories therefore differ from the original code: optimization is not bit-identical to legacy. Small waveform differences are numerical evidence, not a perceptual equivalence test. The separate emotion/quality study evaluates actual free-running output with ASR and learned proxies.

Reproducibility and limitations

Three trials per main condition use seeds 777, 778 and 779; medians and observed ranges are shown, not an invented precise confidence interval. Each timing workload forces 100 frames per stream (eight seconds) to stop random early EOS from changing the workload. This is scheduling capacity, not a claim that naturally generated speech always lasts eight seconds. The quality study does not force a minimum length.

Reference codec preparation, prompt tensors and all model weights are resident before the timer. The live GPU server was stopped; no scorer or other model ran concurrently on GPU 0 during measurements. Display-server memory remains. CUDA 1 is not used to produce these throughput numbers. Warm-up uses 20 frames; the first measured longer-cache decode may still incur shape-specific work. A small chunk-size sweep and a 60-second cache stress check are included. GPU clocks were not fixed; long-duration thermal equilibrium, latency under request queuing and independent asynchronous multi-user arrivals were not measured.

{
  "gpu": "NVIDIA GeForce RTX 3090",
  "vram_gib": 23.68426513671875,
  "torch": "2.6.0+cu124",
  "cuda": "12.4",
  "transformers": "5.15.0",
  "python": "3.12.13",
  "model": "laion/Humaneness-Voice-Small",
  "revision": "5de86032771c37a3008d14e78ee7cdb39368e2a8",
  "stage": "S3",
  "codec_revision": "f6e20e543b33d2c252a7ef71bdf8aa71e5ff9169",
  "frames": 100,
  "cfg": 1.5,
  "temperature": 1.0,
  "reference_frames": 36
}

Raw trial records: baseline.json · optimizations.json · compile.json · sweep.json · validation.json · latency.json · long.json