Consolidated record of every shipped optimization, every rejected path
with the data behind the rejection, the production-recommended config,
the architectural lessons captured, and the deferred multi-session
work with realistic sizing.
Headline:
- client first_audio_ms p50 = 529 ms (mock LLM, Q8 + stream + extended warmup + VAD gate)
- real Z.AI loop: ~2 s TTFA p50
- boot cost: ~2.7 s (one-time)
Rejected paths captured (so future sessions don't redo the work):
Q4_K_M (2.85x slower than Q8), whisper-rs linkage (2-3x CSM regression),
Silero V5 via ort (protobuf 3.14 vs 3.21 conflict), Mimi codec Q8
(no candle conv-quant path), KV cache reuse (variance is content-
dependent not state-dependent), rayon for single-connection
(overhead exceeds gain on <100µs tasks), codec swaps (require
backbone retrain), custom distillation (no published checkpoint),
Kyutai 4x flush (hardware-bound on M-series).
Architectural lessons:
1. In-process linkage of external ML runtimes is a recurring trap;
default to sidecar-process pattern.
2. Bench thermals dominate single-machine A/B; 90s cooldown often
necessary.
3. First-frame compilation is the dominant cold-start cost — long
warm-ups are essential.
4. Conv-phase variance is content-dependent, not state-dependent.
5. tokio::join! polls cooperatively — spawn separate tasks for real
concurrency between sync compute and async pump.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
