build.rs was hardcoded for sm_90 (incorrectly labelled Ada Lovelace/RTX
5090). Fix for RTX 5060 Ti (sm_120, Blackwell):
- Auto-detect SM via CUDA_ARCH env var (default sm_120); compute_ prefix
derived automatically so compute_120/sm_120 are no longer hardcoded.
- nvcc resolution: try PATH first, then CUDA_PATH/bin/nvcc, CUDA_HOME,
and common installation prefixes — no longer panics when nvcc is at
/usr/local/cuda-13.1/bin but not in $PATH.
- PTX version: sm_100+ → .version 8.0 (PTX ISA 8.0 for Blackwell).
- No-GPU branch: remove the warning — CPU fallback is valid, there is
no reason to warn every build when cuda/metal are intentionally off.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Adds d350_gpu_backbone_training — 200-step Adam loop on a 4-regime
corpus that prefers Device::cuda(0) and falls back gracefully to CPU.
Measured numbers:
- CPU (DIM=16): 886 steps/s, MSE 0.2163 → 0.0024
- GPU (DIM=16): 803 steps/s, MSE 0.2163 → 0.0024
GPU is marginally slower at DIM=16 because the SSM scan and conv1d
remain on CPU in both paths; cuBLAS only helps the four linear
projections, which are tiny at dim=16. The GPU advantage emerges at
larger dims (≥256) where the projections dominate. Correctness is
identical on both devices.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
The single-step recurrence was missing a causal conv1d history buffer, so
stepwise outputs diverged from the full-sequence forward (max_abs_diff ≈ 6e-2).
Add `conv_buf: Vec<f32>` to `MambaState` (oldest-first per channel), thread it
through `MambaRecurrence::step` so the kernel sees the correct `kc-1` prior
x_in values, and shift the buffer after each step.
Ergonomic additions:
- `MambaRecurrence::init_state()` — zero-initialised state with correct dims
- `MambaRecurrence::state_size()` / `d_model()` — accessor methods
- `MambaState::hidden()` — slice accessor for the SSM h vector
- `MambaState: PartialEq` — enables determinism assertions in tests
Both D309 tests now pass: `step_matches_full_forward` and
`fresh_state_is_zero_and_deterministic`.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Implements four new rtx-transformers layers required by omni-cortex's omni-think
crate: MambaRecurrence/MambaState (single-token S6 recurrent step),
ClonedMemoryUpdater (linear-tanh cell with SGD + rollout refinement),
GatedMemoryUpdater (GRU-style cell with full backward pass), and
SetEncoderTeacher (time-parallel set encoder with named_params persistence API).
Fixes 10 compile errors in omni-think.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Adds an opt-in learned-query attention pool to SetEncoderTeacher
(SetEncoderConfig::with_learned_pool): pool = softmax(q_mem·hᵀ)·h instead of the
fixed mean/decay pool — Isola's transformer-teacher [MEM] query. New q_mem param
(threaded through graph/train_step/run/named_params; only updated in learned-pool
mode). Default off → existing teachers byte-unchanged.
Finding (test teacher_content_addresses_selective_retrieval): on a selective-
retrieval task (signal in one marked token among distractors) BOTH the mean-pool
and learned-query teachers recover the marked token to low MSE (~0.0006 / ~0.002)
— because the self-attention layer already routes the marked token's signal to
every position before the pool. So the mean-pool was NOT the recall bottleneck
(correcting the D318 hypothesis): the teacher can content-address; the real
recall bottleneck is the recurrent *cell* that imitates it. The learned-query
pool is shipped as an equally-capable, faithful-to-the-talk alternative.
All 5 teacher unit tests pass; clippy(-D)/fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
ClonedMemoryUpdater::train_rollout rolls the cell forward over K inputs on its
OWN memory (not teacher-forced) and backprops the accumulated predict-the-future
loss through the whole K-step graph — the tape's first multi-step training path,
directly optimizing the free-rollout behavior the cell is evaluated on (vs the
one-step BC/DAgger paths).
To stay within the finite-diff-gated op set (matmul/gelu/add/mul/sub/sum — no
tensor concat), W_in is split into its memory rows (applied to M) and input rows
(applied to x); the two gradient halves are re-stacked for the Adam update.
Non-finite guard + gradient clip as in the other training paths.
Test: rollout training reduces the K-step loss (>2x). clippy(-D)/fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
SetEncoderConfig gains `recency_decay` (default 0.85) + `with_recency_decay`,
threaded into pool_weights. A smaller decay concentrates the recency pool on the
most recent tokens — a *sharper* oracle that reacts fast to regime switches
(less denoising). Existing teachers default to 0.85 (unchanged). Lets omni-think
tune the teacher's reaction speed for non-stationary streams.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
F.1 — finite-difference gradchecks for tanh, sigmoid, and the exact gated
composite (1-z)*m + z*c (z=sigmoid, c=tanh) added to tape_cpu_gradcheck.rs.
All pass (rel-err < 2e-2) — the activation VJPs that already existed are now
proven correct on the real CpuBackend (E0 discipline), so the gated cell can
rely on them.
F.2 — GatedMemoryUpdater: a GRU-style gated recurrent memory cell.
z = sigmoid([M‖x]·W_z); c = tanh([M‖x]·W_c); M_t = (1-z)⊙M + z⊙c
The convex update is a non-expansion (|M_t| ≤ max(|M_0|, 1)), so free rollout
stays bounded with NO clamp — and the learned gate can both jump at a regime
switch (z≈1) and hold+denoise in steady state (z≈0), which the residual+leaky
ClonedMemoryUpdater cannot. Same method surface (new/step/predict/train_step/
train_step_memory) so the omni-think facade is cell-generic.
Tests: gated cell trains (loss drops); 500-step free rollout stays in [-1,1]
without a clamp. clippy(-D)/fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
SetEncoderConfig gains an opt-in `recency` flag (default false = unchanged
mean-pool stationary behavior). When set, the encoder adds sinusoidal timestamp
embeddings to the token embeddings (so attention can reason about position) and
pools with an exponential-decay (recent-weighted) reduction instead of a uniform
mean — a recent-window sufficient statistic that tracks non-stationary signals.
Both use only existing-VJP tape ops (add + matmul/softmax); no new params.
Test: recency-mode teacher trains end-to-end (loss >5x drop). Existing
stationary tests unchanged. clippy(-D)/fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Two robustness additions for on-policy distillation, which trains on the
policy's own (clamped, sometimes extreme) visited states:
- train_step now skips non-finite-loss steps and clips gradients, so an extreme
rollout state can't poison every weight via Adam.
- train_step_memory: a memory-only correction at a caller-chosen lr — trains
just the recurrence (W_in, W_mem) to map (prev,x)->target_mem, leaving the
BC-trained readout (W_read) intact. This is the key to DAgger working: it
pulls the free-running memory back toward the oracle trajectory without the
readout retraining that otherwise blows the predictions up.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Make the cloned updater's recurrence M_t = ρ·M_{t-1} + Δ a contraction (ρ=0.9)
and clamp the free-running state (|M| ≤ 4). One-step BC has no signal against
autoregressive drift (that is E4/DAgger's job); these keep a free rollout
numerically bounded — no 1e25 blow-up — so E4 has a stable base to refine.
Training is teacher-forced on the bounded oracle memory, so neither ever binds
during training.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A trainable recurrent cell g(M_{t-1}, x_t) -> M_t (gelu MLP, residual memory
update) plus a predict-the-future readout, trained on Autodiff<CpuBackend> with
the same host-Adam pattern as SetEncoderTeacher.
train_step is one-step behavioral cloning, TEACHER-FORCED on the oracle memory:
each example feeds the oracle's previous memory and regresses (new_mem ->
target_mem) + (readout -> next latent) in a single tape graph. No rollout, no
backprop-through-time. Off-tape step/predict use a gelu byte-identical to the
CPU tape's (tanh approx, sqrt(2/pi), 0.044715) so rollout matches training.
The residual update M_t = M_{t-1} + delta lets the recurrence rule stay
length-invariant — the basis for extrapolating past the teacher's horizon (E3b
generalization test, omni-cortex side).
Tests: updater learns a one-step transition (>10x loss drop); off-tape
step/predict shapes + finiteness. clippy(-D)/fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
MambaRecurrence reconstructs MambaBlock's selective-scan as an explicit
step(state, x_t) -> (state', y_t) recurrence over host f32 weights, with the
recurrent state (h, conv-ring) carried between steps. This is the substrate the
SMT memory updater is behaviorally cloned on (E3b) — no BPTT, one step at a time.
Dimensions are derived from the persisted tensor shapes, so mamba.rs is left
untouched (it is near the 1250-LOC cap). silu/softplus are byte-identical copies
of the forward's.
Equivalence pin (the plan's highest-risk item): stepping a window one token at a
time from a zero state reproduces the full-sequence forward EXACTLY —
max_abs_diff = 0.0 (bit-identical), on an active_block with wide Delta so the
scan genuinely drives the output. Plus a fresh-state determinism test.
clippy(-D warnings) + fmt clean on the new module and test.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Promotes the proven attention set-encoder smoke test (PR #4,
tape_train_smoke.rs) into a reusable rtx-transformers layer for Supervised
Memory Training. The teacher maps a window of past tokens to a fixed-size
memory M via embed → self-attention → residual → mean-pool → memory
projection, with a decoder head supervised by predict-the-future MSE so that
M becomes a sufficient statistic of the past.
- Trains end-to-end on Autodiff<CpuBackend> (the gradient-correct real backend
from PR #3), with a self-contained deterministic host-side Adam.
- Time-parallel by construction (one window → one memory, no recurrence to
unroll) — this is the oracle whose trajectory the recurrent Mamba updater is
later behaviorally cloned against, so the recurrent net never needs BPTT.
- Exposes named_params/set_named_params so the caller (omni-think's
PredictiveStateTeacher facade) owns safetensors persistence + BLAKE3 sealing.
- Adds rtx-backend + rtx-backend-cpu deps (the tape needs a concrete backend).
Tests: teacher trains (loss >5x drop), encode is deterministic + fixed-size,
params round-trip. fmt + clippy(-D warnings) clean on the new module.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
End-to-end training on `Autodiff<CpuBackend>` (forward → backward →
extract grads → SGD → repeat), guarding the gradient-correctness fixes:
- `mlp_trains_and_loss_decreases`: 2-layer MLP (matmul + gelu), loss 8.65 → 0.39.
- `attention_set_encoder_learns_window_mean`: a minimal attention set-encoder
(the SMT predictive-state teacher shape) learns to predict its input window's
per-dim mean, loss 0.27 → ~0.0. The embedding fans out to Q/K/V and the
residual (4 uses), so this also regression-guards the fan-out gradient
accumulation fix inside a real attention block.
All ops used are gradient-checked in `tape_cpu_gradcheck.rs`.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The decorator autograd (`Autodiff<B>`) had never been gradient-checked
against a real tensor backend — the entire test suite runs on a shape-only
`MockBackend` whose ops return their input, so they validate graph structure
but never gradient values. Running it through `CpuBackend` for the first time
(new `tests/tape_cpu_gradcheck.rs`, finite-difference checks) surfaced three
bugs that made the tape unusable for training; this fixes all three.
1. Double-free / UB in the dimension-erasure cast. The backward ops cast a
tensor to its runtime const-generic dimension via
`mem::transmute_copy::<_, TensorPrimitive<N>>(&src)` in ~100 sites. That
bit-copies the owned `Vec` without forgetting the source, so two values own
one buffer → double-free on any heap-backed backend (and Stacked-Borrows UB
from the typed pun). Replaced every site with a single `into_dim` helper
that is now **fully safe** — it round-trips through `to_data`/`from_data`
and rebuilds the shape with `array::from_fn`, no `unsafe` at all. (This is
why the whole repo previously bypassed the tape with analytic backward.)
2. Fan-out gradients were silently dropped. `accumulate_gradients` was a stub
that returned one path and discarded the other, and `AutodiffTensor::clone`
minted a fresh `TensorId`. Together, reusing a tensor (residuals,
`mul(s, s)`, shared Q/K/V — universal in transformers) split its gradient
across two ids and summed neither, yielding a fraction of the true value.
`accumulate_gradients` now sums via `B::add`; `clone` preserves the id so
fan-out paths collide on one sink.
3. Softmax backward panicked. `SoftmaxBackward` / `stable_softmax_backward`
subtracted a keep-dim row-sum from the full-shape grad, but the elementwise
backends assert equal shapes (no broadcasting). Added `broadcast_along_dim`
to tile the row-sum to full width first.
Verified: `tape_cpu_gradcheck` (matmul, fan-out add·mul, softmax) passes with
rel-err < 2e-2 vs central differences; full `rtx-autograd` suite green (263
passed, 0 failed); lib clippy `-D warnings` clean.
Known follow-up (out of scope): `cargo miri test` still aborts on a
Stacked-Borrows / integer-to-pointer violation inside `rtx-backend-cpu`'s
buffer internals — a grad-free `from_data`+`add`+`sum` probe reproduces the
identical error, so it is pre-existing backend UB, not an autograd issue.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Persist accumulated WIP across rtx-tensor / rtx-transformers / rtx-runtime.
Two related feature groups:
GPU enablement (unblocks the Phase-3 spec §8 CUDA Mamba path):
- rtx-tensor: `Tensor::cuda_device_ptr()` + storage GPU-buffer accessors
(`raw_ptr.rs`) expose the raw CUdeviceptr that kernel launches need —
the "rtx-tensor GPU memory access API" the Mamba CUDA kernels were
blocked on.
- rtx-transformers: `MambaBlock::forward_cuda` runs the four linear
projections through cuBLAS on-device (in/out/x/dt_proj), keeping the
selective scan + conv1d + activations on CPU; dispatched automatically
from `forward` when on a CUDA device under the `cuda` feature. Updated
`mamba_cuda_kernels.rs` accordingly.
- supporting plumbing in rtx-runtime stream/bridge and rtx-tensor
storage/conversion/concatenation/creation + rtx-flash-attention.
Linear algebra (rtx-tensor):
- `linalg/matrix_exp.rs`: real matrix exponential via scaling-and-squaring
with a degree-13 Padé approximant (Higham 2005), f64 internally.
- `complex/linalg.rs`: complex matmul/adjoint, Hermitian eigendecomposition
(`ComplexEigenResult`), and the complex matrix exponential, nalgebra-backed.
- tests for both.
Builds verified on the CPU path (`cargo check -p rtx-tensor -p rtx-transformers
-p rtx-runtime -p rtx-flash-attention` clean). The `cuda` feature and the
rtx-backend-cuda NVCC build remain unbuildable on this host (CUDA/glibc header
mismatch) — pre-existing and unrelated to these changes.
End-to-end demonstration that training the real selective-scan Mamba buys
a genuine temporal-modeling win — the payoff of M1–M3.
Task: next-token prediction on a multi-regime sequence (x[t] = μ_regime +
noise). Predicting x[t+1] inside a regime requires integrating recent
history to average out the noise — a memoryless model can't.
Result (same sequence, all three):
persistence (memoryless) MSE = 0.0895
untrained Mamba MSE = 0.2167
trained Mamba (400 Adam) MSE = 0.0002
The trained backbone integrates history to de-noise the regime mean,
beating the memoryless persistence baseline by ~450× and improving
~1000× over its untrained self. (Single-sequence fit: demonstrates the
SSM's temporal-modeling capacity, not held-out generalization.)
This replaces the old non-result ("random Mamba 6.5% vs linear 48%") with
a real "trained SSM exploits temporal structure" demonstration.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Plain randn(0,1) init made A=-exp(A_log) and Δ wildly large, overflowing
the real exp(Δ·A) scan to NaN (the stub never cared — it discarded these
weights). new()/new_seeded() now share a build() with canonical S6 init:
- A_log = ln(1..=d_state) ⇒ A = -(1..=d_state), bounded
- dt_bias so softplus(dt_bias) ≈ 0.01 (small, stable Δ; near-identity
scan at init — intentional for gradient flow)
- D = 1, zero conv bias, projections scaled by 1/√fan_in (capped 0.5)
This fixes the NaN that broke omni-cortex's d231 action-conditioned
predictor training (now green). Seeded determinism preserved.
Tests: active_block helper (Δ overridden to ≈0.69) exercises the
scan-active regime so the liveness check can observe each weight; the
training test asserts a seed-varying backbone weight (conv1d_weight)
moves. All 6 selective-scan tests green incl. the finite-diff grad check.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Demonstrates end-to-end trainability of the real selective scan. A
self-contained Adam loop (the rtx-transformers AdamOptimizer has no
public gradient setter — the spec permits a bespoke loop) fits a teacher
block's output on a fixed input: forward → MSE → analytic backward →
Adam step → rebuild. Over 200 steps the loss drops >50% and the backbone
weight A_log moves, confirming gradients actually train the model (not
just the head). All 6 selective-scan tests green.
The production AdamOptimizer can be wired once it exposes a gradient
setter; the M2 backward already returns grads in its HashMap shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Hand-written VJP of the selective-scan forward (Phase-3 spec §5/§7):
MambaBlock::backward(x, d_out) -> per-parameter gradients keyed by
persistence name, summed over the batch. Differentiates the scan
analytically (reverse-time recurrence over the cached h trajectory)
rather than through the immature rtx-tensor autograd tape.
Covers every parameter: in_proj, conv1d_weight, conv1d_bias, A_log
(via A=-exp(A_log) ⇒ dA_log = dA·A), x_proj, dt_proj, dt_bias, D,
out_proj. Adds stable sigmoid_f32 / silu_grad_f32 helpers.
New test analytic_gradients_match_finite_differences: on a small
well-conditioned instance, ≥30 sampled grad elements across all 9
params match central finite differences within (5e-3 + 5e-2·|fd|).
All 5 selective-scan tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The MambaBlock forward was a stub — SelectiveScan::forward passed input
through, discretize returned zeros, conv1d was a no-op, and B/C were
randn per call, leaving conv1d_weight/dt_proj/A_log as dead weights with
zero temporal mixing. This implements a genuine S6 selective-scan:
- New params: x_proj [d_inner, dt_rank+2*d_state] (data-dependent dt,B,C),
dt_bias [d_inner], D [d_inner] (skip). Added to new()/new_seeded() and
the persistence contract (persistence_tensors/from_persistence_tensors).
- Real forward (CPU f32, looped — backbone is small): in_proj -> causal
depthwise conv1d -> SiLU -> x_proj->(dt,B,C) -> delta=softplus(dt.dt_proj
+dt_bias) -> A=-exp(A_log) -> sequential scan h=dA.h+dBu, y=sum C.h + D.u
-> gate by SiLU(z) -> out_proj. Residual moved OUT (canonical).
Numerically-stable silu_f32/softplus_f32 helpers.
- The scan runs inline (not via the immature rtx-tensor autograd tape);
the analytic backward lands in M2 per docs/phase3_real_ssm_spec.md.
New tests/real_selective_scan.rs (4 cases, all green): liveness (each
formerly-dead weight now moves the output), causality (no future
leakage), seeded determinism, and finite/non-constant output.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add 10 new tests exercising previously uncovered GenericTensor operations:
leaky_relu, elu, gt_scalar, var, var_dim, conv2d (identity kernel,
stride, grad propagation), max_pool2d, and avg_pool2d. Also add
test_utils.rs module for shared test helpers.
SDLC-Coverage: rtx-tensor generic/tensor.rs 19.6% → targeting ~40%+
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Two coordinated additions for the omni-cortex D249/D250 work:
rtx-tensor: Tensor::randn_seeded(shape, device, seed) — like
randn() but routes through StdRng::seed_from_u64(seed) so two
calls with (shape, device, seed) produce bit-exact identical
tensors. CPU is the canonical generator; GPU calls go via to_device
transfer. Required for reproducible model init.
rtx-transformers: MambaBlock gains:
- new_seeded(config, device, seed) — every internal weight tensor
initialised via randn_seeded() with per-tensor SplitMix64-derived
seeds. Two calls with the same (config, device, seed) → bit-
exact identical block.
- persistence_tensors() -> Vec<(&'static str, &Tensor)> — read-only
view of the six (or seven, with conv_bias) internal weight
tensors with canonical names (in_proj, conv1d_weight,
conv1d_bias?, A_log, dt_proj, out_proj). Stable across versions
so safetensors round-trip works.
- from_persistence_tensors(config, device, HashMap<String, Tensor>)
— rebuild a MambaBlock from a name → tensor map. Validates each
tensor's shape against the config and surfaces clean errors on
mismatch (so wrong-DIM safetensors loads fail explicitly).
These three primitives together give omni-cortex's D249 (operator-
seeded determinism) and D250 (safetensors round-trip + BLAKE3 hash
pin) clean library hooks without exposing MambaBlock's private
fields.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the LoRA inference path that was previously stubbed. Two new
public APIs in rtx-csm:
1. lora::load_lora_set_from_safetensors(path, config) -> LoraSet
Reads a trained adapter file (produced by training::
save_lora_adapter[_with_metadata]). Pairs the .lora_a / .lora_b
tensors by base-weight prefix into LoraAdapter entries.
2. lora::merge_into_safetensors(base, lora, scale, output)
Reads the base CSM safetensors, folds in the LoRA deltas at the
given scale (typically alpha/rank from training), writes a merged
safetensors. Original dtype preserved (F16 on Metal, BF16 on
CUDA, F32 on CPU). Tensors LoRA doesn't target are passed
through unchanged.
3. Generator::load_csm_1b_from_path(path, device)
Variant of load_csm_1b that takes an explicit weights path
instead of going through the HF cache. Mimi + tokenizer still
resolve via the hub. This is the path consumers use to load a
merged checkpoint.
MergeReport struct restructured to expose merged/skipped/passthrough
counts so callers can verify the adapter actually targeted weights.
The previous typed-error test is replaced with a missing-base-file
test that exercises the real code path.
Used by zeroclaw-channel-voice's `--lora-adapter` flag to bake a
LoRA adapter into a per-process merged checkpoint at boot, with
zero per-inference overhead.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Two additions for downstream voice-channel prosody / quality wiring:
1. Converse::with_pre_sentence_hook
New PreSentenceHook (Box<dyn FnMut(&mut Generator, &str) -> Result<()>>)
that fires before each sentence's synth call. Receives mutable
access to the underlying Generator + the sentence text — lets
callers apply per-sentence steering (e.g. emotion shifts mid-reply)
without touching crate internals. Wired into both `synthesize` and
`synthesize_streaming` paths.
2. PostProcess streaming split
- New StreamingHpfState — stateful biquad whose IIR taps carry
across chunk boundaries so streaming HPF doesn't click at chunk
joins. Identical filter coefficients to the one-shot path.
- PostProcess::apply_chunk_safe(samples, hpf_state) — HPF + declick
per chunk, no LUFS (needs full utterance).
- PostProcess::apply_lufs(samples, sample_rate) -> Result<f32> —
full-utterance loudness gain, returns the linear gain applied
so streaming pipelines can compensate retroactively if needed.
- compute_lufs_gain helper extracted from loudness_normalize.
Used by zeroclaw-channel-voice for the Maya-gap-closure pack.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
ring_allreduce() contained its own simulation that multiplied each gradient
value by world_size (to fake an AllReduce sum), bypassing the ProcessGroup
dispatch entirely. This meant the overlapped synchronization path never used
NCCL or RNCCL, even when those features were compiled in.
Replace the hand-rolled simulation with a call to
self.process_group.allreduce(tensor, ReduceOp::Sum) so the overlapped path
uses the same backend as synchronize_gradients_sequential. The communication
latency sleep is kept for benchmarking purposes.
Add test_ring_allreduce_matches_sequential_path to verify both paths produce
identical gradient values under CPU simulation.
Closes#10
Without a voice anchor, CSM-1B picks a different speaker each turn
and drifts mid-sentence on longer outputs (high-pitch squeaks,
female/male swap mid-utterance). The fix is the standard CSM
speaker-prompt pattern: pass a Segment with reference audio + its
transcript as context to every generate() call.
Previously Converse::synthesize and synthesize_streaming hardcoded
`&[]` for the context arg. Add a `context: Vec<Segment>` field on
Converse plus a builder method:
let conv = Converse::new(&llm, &mut gen)
.with_context(vec![Segment::new(0, transcript, audio)]);
Both synth paths now pass `&self.context` instead of `&[]`. Empty
context (default) keeps prior behavior.
Verified end-to-end with zeroclaw-channel-voice + macOS `say`-
generated reference: same input now produces deterministic-length
output across turns (2.64s vs. previously varying 6/19/38s) and the
voice matches the seed throughout.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The canonical voice loop now lives in zeroclaw-channel-voice
(`~/projects/zeroclaw/crates/zeroclaw-channel-voice`, binary
`voice_server`). It routes the LLM path through zeroclaw's agent
runtime — multi-turn history, tools, memory, provider routing —
instead of the OpenAI-compatible direct path here.
Same WS wire protocol so `examples/converse_client.rs` drives both;
no client-side migration needed.
This binary is intentionally kept buildable for:
1. Reproducing perf_history.md Phase 8.10 benches.
2. Standalone (no-agent) use when zeroclaw isn't desired.
Module doc + main() startup banner updated to point at the new home.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk
Whisper-LV3:
target RAVDESS CREMA-D
happy happy (0.999) ✓ happy (0.999) ✓
angry neutral (0.92) sad (0.99)
fearful happy (0.998) fearful (0.984) ✓
sad angry (0.99) fearful (0.99)
CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus
produces more class-pure fearful direction. Neither corpus solves
angry or sad — recipe shifts into 'vague expressivity' rather than
class-specific corners.
Practical: prefer CREMA-D when available; A/B both per emotion if
class precision matters.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Python sidecar for scoring TTS outputs against the firdhokk
Whisper-LV3 SER classifier (sanity-verified non-saturated, 3/5
correct on RAVDESS ground-truth).
Replaces the in-process emotion2vec_plus_base path which collapses
to 'Surprised' on every input (documented in
emotional_speech_guide.md and quality_eval.rs caveat).
Reads JSONL with {gen_wav, target_emotion} rows; writes JSONL with
top_emotion, top_prob, target_prob, match (bool), and the full
8-class probability distribution.
Class set is firdhokk's 7 (no calm — calm aliases to neutral on
input). Excited aliases to happy.
Smoke-verified on the 4 prior decoder-route outputs (Amini ctx,
seed=42, recipe defaults):
happy → neutral (0.80) ✗
angry → happy (0.999) ✗
fearful → fearful (0.68) ✓
sad → fearful (0.998) ✗ (sad↔fearful confusion)
Top-1 match: 1/4 — confirms the gap documented in
emotional_speech_guide.md 'Known Limitations'.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
P2 fixes:
- Run cargo clippy --fix to auto-fix ~20 warnings (format strings, closures, etc.)
- Fix post.rs: &mut Vec<f32> -> &mut [f32] (prefer slice over Vec reference)
- Fix prompt.rs, longform.rs, training.rs: add blank lines before doc list items
- Fix diarize.rs: replace loop with slice.fill(true) for cleaner code
- Add #[allow(clippy::needless_range_loop)] where i is used in arithmetic
- Add #[allow(clippy::too_many_arguments)] to functions needing builder refactor
Down from 37 warnings to 0 warnings in rtx-csm.
P0 fixes:
- Add [[test]] required-features=["cuda"] to Cargo.toml so backend_parity_tests
only compile when the cuda feature is enabled (avoids E0425 compile errors)
- Add missing Lazy/OnceCell/RwLock/Arc/HashMap imports to device.rs
- Add missing Arc import to tensor.rs
- Add missing Backend/BoolU8/DeviceId/DeviceOps imports to lib.rs
All 16 backend parity tests now pass with --features cuda.
Wired up firdhokk/speech-emotion-recognition-with-openai-whisper-large-v3
as a working alternative to the broken emotion2vec_plus_base. Sanity
verified on real RAVDESS clips: 3/5 correct, 2/5 near-miss (happy↔
surprised, sad↔fearful). Probabilities are NOT saturated — the
classifier actually distinguishes per-input.
Then scored our 4 decoder-route outputs (Amini context, seed=42,
recipe defaults) and found that **only fearful registers as the
intended class**:
target verdict conf
happy neutral 0.80 ✗ (steering produces neutral output)
angry happy 0.999 ✗ (high-arousal cross-class)
fearful fearful 0.68 ✓
sad fearful 0.998 ✗ (sad↔fearful confusion)
Honest framing: the recipe shifts speaker character toward an
expressive-sounding direction (cosine evidence) and preserves text
(decoder vs backbone) but does NOT produce class-distinct emotion.
The metric stack we used through Phase 9 (cosine + WER) couldn't
see this gap because it measures voice fidelity and text rendering,
not emotion class.
Hypothesized fixes (not yet tested):
- CREMA-D extraction (91 actors vs RAVDESS 24) for class-purer
steering vectors
- Mixed backbone+decoder steering (backbone for prosody)
- EmoNet classifier (TTS-aware, may give different verdicts)
Doc'd in emotional_speech_guide.md as a known limitation. Closes
out an honest scientific picture: today's work successfully ports
the architectural finding (decoder route preserves text), but
class-precise emotion control remains unsolved.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wires emotion2vec into quality_eval so per-row metrics include
target_emotion_prob, top_emotion, top_emotion_prob. Pure inference,
optional via --emotion2vec / --target-emotion flags.
Critical empirical finding documented in code + user guide: the
emotion2vec_plus_base checkpoint classifies every input as
"Surprised" with prob ≥ 0.99, INCLUDING ground-truth RAVDESS clips
with explicit emotion labels. Real angry-RAVDESS → "Surprised"
(0.9999999). Real neutral-RAVDESS → "Surprised" (0.9999996).
The metric implementation is correct (matches the trait's
EmotionDetector::classify code path with same per-utterance zero-
mean unit-variance normalization); the underlying classifier
collapses to a dominant class on most input — likely the same
"9→5 fold collapse" the project already documented in the data-
labeling path.
Practical implication: target_emotion_prob is near-zero for almost
every (target, output) that isn't "surprised", so it can't be used
as a picker score. The emotion2vec metric still works as a
diagnostic ("did the model produce something that classifies as
audio at all?") but not as a generation-quality validator.
Doc'd in:
- examples/quality_eval.rs CLI doc (caveat block on --emotion2vec)
- docs/emotional_speech_guide.md (Known limitations section with
full sanity-check table)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Brief addition to the recipe section explaining the WER + length
floor scoring used by emotional_speech_n.sh (committed in b7b267b).
Validates that the new scoring preserves canonical winners on
happy and calm while flipping surprised to the long-and-correct
candidate.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Original picker used min(WER), tie-broken by max(cosine). Fragile:
on emotion=surprised it picked "You can." (2 words, WER 0.93) over
"...Today I want to share something" (13 words, WER 1.00) because
WER weights all errors uniformly — terse-and-mostly-wrong beats
long-and-mostly-right.
New scoring:
score = WER + (1.0 if words(transcript) < 5 else 0)
sort_by(score, -cosine)
Verified on existing benches:
surprised: now picks seed=100 ("...Today I want to share something
with...", 13 words, score 1.0) over seed=7 ("You can.",
2 words, score 1.929 with +1 length penalty).
calm: still picks seed=100 (full transcript revealed: "It's a
good reflection. Not that that. I want to share
something with you that I've been thinking about." —
near-verbatim! the earlier 55-char display had been
truncating it).
disgust: all 3 candidates score ~1.93 (no seed has > 5 words,
all get the length penalty); picker honestly admits
none is good rather than picking a fake winner.
Worth noting: the calm seed=100 case is ANOTHER near-verbatim
single-shot result we missed in the previous bench because the
display truncation hid the full transcript content.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-seed picker run (7, 42, 100) on calm/disgust/surprised refines the
single-shot characterization:
calm: winner seed=100, WER 0.64
"It's a good reflection. Not that that. I want to
share somet..."
(recipe lands the prompt — single-shot at seed 42 only
produced hesitation markers; the picker found a seed
with actual content)
disgust: no reliable seed
(all 3 seeds WER ≥ 0.93; likely RAVDESS corpus issue —
disgust clips are low-energy / acoustically close to
neutral. Try CREMA-D or ESD for this emotion.)
surprised: picker chose seed=7 (WER 0.93, short "You can.") over
seed=100 (WER 1.0, "...Today I want to share something")
— WER weighting issue: deletions and insertions count
uniformly, so terse-but-mostly-wrong beat long-and-
mostly-right. Manual selection or weighting WER less
heavily would help here.
Updated per-emotion table marks disgust as ✗ (corpus limitation),
surprised as ⚠ (picker scoring artifact), calm as ✓ (works with
N-seed picker).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Extracted decoder steering vectors for the remaining 3 RAVDESS
emotions (calm, disgust, surprised). Single-seed bench at
seed=42 on Amini context, decoder route, recipe defaults:
emotion cos_ctx WER transcript
calm 0.70 0.86 "I'm sorry. Um, I don't know."
(natural hesitation markers — the
recipe produces semantically-emotion-
matched content, not just acoustic
shift)
disgust 0.81 0.93 "For that, that..." (truncated)
surprised 0.95 2.57 "Too couple, sorry, and that's saying,
even a premier and super driver..."
(long rambling; voice migrates well,
text drifts)
All 7 RAVDESS emotions now produce coherent English on the decoder
route — calm is solid first-shot, disgust truncates, surprised
rambles. Roll N seeds via emotional_speech_n.sh for the latter two.
emotional_speech_guide.md updated with the per-emotion table now
covering all 7. Voice character preservation (cos vs context > 0.7)
holds for every emotion in the pack.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
osobhandClaude Opus 4.8