Whole-workspace rustfmt pass picked up while iterating on Mamba GPU
backward work. Verified formatting-only via diff sampling; no logic
changed.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
- Eval: run_jepa_training now runs k-NN (k=5) + linear-probe evaluation
every eval_every steps and at the end (deduped when aligned);
JepaEvalResult recorded in JepaTrainingSummary (final_knn_acc /
final_probe_acc), printed by the CLI, appended as an # eval section
to the metrics CSV. Probe set is deterministic LCG synthetic (offset
seed, never aliases training batches) or real shard labels when
loaded.
- NCCL: real GPU-direct AllReduce backend behind the new `nccl`
feature (cudarc/nccl, dlopen-based so builds don't need libnccl).
NCCL unique id is bootstrapped over the existing TCP rendezvous
(master_port+137); data path is htod -> ncclAllReduce(Sum) -> dtoh
-> mean. catch_unwind guards cudarc's panic-on-missing-lib so
training falls back instead of aborting. Verified for real on the
RTX 5060 Ti: single-rank GPU all_reduce identity test passes
(26/26 with --features nccl).
- GPU checkpointing/eval: JepaTrainerV2::context_encoder_cpu_weights()
exposes host-side weights for both CPU and GPU encoders
(GpuViTEncoder::cpu_weights); checkpoint save and eval now work for
GPU training runs (verified: .jepa binaries written and 2 eval
passes during a live GPU CLI run). Resume with a GPU encoder
restores the step counter and warns that weight re-upload is not
yet implemented rather than silently training on stale weights.
125 runner/distributed/vit tests pass; CLI 8/8; cuda check clean.
Co-Authored-By: Claude Fable 5 <[email protected]>
- jepa_gpu: remove the bring-up 2-block cap; full-depth GPU-resident
ViT verified against a full-depth CPU reference on the RTX 5060 Ti
(depth-12 ViT-Tiny max_rel_err <= 6.6e-5). CpuViTEncoder's own hidden
min(depth,2) cap removed too — CPU-path callers now get the model
they configured.
- jepa_distributed: real TCP parameter-server AllReduce backend
(rendezvous handshake with world-size/rank validation, length-
prefixed f32 payloads, connect/read/accept timeouts, connect retry
until deadline so early peers survive rank 0 still computing);
jepa_runner wires it for world_size > 1 and fails hard on collective
errors. Two-rank loopback training run covered by test.
- rtx-jepa-cli (new crate): rtx-jepa binary with train/bench/plan/
validate subcommands driving JepaRunConfig, run_jepa_training,
run_jepa_benchmark, and ClusterTrainingPlan (plan --emit-config
round-trips through a config serializer). GPU bench on this node:
86k patches/sec vs 1.3k CPU (~64x).
- ViTSizeStr::Micro (d=32, depth=2) added as an explicit test/smoke
size now that no hidden caps keep full-size configs cheap; heavy
tests moved onto it (rtx-transformers suite: 367s -> 5s, and the
runner subset had ballooned to 35min at full depth before this).
966 lib tests pass; 35/35 jepa_gpu with cuda; 8/8 CLI tests.
Co-Authored-By: Claude Fable 5 <[email protected]>
The workspace root was upgraded to thiserror = "2" in an earlier commit,
but 56 per-crate Cargo.toml files still independently declared "1.0".
These crates do not use workspace.dependencies inheritance for thiserror.
All updated to thiserror = "2" for complete fleet alignment.
Includes: rtx-backend, rtx-tensor, rtx-losses, rtx-backend-cuda/rocm/metal,
all training crates (rtx-auto, rtx-rl, rtx-distributed, rtx-federated, etc.),
specialized crates (rtx-science, rtx-platform, rtx-nmf, rtx-neuro-*),
production crates (rtx-streaming, rtx-serving-api), and all demo crates.
cargo check --workspace: PASSES.
