- orchestrator_core: execute_classical is genuine seeded QKV
self-attention; execute_flash_attention delegates to
rtx_flash_attention::flash_attention_forward (8-head reshape, clear
Err on indivisible hidden); execute_hybrid composes the two;
execute_edge/execute_distributed return explicit
"modality not implemented" errors instead of fake success — which
makes the previously-dead fallback_modalities retry loop real.
- tensor_core_kernels: AttentionComputationOptimizer::execute
dispatches Standard/Online to scaled_dot_product_attention, Flash to
rtx-flash-attention, Approximated to an explicit Err (no
approximation kernel exists; refuses to compute exact attention
under an approximated label).
- 10 new always-on tests incl. flash-vs-standard 1e-3 agreement and an
Edge->Distributed->Classical fallback end-to-end.
- docs/consolidation.md updated: both entries moved from
scaffolding/no-op-stub status to real dispatch descriptions.
982 rtx-transformers lib tests pass.
Co-Authored-By: Claude Fable 5 <[email protected]>
Call-site rewrite pass audited every flagged site; none needed
consolidation: metal_moe is an API-consistent backend specialization,
modular/router.rs is module-level (not expert-token) routing, glam.rs
is disabled dead code with a pre-existing bug (noted for whoever
re-enables it), and the three flash-attention "reimplementations" turn
out to be planner scaffolding, no-op stubs, and a doc comment — no
attention math exists to delegate. jepa_gpu's attention is documented
as part of the fused GPU ViT block by design.
Verified no regressions: rtx-transformers 961 lib tests pass, jepa_gpu
34/34 with cuda, rtx-training cuda check clean.
Co-Authored-By: Claude Fable 5 <[email protected]>
Extends jepa_train with distributed launcher, jepa_data with advanced
sampling and preprocessing, jepa_gpu with full CUDA kernel wiring,
jepa_distributed/runner/metrics/vit with additional training stages.
Adds jepa_integration module and project-local cargo config.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Implements variable-length (varlen) FlashAttention that processes
mixed-length batches without padding waste:
- New CUDA kernel flash_attention_varlen_forward with BLOCK_Q=64 /
BLOCK_K=64 tiling; grid=(ceil(max_seqlen_q/64), num_heads, 1).
Each block uses a linear scan over cu_seqlens_q to identify its
owning sequence and exits early when past sequence end.
- New Rust module flash_varlen_forward: always-compiled CPU simulation
(varlen_attention_cpu) for testing + #[cfg(cuda)] FlashVarlenKernel.
- SdpaBackend::VarLen variant added to backend_selector.
- 8 new CPU-only tests; total test count: 50.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Comprehensive layered architectural review covering all 109 crates,
306K LOC, 13,351 tests. Identifies 9 gaps (G0-G8) with the highest-
priority being 45 unimplemented! panics across rtx-backend-cuda/rocm/sycl
and the rtx-distributed workspace exclusion. Includes 17-item 4-phase
roadmap through 90 days.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
