rustyverse/rustytorch
2
0
Fork 0
Code Issues Pull Requests Actions 28 Packages Projects Releases Wiki Activity
431 Commits 1 Branch 0 Tags
0dc95a8de5eb08a44407d7f4de07979cd11d9b18
Commit Graph
3 Commits
Author SHA1 Message Date
osobhandClaude Sonnet 5 4aaa36a57a style: cargo fmt --workspace (whitespace/wrapping only, no semantic change) 
Whole-workspace rustfmt pass picked up while iterating on Mamba GPU
backward work. Verified formatting-only via diff sampling; no logic
changed.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
 2026-08-10 07:09:36 -07:00
Omar SobhandClaude Sonnet 4.6 b45a58792d feat(batch7): interleaved 1F1B, attention-selective checkpointing, flash decoding
CI / Format Check (push) Failing after 6s
Details
GPU Tests / Check GPU Availability (push) Successful in 0s
Details
Performance Benchmarks / Run Benchmarks (push) Successful in 10s
Details
GPU Tests / CUDA Tests (12.1) (push) Has been skipped
Details
GPU Tests / CUDA Tests (11.8) (push) Has been skipped
Details
CI / Clippy Check (push) Failing after 11s
Details
Documentation / Build API Documentation (push) Failing after 14s
Details
CI / Build (ubuntu-latest) (push) Failing after 50s
Details
CI / Build CPU-Only (Explicit) (push) Failing after 1m2s
Details
Documentation / Build User Guide (push) Successful in 7s
Details
CI / Build (macos-latest) (push) Failing after 39s
Details
CI / Test (macos-latest) (push) Has been skipped
Details
CI / Test (ubuntu-latest) (push) Has been skipped
Details
CI / Python Bindings (maturin) (macos-latest) (push) Has been skipped
Details
CI / Python Bindings (maturin) (ubuntu-latest) (push) Has been skipped
Details
CI / WASM Build + Size Check (push) Has been skipped
Details
CI / Distributed Training Tests (push) Has been skipped
Details
CI / CI Success (push) Failing after 0s
Details
GPU Tests / Metal Tests (push) Has been skipped
Details 
Interleaved 1F1B pipeline schedule (rtx-distributed):
- PipelineConfig: num_virtual_stages (default 1) + rank fields; validate()
- PipelineScheduler::generate_interleaved_schedule(): real Megatron-LM
  virtual-stage assignment (mb % m) * p + rank; warmup/steady/drain phases
  with SendActivation/SendGradient pairs
- bubble_ratio(): (p-1)/(p*m) interleaved vs (p-1)/p standard; p=4,m=2
  reduces bubble 0.750 → 0.375; 4 new tests, 24 total pass

Attention-selective activation checkpointing (rtx-distributed):
- CheckpointPolicy::AttentionSelective { attention_patterns } — name-match
  on attn/attention/self_attn/cross_attn/mha; ~40% memory savings
- CheckpointPolicy::Adaptive: replaced layer%2 stub with 3-tier heuristic
  (>4096MB→sqrt(n), >1024MB→every-other, ≤1024MB→all)
- MemoryAwareCheckpointer: AtomicUsize pressure tracking, fallback-to-all
  when over target; re-exported from crate root; 14 new tests, 29 total pass

Flash decoding (rtx-flash-attention):
- flash_decode_cpu(): split-K attention with log-sum-exp chunk reduction;
  matches naive attention within 1e-4 for all tested configs
- FlashDecodeKernel wrapper; num_splits_for_seq_len heuristic (256 tok/chunk)
- flash_decode_forward.cu: 2-phase CUDA (per-chunk partial + reduce kernel)
- SdpaBackend::FlashDecode: score 0.97 for seq_q=1 && kv>=1024; up to 50×
  speedup at 32K tokens; selected over other backends for long-context decode
- 10 unit tests + 3 doctests + 1 backend selector test; all pass

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
 2026-06-27 04:06:55 +00:00
redclawsystems 4d88dc0584 Initial commit 2026-03-04 00:08:42 +00:00