rustytorch/CHANGELOG.md

RustyTorch++ Changelog

[December 28, 2025]

Added - TIER 2 Feature Completeness

FlashAttention CPU/Metal Backward

• Files: rtx-flash-attention/src/core.rs, src/lib.rs
• CPU backward pass (naive_attention_backward)
• Metal backward pass (trait impl to kernel)
• 30 gradient tests passing

Nested vmap Support

• Files: rtx-autograd/src/vmap.rs
• BatchDimStack for tracking batch dimensions through nested calls
• MultiBatchedVariable for dimension collapsing/expansion
• VmapLevelGuard RAII guards for level tracking
• 12 vmap tests passing

Hessian-Vector Product (hvp)

• Files: rtx-autograd/src/func.rs
• hvp(f, primals, tangents) with forward-over-reverse mode
• hvp_finite_diff for numerical validation
• vhp (vector-Hessian product)
• Fixed f32 precision issues (use f64 intermediate, larger epsilon 1e-3)
• 7 hvp tests passing

Dynamic Shape Guards

• Files: rtx-synthesis/src/aot_impl/shape_guards.rs, src/aot.rs
• Shape guard generation (generate_guards_from_operations, generate_symbolic_guards)
• Recompilation triggers (GuardCheckResult, GuardFailure)
• Shape dimension types: Concrete, Symbolic, Bounded, Dynamic
• Shape bucketing for cache efficiency (ShapeGuardManager, ShapeSignature)
• 20 tests passing

Advanced Quantization (AWQ, GPTQ, SmoothQuant)

• Files: rtx-compress/src/quantization/advanced.rs
• AWQ (Activation-aware Weight Quantization) - per-channel scales, group quantization
• GPTQ (Accurate Post-Training Quantization) - Hessian-based, block-wise quantization
• SmoothQuant (migration difficulty from activations to weights) - configurable alpha
• QuantizedTensorData dequantization support
• 9 tests passing

---

Added - TIER 3 Nice to Have Features

Distributed Checkpoint (DCP)

• Files: rtx-distributed/src/dcp.rs
• Async checkpointing (AsyncSaveHandle with progress tracking)
• Sharded state dict save/load (per-rank parallel I/O)
• Resumption from partial checkpoints (configurable min_shards_for_partial)
• World size change handling (shard redistribution)
• Atomic writes with fsync
• 11 tests passing

Context Parallel

• Files: rtx-distributed/src/context_parallel.rs
• Sequence dimension parallelism (SequenceShardInfo with even/uneven splits)
• Ring attention integration (RingAttentionState, ring_attention method)
• Long-context training support (max 128K tokens)
• KV cache distribution across CP ranks
• Async KV prefetch configuration
• 11 tests passing

Autograd Profiler

• Files: rtx-autograd/src/profiler.rs
• Operation timing (ProfiledEvent with duration tracking)
• Memory tracking per op (MemorySnapshot, allocation/deallocation)
• Gradient flow visualization (GradientFlow, DOT export)
• Bottleneck detection (BottleneckInfo, severity analysis)
• Chrome trace export for visualization
• RAII RecordGuard for scoped profiling
• 12 tests passing

SDPA Backend Auto-Selection

• Files: rtx-flash-attention/src/backend_selector.rs
• Automatic FlashAttention vs Math vs Memory-efficient selection
• Hardware detection for optimal backend (HardwareCapabilities)
• Fallback chain management (alternatives ranking)
• Performance-based scoring (sequence length, memory, speedup)
• Auto-tuning with performance history
• Debug mode and preferred backend options
• 12 tests passing

---

[Unreleased] - December 20, 2025

Added - Metal + WASM Benchmarking Suite (Phase 10)

Comprehensive benchmarking infrastructure for comparing RustyTorch++ against PyTorch MPS and WASM alternatives.

Metal Benchmarks (PyTorch MPS Comparison)

• benchmarks/metal/bench_flash_attention.py: Flash Attention benchmark with JSON output

  • BS1-BS256 scenarios, causal/non-causal modes
  • P50/P95/P99 latency statistics, throughput metrics
  • Tested: 0.2-12ms latency, 200M-3500M elements/sec on Apple Silicon

• benchmarks/metal/bench_moe_mps.py: Mixture of Experts benchmark

  • 4-16 expert configurations, top-k routing
  • Routing vs expert compute time breakdown
  • Tested: 1K-70K tokens/sec throughput

• benchmarks/metal/bench_mamba_mps.py: Mamba/SSM benchmark

  • Simplified selective scan for benchmarking
  • Long sequence tests (128-2048 tokens)
  • Tested: 28K-160K tokens/sec throughput

WASM Browser Benchmarks

• benchmarks/wasm/comparison.html: Visual browser benchmark page

  • Side-by-side comparison with ONNX.js and TensorFlow.js
  • Chart.js visualization of latency, throughput, memory
  • System capability detection (SIMD, threading, SharedArrayBuffer)

• benchmarks/wasm/rtx_wasm_bench.js: RustyTorch WASM benchmark module

• benchmarks/wasm/package.json: Node.js dependencies

Infrastructure

• scripts/run_metal_benchmarks.sh: Unified benchmark runner
  • --quick mode for fast validation
  • --rust-only / --python-only for selective runs
  • --wasm for browser benchmark setup
  • JSON output to benchmarks/reports/

Key Results (Apple Silicon M-series)

Feature	PyTorch MPS	Expected RustyTorch	Target Speedup
Flash Attention BS64	1.35ms	<0.7ms	2x
MoE 8 experts	99ms	<50ms	2x
Mamba Seq1024	30ms	<15ms	2x
WASM Inference	N/A	3-5x vs ONNX.js	-

---

Added - WASM Inference Runtime (Phase 9)

WebAssembly inference runtime for deploying ML models in browsers and Node.js, implemented as new rtx-wasm-inference crate.

New Crate: rtx-wasm-inference

• lib.rs: Main WASM exports (~350 lines)

  • WasmInferenceEngine with async model loading and inference
  • InferenceConfig with fast/quality presets
  • InferenceResult with timing and token statistics
  • wasm-bindgen exports for JavaScript interop

• runtime.rs: Environment detection (~220 lines)

  • WasmRuntimeInfo detecting Browser, Node.js, Deno, Web Worker
  • SIMD, threading, and SharedArrayBuffer capability detection
  • Performance timing utilities

• tensor.rs: CPU tensor operations (~450 lines)

  • WasmTensor with matmul, softmax, activations (GELU, SiLU, ReLU)
  • Layer normalization and element-wise operations
  • WasmKvCache for transformer inference

• model.rs: Model loading (~360 lines)

  • WasmModel with forward pass and embedding lookup
  • ModelConfig with tiny/small presets
  • RMS normalization and FFN layers

• tokenizer.rs: Text processing (~200 lines)

  • WasmTokenizer with encode/decode
  • Special token handling (BOS, EOS, PAD, UNK)

• quantization.rs: Model compression (~250 lines)

  • INT8 and INT4 quantization
  • Per-block scaling with configurable block size
  • 4x compression ratio with INT8

Key Features

• Zero Python/CUDA dependencies - pure Rust compiled to WASM
• Browser and Node.js runtime detection
• Optional SIMD and threading support
• Memory-efficient quantized inference
• 17 tests passing

---

Added - Continuous Batching System (Phase 7)

Production-ready continuous batching controller for LLM inference with iteration-level scheduling, implemented in rtx-serving-api crate.

New Files

• continuous_batch.rs: Full continuous batching implementation (~1000 lines)
  • ContinuousBatchingController for managing request lifecycle
  • ContinuousBatchingConfig with batch size, wait time, memory limits
  • BatchRequest with priority, SLA tracking, memory estimation
  • ActiveBatch with dynamic request joining/leaving
  • Priority enum: Low, Normal, High, Critical
  • RequestState: Queued, Processing, Generating, Preempted, Completed, Failed, Cancelled

Key Features

• Iteration-level scheduling: Requests join/leave batches at each decode step
• Preemption support: High-priority requests interrupt lower-priority batches
• Memory-aware batching: Respects KV-cache and GPU memory limits
• SLA enforcement: Deadline-driven scheduling with priority boosting
• Scheduling score: priority_weight + urgency + wait_penalty + preemption_boost
• 7 tests passing

---

Added - Ring Attention for Long Context (Phase 8)

Ring attention implementation for 16M+ token context windows using distributed attention across device rings, implemented in rtx-transformers crate.

New Files

• ring_attention.rs: Ring attention with online softmax (~700 lines)
  • RingAttention for distributed attention computation
  • RingAttentionConfig with num_devices, chunk_size, overlap settings
  • RingTopology for device ring management and rotation
  • SequenceChunk for per-device sequence partitioning
  • OnlineSoftmaxState for numerically stable accumulation
  • RotationBuffer for efficient KV rotation
  • RingAttentionBuilder for fluent configuration

Key Features

• Sequence partitioning across device ring
• Online softmax for numerically stable accumulation
• Causal masking support for autoregressive models
• Overlapped communication with compute
• Gradient checkpointing support
• Memory estimation per device
• 10 tests passing

---

Added - KV-Cache Optimization (Phase 6)

Entropy-guided KV-cache eviction system for 50% memory reduction in LLM inference, implemented in rtx-memory crate.

New Files

• entropy_cache.rs: Core entropy-guided eviction (~520 lines)

  • EntropyMetrics with attention/access entropy, token importance, cumulative attention
  • EntropyTracker for per-block entropy tracking with temporal decay
  • EntropyConfig with configurable thresholds and decay factors
  • EntropyEvictionPolicy enum: PureLowEntropy, AttentionWeighted, EntropyLru, Adaptive
  • Shannon entropy calculation from attention weight distributions
  • Eviction candidate selection with combined scoring (entropy + recency + frequency)

• paged_attention.rs: Page table management (~770 lines)

  • PageTable with logical-to-physical page mapping
  • PagedAttentionConfig with block size, num blocks, CoW support
  • PhysicalPageInfo tracking allocation state, entropy scores, access patterns
  • SequencePages for per-sequence page tracking with ref counting
  • BlockTable for batched attention kernel dispatch
  • Copy-on-Write support for beam search and prefix sharing
  • Memory pressure levels: Low, Medium, High, Critical

• kv_cache.rs: Full KV-cache allocator (~720 lines)

  • KvCacheAllocator integrating entropy tracking and paged attention
  • KvCacheConfig with page size, head dim, num heads, dtype, memory tiers
  • KvCacheHandle for safe sequence cache access
  • KvDataType enum: Float16, BFloat16, Float32, Int8 (quantized)
  • KvMemoryTier enum: Gpu, Cpu, Disk for tiered storage
  • SequenceCache with generation tracking and memory tier placement
  • Entropy-guided eviction with configurable policies
  • LRU fallback when entropy tracking disabled

Integration

• Updated lib.rs with module declarations and re-exports
• Compatible with existing gpu_oom.rs OOM recovery strategies
• 24 tests passing across all three modules

Key Algorithms (MorphKV-inspired)

• Shannon entropy: -sum(p * log(p)) from attention weights
• Normalized entropy for [0,1] scoring
• Combined eviction score: entropy_factor * 0.5 + recency * 0.3 + frequency * 0.2
• Adaptive threshold scaling with memory pressure

---

Added - Sparse Autoencoders (SAE) / Mechanistic Interpretability

Complete SAE module for LLM feature extraction and mechanistic interpretability, integrated into rtx-interpret crate.

New Files

• sae/mod.rs: Core SAE implementation

  • SparseAutoencoder struct with encoder/decoder weights
  • SparsityType enum: L1, TopK, JumpReLU, BatchTopK (Anthropic-style)
  • SAEConfig with expansion factor, sparsity settings, normalization
  • Forward pass with sparsity enforcement and loss computation
  • Dead neuron detection and activation tracking

• sae/hooks.rs: Layer hooking system

  • ActivationHook for capturing intermediate activations
  • LayerHooks for managing multiple hooks across layers
  • BatchActivationCollector for memory-efficient batch collection
  • Forward/backward hook support with gradient capture
  • Streaming mode for large model activation extraction

• sae/training.rs: SAE training utilities

  • SAETrainer with Adam optimizer and learning rate scheduling
  • Dead neuron detection and resampling (configurable interval)
  • Decoder normalization constraint enforcement
  • Warmup, cosine decay, and auxiliary loss computation
  • TrainingHistory for loss/sparsity tracking

• sae/features.rs: Feature analysis tools

  • FeatureAnalyzer for activation statistics
  • FeatureStats: activation frequency, mean/max values, sparsity
  • TopActivation tracking for max-activating examples
  • Co-activation matrix computation
  • FeatureImportance with multiple ranking methods
  • SparsityStats for L0/L1 norms, dead feature detection

Integration

• Updated lib.rs with sae module and re-exports
• Extends existing attribution/neuron analysis in rtx-interpret
• Compatible with rtx-tensor sparse tensor support (SparseCOO, SparseCSR)

---

Added - SlideScope Pathology Demo

Complete GPU-accelerated stain separation demo for digital pathology, integrated into the RustyTorch++ Medical Demos Tauri application.

New Crates

• slidescope-shared: IPC types for Tauri communication

  • SlideMetadata, NmfConfig, NmfResult, JobStatus, TileRequest/Response
  • SlidescopeStatus, SlideFilter, JobProgress, StainType, ImageFormat
  • 19 tests passing

• rtx-slidescope: Core pathology processing crate

  • nmf.rs - CPU-based NMF with multiplicative updates
  • optical_density.rs - RGB to optical density conversion (Beer-Lambert law)
  • stain_vectors.rs - Macenko and Ruifrok stain estimation
  • pyramid.rs - Deep zoom tile pyramid generation
  • 22 tests passing

GPU Abstraction Layer

• gpu/mod.rs - GpuBackend trait with CpuFallbackBackend
• gpu/cuda.rs - CUDA backend using cudarc with custom NMF kernels
• gpu/metal.rs - Metal backend using wgpu with WGSL compute shaders
• gpu_nmf.rs - GPU-accelerated NMF processor
• 32 tests with Metal feature enabled

Server Integration

• slidescope_service.rs - Full service with slide import, tile serving, NMF processing
• Job queue management with progress tracking
• LRU tile cache for efficient deep zoom viewing

Tauri Commands (11 new)

• slidescope_initialize, slidescope_import_slide, slidescope_list_slides
• slidescope_get_slide, slidescope_get_tile, slidescope_queue_processing
• slidescope_job_status, slidescope_get_result, slidescope_status
• slidescope_reset, slidescope_delete_slide

Frontend

• SlidescopeDemo.tsx - React page with dual viewport for slide/stain viewing
• NMF configuration panel (components, iterations, GPU toggle)
• Progress tracking for processing jobs
• Added to demo gallery and App.tsx routing

---

[December 11, 2025]

Added - Apple Metal GPU Backend

• Native Metal Support: Complete Apple Metal GPU backend for Apple Silicon (M1/M2/M3/M4)
  • metal_backend.rs - Device discovery, buffer allocation, command encoding
  • metal_compute.rs - Shader compilation and pipeline management
  • metal_blas/mod.rs - MPS GEMM wrapper for matrix multiplication
  • metal_ops.rs - High-level tensor operation dispatch
• Metal Shading Language (MSL) Kernels:
  • elementwise.metal - Add, sub, mul, div, neg, abs, sqrt, exp, log, fma
  • activations.metal - ReLU, sigmoid, tanh, GELU, SiLU with forward/backward passes
  • fourier.metal - Sin/cos for Fourier features, positional encoding (PINN optimized)
  • reductions.metal - Sum, mean, max, min with threadgroup memory
• Metal Performance Shaders (MPS): Hardware-accelerated GEMM (~7 TFLOPS on M1 Max)
• Unified Memory: Zero-copy CPU/GPU access via MTLStorageModeShared
• Storage Integration: MetalGpu variant in StorageData enum
• Device Detection: Real metal_device_count() using MTLCreateSystemDefaultDevice

Changed

• rtx-tensor/Cargo.toml: Added objc2-metal, objc2-metal-performance-shaders dependencies
• rtx-runtime/Cargo.toml: Added objc2-metal dependencies for Metal backend
• matrix_multiplication.rs: Added Metal matmul dispatch using MPS
• storage/core.rs: Added Metal buffer accessors (get_metal_data(), is_metal_native())
• device.rs: Implemented real Metal device detection for macOS
• lib.rs: Added metal_compute, metal_blas, metal_ops module exports

Dependencies Added (macOS only)

• objc2 = "0.6" - Rust ObjC runtime bindings
• objc2-metal = "0.3" - Metal API bindings
• objc2-metal-performance-shaders = "0.3" - MPS bindings
• objc2-foundation = "0.3" - Foundation framework bindings

---

[Previous] - October 26, 2025

Removed (Honesty & Focus Improvements)

• Quantum computing stub code - Removed non-functional placeholder implementations
  • rtx-transformers/src/revolutionary/quantum_types.rs (162 lines of stubs)
  • Quantum Flash Attention variants (placeholders only)
  • Quantum pattern detection in fusion analyzer
• Neuromorphic computing stub code - Removed aspirational features
  • Neuromorphic Flash Attention variants
  • Neuromorphic test files and benchmarks
  • Spiking neural network stubs
• Misleading documentation - Removed unsubstantiated claims
  • "World's First Quantum-Classical-Neuromorphic" title
  • 30+ quantum/neuromorphic references in README
  • Performance claims without validation (1000x, ∞ advantages)
• Test files for non-existent features
  • neuromorphic_integration_tests.rs
  • neuromorphic_efficiency_benchmark.rs
  • revolutionary_integration_tests.rs (quantum-focused)
  • hybrid_orchestrator_demo.rs (quantum-focused)

Changed

• README.md - Complete honesty overhaul
  • New title: "Production-Ready GPU-Accelerated ML Framework in Pure Rust"
  • Replaced aspirational claims with actual capabilities
  • Added honest status: "Core Infrastructure Stable"
  • Removed quantum/neuromorphic sections, examples, and claims
• Module structure - Cleaned revolutionary module
  • Renamed HybridQuantumClassicalOrchestrator → HybridOrchestrator
  • Removed quantum_types module exports
  • Focused on edge-classical orchestration only
• Flash Attention variants - Simplified to actual implementations
  • Removed quantum/neuromorphic placeholder modules
  • Clear documentation about supported variants

Fixed (October 26, 2025 Session)

• Device::Cpu bugs - Fixed 16 critical bugs from incorrect find-and-replace
  • device.rs: Fixed is_cpu(), cpu() constructor, match arms
  • serialization.rs: Fixed device serialization
  • matrix_multiplication.rs: Fixed CPU device handling
• Compilation errors - Restored 11/11 core crates to compiling state
• Rust 1.93 nightly - Installed latest toolchain
• CUDA integration with cudarc 0.17.3
• Removed all PyTorch dependencies
• Implemented pure Rust tensor operations
• Fixed autograd compilation errors
• Fixed runtime stream issues
• Resolved tensor core operations
• Fixed cuBLAS batched operations
• Fixed cuDNN convolution descriptors
• Fixed cuSparse advanced operations
• Fixed cuSolver context issues
• Resolved character addition errors in tokenization
• Fixed visibility qualifiers in kernel modules
• Fixed cuda_launch_kernel references across modules
• Fixed Python binding issues in rtx-bindings
• Resolved PoolType imports in memory module

Added

• SESSION_PROGRESS_REPORT.md - Detailed progress tracking
• Honest assessment in documentation
• Clear roadmap with realistic timeline
• Comprehensive error handling for CUDA operations
• Improved sparse tensor support
• Enhanced streaming capabilities
• Production monitoring improvements

Migration from Rust 2024 Edition

• Migrated from RustaCUDA to cudarc
• Updated to Rust 2024 edition
• Consolidated compilation status reporting
• Reorganized documentation structure

[Previous Versions]

See docs/archive/legacy/ for historical development phases and changes.