# RustyTorch++ Changelog ## [December 28, 2025] ### Added - TIER 2 Feature Completeness #### FlashAttention CPU/Metal Backward - **Files**: `rtx-flash-attention/src/core.rs`, `src/lib.rs` - CPU backward pass (`naive_attention_backward`) - Metal backward pass (trait impl to kernel) - 30 gradient tests passing #### Nested vmap Support - **Files**: `rtx-autograd/src/vmap.rs` - `BatchDimStack` for tracking batch dimensions through nested calls - `MultiBatchedVariable` for dimension collapsing/expansion - `VmapLevelGuard` RAII guards for level tracking - 12 vmap tests passing #### Hessian-Vector Product (hvp) - **Files**: `rtx-autograd/src/func.rs` - `hvp(f, primals, tangents)` with forward-over-reverse mode - `hvp_finite_diff` for numerical validation - `vhp` (vector-Hessian product) - Fixed f32 precision issues (use f64 intermediate, larger epsilon 1e-3) - 7 hvp tests passing #### Dynamic Shape Guards - **Files**: `rtx-synthesis/src/aot_impl/shape_guards.rs`, `src/aot.rs` - Shape guard generation (`generate_guards_from_operations`, `generate_symbolic_guards`) - Recompilation triggers (`GuardCheckResult`, `GuardFailure`) - Shape dimension types: Concrete, Symbolic, Bounded, Dynamic - Shape bucketing for cache efficiency (`ShapeGuardManager`, `ShapeSignature`) - 20 tests passing #### Advanced Quantization (AWQ, GPTQ, SmoothQuant) - **Files**: `rtx-compress/src/quantization/advanced.rs` - AWQ (Activation-aware Weight Quantization) - per-channel scales, group quantization - GPTQ (Accurate Post-Training Quantization) - Hessian-based, block-wise quantization - SmoothQuant (migration difficulty from activations to weights) - configurable alpha - QuantizedTensorData dequantization support - 9 tests passing --- ### Added - TIER 3 Nice to Have Features #### Distributed Checkpoint (DCP) - **Files**: `rtx-distributed/src/dcp.rs` - Async checkpointing (`AsyncSaveHandle` with progress tracking) - Sharded state dict save/load (per-rank parallel I/O) - Resumption from partial checkpoints (configurable `min_shards_for_partial`) - World size change handling (shard redistribution) - Atomic writes with fsync - 11 tests passing #### Context Parallel - **Files**: `rtx-distributed/src/context_parallel.rs` - Sequence dimension parallelism (`SequenceShardInfo` with even/uneven splits) - Ring attention integration (`RingAttentionState`, `ring_attention` method) - Long-context training support (max 128K tokens) - KV cache distribution across CP ranks - Async KV prefetch configuration - 11 tests passing #### Autograd Profiler - **Files**: `rtx-autograd/src/profiler.rs` - Operation timing (`ProfiledEvent` with duration tracking) - Memory tracking per op (`MemorySnapshot`, allocation/deallocation) - Gradient flow visualization (`GradientFlow`, DOT export) - Bottleneck detection (`BottleneckInfo`, severity analysis) - Chrome trace export for visualization - RAII `RecordGuard` for scoped profiling - 12 tests passing #### SDPA Backend Auto-Selection - **Files**: `rtx-flash-attention/src/backend_selector.rs` - Automatic FlashAttention vs Math vs Memory-efficient selection - Hardware detection for optimal backend (`HardwareCapabilities`) - Fallback chain management (alternatives ranking) - Performance-based scoring (sequence length, memory, speedup) - Auto-tuning with performance history - Debug mode and preferred backend options - 12 tests passing --- ## [Unreleased] - December 20, 2025 ### Added - Metal + WASM Benchmarking Suite (Phase 10) Comprehensive benchmarking infrastructure for comparing RustyTorch++ against PyTorch MPS and WASM alternatives. #### Metal Benchmarks (PyTorch MPS Comparison) - **benchmarks/metal/bench_flash_attention.py**: Flash Attention benchmark with JSON output - BS1-BS256 scenarios, causal/non-causal modes - P50/P95/P99 latency statistics, throughput metrics - Tested: 0.2-12ms latency, 200M-3500M elements/sec on Apple Silicon - **benchmarks/metal/bench_moe_mps.py**: Mixture of Experts benchmark - 4-16 expert configurations, top-k routing - Routing vs expert compute time breakdown - Tested: 1K-70K tokens/sec throughput - **benchmarks/metal/bench_mamba_mps.py**: Mamba/SSM benchmark - Simplified selective scan for benchmarking - Long sequence tests (128-2048 tokens) - Tested: 28K-160K tokens/sec throughput #### WASM Browser Benchmarks - **benchmarks/wasm/comparison.html**: Visual browser benchmark page - Side-by-side comparison with ONNX.js and TensorFlow.js - Chart.js visualization of latency, throughput, memory - System capability detection (SIMD, threading, SharedArrayBuffer) - **benchmarks/wasm/rtx_wasm_bench.js**: RustyTorch WASM benchmark module - **benchmarks/wasm/package.json**: Node.js dependencies #### Infrastructure - **scripts/run_metal_benchmarks.sh**: Unified benchmark runner - `--quick` mode for fast validation - `--rust-only` / `--python-only` for selective runs - `--wasm` for browser benchmark setup - JSON output to benchmarks/reports/ #### Key Results (Apple Silicon M-series) | Feature | PyTorch MPS | Expected RustyTorch | Target Speedup | |---------|------------|---------------------|----------------| | Flash Attention BS64 | 1.35ms | <0.7ms | 2x | | MoE 8 experts | 99ms | <50ms | 2x | | Mamba Seq1024 | 30ms | <15ms | 2x | | WASM Inference | N/A | 3-5x vs ONNX.js | - | --- ### Added - WASM Inference Runtime (Phase 9) WebAssembly inference runtime for deploying ML models in browsers and Node.js, implemented as new rtx-wasm-inference crate. #### New Crate: rtx-wasm-inference - **lib.rs**: Main WASM exports (~350 lines) - `WasmInferenceEngine` with async model loading and inference - `InferenceConfig` with fast/quality presets - `InferenceResult` with timing and token statistics - wasm-bindgen exports for JavaScript interop - **runtime.rs**: Environment detection (~220 lines) - `WasmRuntimeInfo` detecting Browser, Node.js, Deno, Web Worker - SIMD, threading, and SharedArrayBuffer capability detection - Performance timing utilities - **tensor.rs**: CPU tensor operations (~450 lines) - `WasmTensor` with matmul, softmax, activations (GELU, SiLU, ReLU) - Layer normalization and element-wise operations - `WasmKvCache` for transformer inference - **model.rs**: Model loading (~360 lines) - `WasmModel` with forward pass and embedding lookup - `ModelConfig` with tiny/small presets - RMS normalization and FFN layers - **tokenizer.rs**: Text processing (~200 lines) - `WasmTokenizer` with encode/decode - Special token handling (BOS, EOS, PAD, UNK) - **quantization.rs**: Model compression (~250 lines) - INT8 and INT4 quantization - Per-block scaling with configurable block size - 4x compression ratio with INT8 #### Key Features - Zero Python/CUDA dependencies - pure Rust compiled to WASM - Browser and Node.js runtime detection - Optional SIMD and threading support - Memory-efficient quantized inference - 17 tests passing --- ### Added - Continuous Batching System (Phase 7) Production-ready continuous batching controller for LLM inference with iteration-level scheduling, implemented in rtx-serving-api crate. #### New Files - **continuous_batch.rs**: Full continuous batching implementation (~1000 lines) - `ContinuousBatchingController` for managing request lifecycle - `ContinuousBatchingConfig` with batch size, wait time, memory limits - `BatchRequest` with priority, SLA tracking, memory estimation - `ActiveBatch` with dynamic request joining/leaving - `Priority` enum: Low, Normal, High, Critical - `RequestState`: Queued, Processing, Generating, Preempted, Completed, Failed, Cancelled #### Key Features - Iteration-level scheduling: Requests join/leave batches at each decode step - Preemption support: High-priority requests interrupt lower-priority batches - Memory-aware batching: Respects KV-cache and GPU memory limits - SLA enforcement: Deadline-driven scheduling with priority boosting - Scheduling score: `priority_weight + urgency + wait_penalty + preemption_boost` - 7 tests passing --- ### Added - Ring Attention for Long Context (Phase 8) Ring attention implementation for 16M+ token context windows using distributed attention across device rings, implemented in rtx-transformers crate. #### New Files - **ring_attention.rs**: Ring attention with online softmax (~700 lines) - `RingAttention` for distributed attention computation - `RingAttentionConfig` with num_devices, chunk_size, overlap settings - `RingTopology` for device ring management and rotation - `SequenceChunk` for per-device sequence partitioning - `OnlineSoftmaxState` for numerically stable accumulation - `RotationBuffer` for efficient KV rotation - `RingAttentionBuilder` for fluent configuration #### Key Features - Sequence partitioning across device ring - Online softmax for numerically stable accumulation - Causal masking support for autoregressive models - Overlapped communication with compute - Gradient checkpointing support - Memory estimation per device - 10 tests passing --- ### Added - KV-Cache Optimization (Phase 6) Entropy-guided KV-cache eviction system for 50% memory reduction in LLM inference, implemented in rtx-memory crate. #### New Files - **entropy_cache.rs**: Core entropy-guided eviction (~520 lines) - `EntropyMetrics` with attention/access entropy, token importance, cumulative attention - `EntropyTracker` for per-block entropy tracking with temporal decay - `EntropyConfig` with configurable thresholds and decay factors - `EntropyEvictionPolicy` enum: PureLowEntropy, AttentionWeighted, EntropyLru, Adaptive - Shannon entropy calculation from attention weight distributions - Eviction candidate selection with combined scoring (entropy + recency + frequency) - **paged_attention.rs**: Page table management (~770 lines) - `PageTable` with logical-to-physical page mapping - `PagedAttentionConfig` with block size, num blocks, CoW support - `PhysicalPageInfo` tracking allocation state, entropy scores, access patterns - `SequencePages` for per-sequence page tracking with ref counting - `BlockTable` for batched attention kernel dispatch - Copy-on-Write support for beam search and prefix sharing - Memory pressure levels: Low, Medium, High, Critical - **kv_cache.rs**: Full KV-cache allocator (~720 lines) - `KvCacheAllocator` integrating entropy tracking and paged attention - `KvCacheConfig` with page size, head dim, num heads, dtype, memory tiers - `KvCacheHandle` for safe sequence cache access - `KvDataType` enum: Float16, BFloat16, Float32, Int8 (quantized) - `KvMemoryTier` enum: Gpu, Cpu, Disk for tiered storage - `SequenceCache` with generation tracking and memory tier placement - Entropy-guided eviction with configurable policies - LRU fallback when entropy tracking disabled #### Integration - Updated `lib.rs` with module declarations and re-exports - Compatible with existing `gpu_oom.rs` OOM recovery strategies - 24 tests passing across all three modules #### Key Algorithms (MorphKV-inspired) - Shannon entropy: `-sum(p * log(p))` from attention weights - Normalized entropy for [0,1] scoring - Combined eviction score: `entropy_factor * 0.5 + recency * 0.3 + frequency * 0.2` - Adaptive threshold scaling with memory pressure --- ### Added - Sparse Autoencoders (SAE) / Mechanistic Interpretability Complete SAE module for LLM feature extraction and mechanistic interpretability, integrated into rtx-interpret crate. #### New Files - **sae/mod.rs**: Core SAE implementation - `SparseAutoencoder` struct with encoder/decoder weights - `SparsityType` enum: L1, TopK, JumpReLU, BatchTopK (Anthropic-style) - `SAEConfig` with expansion factor, sparsity settings, normalization - Forward pass with sparsity enforcement and loss computation - Dead neuron detection and activation tracking - **sae/hooks.rs**: Layer hooking system - `ActivationHook` for capturing intermediate activations - `LayerHooks` for managing multiple hooks across layers - `BatchActivationCollector` for memory-efficient batch collection - Forward/backward hook support with gradient capture - Streaming mode for large model activation extraction - **sae/training.rs**: SAE training utilities - `SAETrainer` with Adam optimizer and learning rate scheduling - Dead neuron detection and resampling (configurable interval) - Decoder normalization constraint enforcement - Warmup, cosine decay, and auxiliary loss computation - `TrainingHistory` for loss/sparsity tracking - **sae/features.rs**: Feature analysis tools - `FeatureAnalyzer` for activation statistics - `FeatureStats`: activation frequency, mean/max values, sparsity - `TopActivation` tracking for max-activating examples - Co-activation matrix computation - `FeatureImportance` with multiple ranking methods - `SparsityStats` for L0/L1 norms, dead feature detection #### Integration - Updated `lib.rs` with sae module and re-exports - Extends existing attribution/neuron analysis in rtx-interpret - Compatible with rtx-tensor sparse tensor support (SparseCOO, SparseCSR) --- ### Added - SlideScope Pathology Demo Complete GPU-accelerated stain separation demo for digital pathology, integrated into the RustyTorch++ Medical Demos Tauri application. #### New Crates - **slidescope-shared**: IPC types for Tauri communication - `SlideMetadata`, `NmfConfig`, `NmfResult`, `JobStatus`, `TileRequest/Response` - `SlidescopeStatus`, `SlideFilter`, `JobProgress`, `StainType`, `ImageFormat` - 19 tests passing - **rtx-slidescope**: Core pathology processing crate - `nmf.rs` - CPU-based NMF with multiplicative updates - `optical_density.rs` - RGB to optical density conversion (Beer-Lambert law) - `stain_vectors.rs` - Macenko and Ruifrok stain estimation - `pyramid.rs` - Deep zoom tile pyramid generation - 22 tests passing #### GPU Abstraction Layer - `gpu/mod.rs` - `GpuBackend` trait with `CpuFallbackBackend` - `gpu/cuda.rs` - CUDA backend using cudarc with custom NMF kernels - `gpu/metal.rs` - Metal backend using wgpu with WGSL compute shaders - `gpu_nmf.rs` - GPU-accelerated NMF processor - 32 tests with Metal feature enabled #### Server Integration - `slidescope_service.rs` - Full service with slide import, tile serving, NMF processing - Job queue management with progress tracking - LRU tile cache for efficient deep zoom viewing #### Tauri Commands (11 new) - `slidescope_initialize`, `slidescope_import_slide`, `slidescope_list_slides` - `slidescope_get_slide`, `slidescope_get_tile`, `slidescope_queue_processing` - `slidescope_job_status`, `slidescope_get_result`, `slidescope_status` - `slidescope_reset`, `slidescope_delete_slide` #### Frontend - `SlidescopeDemo.tsx` - React page with dual viewport for slide/stain viewing - NMF configuration panel (components, iterations, GPU toggle) - Progress tracking for processing jobs - Added to demo gallery and App.tsx routing --- ## [December 11, 2025] ### Added - Apple Metal GPU Backend - **Native Metal Support**: Complete Apple Metal GPU backend for Apple Silicon (M1/M2/M3/M4) - `metal_backend.rs` - Device discovery, buffer allocation, command encoding - `metal_compute.rs` - Shader compilation and pipeline management - `metal_blas/mod.rs` - MPS GEMM wrapper for matrix multiplication - `metal_ops.rs` - High-level tensor operation dispatch - **Metal Shading Language (MSL) Kernels**: - `elementwise.metal` - Add, sub, mul, div, neg, abs, sqrt, exp, log, fma - `activations.metal` - ReLU, sigmoid, tanh, GELU, SiLU with forward/backward passes - `fourier.metal` - Sin/cos for Fourier features, positional encoding (PINN optimized) - `reductions.metal` - Sum, mean, max, min with threadgroup memory - **Metal Performance Shaders (MPS)**: Hardware-accelerated GEMM (~7 TFLOPS on M1 Max) - **Unified Memory**: Zero-copy CPU/GPU access via `MTLStorageModeShared` - **Storage Integration**: `MetalGpu` variant in `StorageData` enum - **Device Detection**: Real `metal_device_count()` using `MTLCreateSystemDefaultDevice` ### Changed - **rtx-tensor/Cargo.toml**: Added objc2-metal, objc2-metal-performance-shaders dependencies - **rtx-runtime/Cargo.toml**: Added objc2-metal dependencies for Metal backend - **matrix_multiplication.rs**: Added Metal matmul dispatch using MPS - **storage/core.rs**: Added Metal buffer accessors (`get_metal_data()`, `is_metal_native()`) - **device.rs**: Implemented real Metal device detection for macOS - **lib.rs**: Added metal_compute, metal_blas, metal_ops module exports ### Dependencies Added (macOS only) - `objc2 = "0.6"` - Rust ObjC runtime bindings - `objc2-metal = "0.3"` - Metal API bindings - `objc2-metal-performance-shaders = "0.3"` - MPS bindings - `objc2-foundation = "0.3"` - Foundation framework bindings --- ## [Previous] - October 26, 2025 ### Removed (Honesty & Focus Improvements) - **Quantum computing stub code** - Removed non-functional placeholder implementations - `rtx-transformers/src/revolutionary/quantum_types.rs` (162 lines of stubs) - Quantum Flash Attention variants (placeholders only) - Quantum pattern detection in fusion analyzer - **Neuromorphic computing stub code** - Removed aspirational features - Neuromorphic Flash Attention variants - Neuromorphic test files and benchmarks - Spiking neural network stubs - **Misleading documentation** - Removed unsubstantiated claims - "World's First Quantum-Classical-Neuromorphic" title - 30+ quantum/neuromorphic references in README - Performance claims without validation (1000x, ∞ advantages) - **Test files for non-existent features** - `neuromorphic_integration_tests.rs` - `neuromorphic_efficiency_benchmark.rs` - `revolutionary_integration_tests.rs` (quantum-focused) - `hybrid_orchestrator_demo.rs` (quantum-focused) ### Changed - **README.md** - Complete honesty overhaul - New title: "Production-Ready GPU-Accelerated ML Framework in Pure Rust" - Replaced aspirational claims with actual capabilities - Added honest status: "Core Infrastructure Stable" - Removed quantum/neuromorphic sections, examples, and claims - **Module structure** - Cleaned revolutionary module - Renamed `HybridQuantumClassicalOrchestrator` → `HybridOrchestrator` - Removed quantum_types module exports - Focused on edge-classical orchestration only - **Flash Attention variants** - Simplified to actual implementations - Removed quantum/neuromorphic placeholder modules - Clear documentation about supported variants ### Fixed (October 26, 2025 Session) - **Device::Cpu bugs** - Fixed 16 critical bugs from incorrect find-and-replace - `device.rs`: Fixed is_cpu(), cpu() constructor, match arms - `serialization.rs`: Fixed device serialization - `matrix_multiplication.rs`: Fixed CPU device handling - **Compilation errors** - Restored 11/11 core crates to compiling state - **Rust 1.93 nightly** - Installed latest toolchain - CUDA integration with cudarc 0.17.3 - Removed all PyTorch dependencies - Implemented pure Rust tensor operations - Fixed autograd compilation errors - Fixed runtime stream issues - Resolved tensor core operations - Fixed cuBLAS batched operations - Fixed cuDNN convolution descriptors - Fixed cuSparse advanced operations - Fixed cuSolver context issues - Resolved character addition errors in tokenization - Fixed visibility qualifiers in kernel modules - Fixed cuda_launch_kernel references across modules - Fixed Python binding issues in rtx-bindings - Resolved PoolType imports in memory module ### Added - **SESSION_PROGRESS_REPORT.md** - Detailed progress tracking - **Honest assessment** in documentation - Clear roadmap with realistic timeline - Comprehensive error handling for CUDA operations - Improved sparse tensor support - Enhanced streaming capabilities - Production monitoring improvements ### Migration from Rust 2024 Edition - Migrated from RustaCUDA to cudarc - Updated to Rust 2024 edition - Consolidated compilation status reporting - Reorganized documentation structure ## [Previous Versions] See `docs/archive/legacy/` for historical development phases and changes.