# RustyTorch++ Technical Context

## Current Status: Rust 2024 Edition Migration Complete

**Last Updated**: 2025-12-16
**Rust Edition**: 2024 (Rust 1.92+ nightly required)
**Build Status**: ✅ `cargo check --workspace` passes (0 errors)

## Core Technologies

### Programming Languages
- **Rust** (nightly 1.92+, 2024 edition)
  - Primary implementation language
  - Required features: const generics, GATs, async traits
  - Compilation targets: x86_64, aarch64, wasm32
  - **Float comparisons**: Uses `total_cmp()` for NaN safety (Rust 2024 requirement)
  
- **C++** (C++20)
  - CUDA kernel implementation
  - High-performance CPU kernels
  - Legacy system integration

- **Python** (3.8+)
  - User-facing API bindings
  - Testing and benchmarking
  - Documentation examples

### GPU Technologies & Targets

#### Primary Target: RTX 5090 (sm_120)
- **Architecture**: NVIDIA Hopper/Ada successor
- **Compiler**: rustg at `/home/osobh/projects/rust/rustg`
- **Kernel Cache**: `target/kernel_cache/` versioned by arch

#### CUDA (12.0+)
```toml
[workspace.dependencies]
# GPU acceleration (cuda-12060 for CUDA 12.6 compatibility)
# f16 feature enables native FP16/BF16 GEMM with Tensor Cores (4-16x faster)
cudarc = { version = "0.18.1", features = ["std", "driver", "runtime", "nvrtc", "cublas", "cublaslt", "nccl", "cudnn", "cusparse", "cusolver", "cufile", "curand", "cuda-12060", "f16"] }
```
- Kernel compilation via rustg + nvcc
- CUDA Graphs for capture/replay
- cuDNN integration for optimized ops
- NCCL for multi-GPU communication
- **cudarc 0.18.1**: Modern CUDA bindings with FP16/BF16 support

#### ROCm (5.0+)
```toml
[dependencies]
rocm-sys = "0.1"
hip = "0.2"  # HIP runtime wrapper
```
- HIP kernel compilation via rustg
- MIOpen for optimized operations
- RCCL for AMD GPU communication

#### Metal (macOS 13+) - Deferred
```toml
[dependencies]
metal = "0.27"
metal-rs = "0.2"
```
- Metal Performance Shaders
- Unified memory architecture
- Darwin apps deferred to later phases

### Build System

#### Cargo Configuration
```toml
[workspace]
members = [
    "crates/rtx-compiler",
    "crates/rtx-runtime", 
    "crates/rtx-kernel",
    "crates/rtx-tensor",
    "crates/rtx-autograd",
    "crates/rtx-ir",
    "crates/rtx-dist",
    "crates/rtx-synth",
    "crates/rtx-serve",
    "crates/rtx-evolve",
    "crates/rtx-profiler",
    "crates/rtx-bench",
    "crates/rtx-governance",
]

[workspace.dependencies]
rustg = { path = "../rust/rustg" }

[profile.release]
lto = "fat"
codegen-units = 1
opt-level = 3
debug = false
strip = true

[profile.bench]
inherits = "release"
debug = true
```

#### Custom Build Scripts
- `build.rs` for GPU kernel compilation
- bindgen for C++ interop
- CUDA/ROCm detection and configuration

### Dependencies

#### Core Crates
```toml
[dependencies]
# Async runtime
tokio = { version = "1.35", features = ["full"] }

# Serialization
serde = { version = "1.0", features = ["derive"] }
bincode = "1.3"

# Numerics
ndarray = "0.15"
num-traits = "0.2"
half = "2.3"  # f16/bf16 support

# Error handling
thiserror = "1.0"
anyhow = "1.0"

# Logging/Tracing
tracing = "0.1"
tracing-subscriber = "0.3"

# Testing
proptest = "1.4"
criterion = "0.5"
```

#### FFI & Bindings
```toml
[dependencies]
# Python bindings
pyo3 = { version = "0.20", features = ["extension-module"] }

# C++ interop
cxx = "1.0"
bindgen = "0.69"

# WebAssembly
wasm-bindgen = "0.2"
web-sys = "0.3"
```

### Development Environment

#### Required Tools
```bash
# Rust toolchain
rustup toolchain install nightly-2024-01-01
rustup component add rustfmt clippy miri

# GPU toolkits
# NVIDIA: CUDA Toolkit 12.0+
# AMD: ROCm 5.0+
# Apple: Xcode 15+ with Metal

# Python environment
python3 -m venv venv
pip install maturin pytest numpy

# Benchmarking
cargo install cargo-flamegraph
cargo install cargo-criterion
```

#### IDE Setup
- **VS Code** with rust-analyzer
- **CLion** with Rust plugin
- **Neovim** with rust-tools.nvim

### Testing Infrastructure

#### Test Frameworks
```rust
// Unit tests
#[cfg(test)]
mod tests {
    use super::*;
    
    #[test]
    fn test_tensor_creation() {
        let tensor = Tensor::zeros([32, 64]);
        assert_eq!(tensor.shape(), &[32, 64]);
    }
}

// Property-based tests
#[cfg(test)]
mod prop_tests {
    use proptest::prelude::*;
    
    proptest! {
        #[test]
        fn test_matmul_associative(
            a in tensor_strategy(),
            b in tensor_strategy(),
            c in tensor_strategy()
        ) {
            // Test (A * B) * C == A * (B * C)
        }
    }
}

// Benchmarks
#[bench]
fn bench_matmul(b: &mut Bencher) {
    let a = Tensor::randn([1024, 1024]);
    let b = Tensor::randn([1024, 1024]);
    b.iter(|| a.matmul(&b));
}
```

#### CI/CD Pipeline
```yaml
# .github/workflows/ci.yml
name: CI
on: [push, pull_request]

jobs:
  test:
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        rust: [stable, nightly]
    
  bench:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cargo bench
      
  gpu-test:
    runs-on: [self-hosted, gpu]
    steps:
      - run: cargo test --features cuda
```

### Deployment Targets

#### Inference Runtime
- **Native Binary**: Standalone executable
- **Dynamic Library**: `.so/.dll/.dylib`
- **Python Package**: Wheel distribution
- **WebAssembly**: Browser/Node.js runtime
- **Container**: Docker with GPU support

#### Package Distribution
```toml
# Cargo.toml
[package]
name = "rustytorch"
version = "0.1.0"
edition = "2021"
license = "Apache-2.0"
repository = "https://github.com/rustytorch/rustytorch"

[package.metadata.docs.rs]
all-features = true
rustdoc-args = ["--cfg", "docsrs"]
```

### Performance Profiling

#### Tools
- **perf**: Linux system profiler
- **flamegraph**: Visualization of hot paths
- **NVIDIA Nsight**: GPU kernel profiling
- **AMD ROCProfiler**: ROCm profiling
- **Intel VTune**: CPU optimization

#### Metrics Collection
```rust
use prometheus::{Counter, Histogram, register_counter, register_histogram};

lazy_static! {
    static ref TENSOR_OPS: Counter = register_counter!(
        "rustytorch_tensor_ops_total",
        "Total tensor operations"
    ).unwrap();
    
    static ref OP_LATENCY: Histogram = register_histogram!(
        "rustytorch_op_latency_seconds",
        "Operation latency in seconds"
    ).unwrap();
}
```

### Security Considerations

#### Memory Safety
- MIRI verification for unsafe code
- Address sanitizer in debug builds
- Fuzzing for input validation

#### Supply Chain
```toml
# Cargo.toml
[dependencies]
# Audit dependencies
cargo-audit = "0.18"

# Vendored dependencies for reproducible builds
cargo-vendor = "0.12"
```

### Documentation

#### Generation
```bash
# API documentation
cargo doc --all-features --no-deps

# Book generation
mdbook build docs/

# Python API docs
pdoc3 --html python/rustytorch
```

#### Hosting
- API docs: https://docs.rs/rustytorch
- User guide: https://rustytorch.ai/guide
- Examples: https://github.com/rustytorch/examples

### Development Workflow

#### Pre-commit Hooks
```bash
#!/bin/bash
# .git/hooks/pre-commit

# Format code
cargo fmt --all -- --check

# Lint
cargo clippy --all-targets --all-features -- -D warnings

# Test
cargo test --lib
```

#### Version Control
```gitignore
# .gitignore
target/
*.rs.bk
Cargo.lock
*.pyc
__pycache__/
.venv/
.idea/
.vscode/
*.swp
```

### External Integrations

#### Model Formats
- **ONNX**: Import/export support (Phase 7)
- **SafeTensors**: Fast tensor serialization
- **PyTorch**: `.pt` file compatibility (Phase 7)
- **DLPack**: Zero-copy interop (Phase 2)

#### Cluster Management  
- **Stratoswarm**: Primary orchestration at `/home/osobh/projects/stratoswarm`
  - Deployment manifests in `deploy/stratoswarm/`
  - Blue/green + canary rollouts (Phase 5)
  - Multi-region support (Phase 9)
- **Kubernetes**: Via Stratoswarm integration
- **SLURM**: HPC cluster integration (future)
- **Ray**: Distributed compute framework (future)

### Technical Constraints

#### Platform Limitations
- **Linux**: Full feature support
- **macOS**: Metal backend only
- **Windows**: Limited CUDA support
- **WebAssembly**: CPU-only, no threading

#### Performance Targets (Phase-Specific)

#### Phase 1 Targets
- CUDA Graph capture hit-rate: ≥ 70%
- Memory fragmentation: < 15%
- Step-time improvement: ≥ 20% vs eager baseline
- Determinism: fp32 ≤ 1e-6, bf16/fp16 ≤ 1e-3

#### Phase 2 Targets  
- Graph capture hit-rate: ≥ 75%
- End-to-end step time: ≥ 20% reduction vs Phase 1
- Autograd correctness: Parity with fp32 goldens

#### Phase 3 Targets
- Single-node scaling: ≥ 0.8x efficiency (1→8 GPUs)
- Multi-node scaling: ≥ 0.7x on IB/NVLink
- FSDP memory reduction: ≥ 40% vs DP

#### Phase 4 Targets
- Auto-kernel speedup: ≥ 20-40% step-time reduction
- Inference speedup: ≥ 1.5x tokens/sec
- Kernel cache hit-rate: ≥ 80%

#### Phase 5 Targets
- Inference latency P99: < 150ms/token on RTX 5090
- KV cache hit-rate: ≥ 85%
- Continuous batching utilization: ≥ 30% improvement

#### Resource Limits
- Max tensor size: 2^63 elements
- Max dimensions: 32
- Max GPU memory: Device-dependent
- Max nodes: 10,000 (Stratoswarm limit)

## Workspace Configuration

### Excluded Crates (December 2024)

| Crate | Reason | Future Work |
|-------|--------|-------------|
| `integration_tests` | Tests reference unimplemented APIs (400+ errors) | Implement APIs when needed |
| `rtx-flash-metal-attention` | macOS/Metal only - not available on Linux | Works on macOS systems |
| `demos/ui/src-tauri` | Different MSRV and dependency requirements | Separate build process |

### Dependency Management

#### nom Version Consolidation
- **Removed**: nom 3.2.1 (legacy)
- **Active versions**: nom 7.1.3, nom 8.0.0
- **Method**: Removed unused `npy` dependency from rtx-vision-advanced

#### Float Comparison Safety (Rust 2024)
```rust
// Before (Rust 2021 - panics on NaN)
values.sort_by(|a, b| a.partial_cmp(b).unwrap());

// After (Rust 2024 - NaN-safe)
values.sort_by(|a, b| a.total_cmp(b));
```

**Pattern applied to**: 200+ files across the workspace

### Build Commands

```bash
# Standard workspace build
cargo check --workspace

# Build specific crate
cargo build -p rtx-tensor

# Full release build
cargo build --release --workspace

# Run with CUDA features
cargo build --workspace --features cuda
```

---

*Technical Context Last Updated: 2025-12-16*