Initial commit

docs/book/src/troubleshooting/memory.md

@@ -0,0 +1,249 @@
+# Memory Problems
+
+Diagnose and fix memory issues.
+
+## Out of Memory (OOM)
+
+### GPU OOM During Training
+
+```
+CUDA error: out of memory
+Tried to allocate 2.00 GB
+```
+
+**Solutions:**
+
+1. **Reduce batch size:**
+   ```rust
+   let batch_size = batch_size / 2;
+   ```
+
+2. **Enable gradient checkpointing:**
+   ```rust
+   use rtx_autograd::CheckpointManager;
+   let checkpoint = CheckpointManager::sqrt_n();
+
+   let hidden = checkpoint.checkpoint(|| {
+       layer.forward(&x)
+   })?;
+   ```
+
+3. **Use mixed precision:**
+   ```rust
+   let model = model.to_dtype(DType::F16)?;
+   ```
+
+4. **Clear unused memory:**
+   ```rust
+   use rtx_tensor::clear_pool;
+   clear_pool();
+   ```
+
+5. **Process in chunks:**
+   ```rust
+   let mut outputs = vec![];
+   for chunk in data.chunks(safe_batch_size) {
+       outputs.push(model.forward(&chunk)?);
+       clear_pool();  // Free between chunks
+   }
+   ```
+
+### GPU OOM During Inference
+
+**Solutions:**
+
+1. **Disable gradient tracking:**
+   ```rust
+   let output = no_grad(|| model.forward(&input))?;
+   ```
+
+2. **Use streaming inference:**
+   ```rust
+   for token in model.generate_stream(&input)? {
+       // Process one token at a time
+   }
+   ```
+
+## Memory Leaks
+
+### Symptoms
+
+- Memory usage grows over time
+- Eventually crashes with OOM
+
+### Finding Leaks
+
+```rust
+use rtx_memory::AllocationTracker;
+
+AllocationTracker::enable();
+
+for epoch in 0..100 {
+    train_epoch(&model, &data)?;
+
+    let stats = AllocationTracker::snapshot();
+    println!("Epoch {}: {} MB", epoch, stats.current_mb);
+
+    if stats.current_mb > threshold {
+        println!("Leak detected!");
+        AllocationTracker::dump_live();
+    }
+}
+```
+
+### Common Leak Causes
+
+1. **Accumulating tensors in loops:**
+   ```rust
+   // Bad - accumulates history
+   let mut history = vec![];
+   for batch in data {
+       let output = model.forward(&batch)?;
+       history.push(output);  // Grows forever!
+   }
+
+   // Good - process and discard
+   for batch in data {
+       let output = model.forward(&batch)?;
+       process(&output)?;
+       // output dropped here
+   }
+   ```
+
+2. **Reference cycles with Arc:**
+   ```rust
+   // Use Weak references to break cycles
+   use std::sync::Weak;
+   ```
+
+3. **Holding autograd graph:**
+   ```rust
+   // Clear tape after backward
+   clear_tape();
+   ```
+
+## Memory Fragmentation
+
+### Symptoms
+
+- Free memory but allocation fails
+- Slow allocations
+
+### Solutions
+
+1. **Use tensor pooling:**
+   ```rust
+   use rtx_tensor::TensorPool;
+
+   let buffer = TensorPool::get(shape, &device)?;
+   ```
+
+2. **Pre-allocate fixed-size buffers:**
+   ```rust
+   let mut output = Tensor::zeros([max_batch, hidden], &device)?;
+
+   for batch in batches {
+       model.forward_out(&batch, &mut output)?;
+   }
+   ```
+
+3. **Periodic defragmentation:**
+   ```rust
+   if pool_stats().fragmentation > 0.5 {
+       clear_pool();
+   }
+   ```
+
+## CPU Memory Issues
+
+### High RAM Usage
+
+**Solutions:**
+
+1. **Use pinned memory for transfers:**
+   ```rust
+   let pinned = PinnedBuffer::new(size)?;
+   ```
+
+2. **Stream data instead of loading all:**
+   ```rust
+   for batch in DataLoader::new(&dataset).batch_size(64) {
+       // Load one batch at a time
+   }
+   ```
+
+3. **Memory-map large files:**
+   ```rust
+   use memmap2::Mmap;
+   let mmap = unsafe { Mmap::map(&file)? };
+   ```
+
+## Debugging Memory
+
+### Memory Snapshot
+
+```rust
+use rtx_memory::Snapshot;
+
+let before = Snapshot::capture(&device)?;
+
+// Run code
+model.forward(&input)?;
+
+let after = Snapshot::capture(&device)?;
+let diff = after.diff(&before);
+
+println!("Memory delta: {} MB", diff.total_mb);
+for alloc in diff.new_allocations() {
+    println!("  New: {} at {}", alloc.size_mb, alloc.location);
+}
+```
+
+### CUDA Memory Debug
+
+```bash
+# Enable CUDA memory debugging
+export CUDA_LAUNCH_BLOCKING=1
+export PYTORCH_NO_CUDA_MEMORY_CACHING=1  # Disable caching for debugging
+```
+
+### Valgrind (CPU)
+
+```bash
+valgrind --leak-check=full ./target/release/my-app
+```
+
+## Memory Estimation
+
+### Estimate Model Memory
+
+```rust
+fn estimate_model_memory(model: &Model) -> usize {
+    let mut total = 0;
+    for param in model.parameters() {
+        total += param.numel() * param.dtype().size_bytes();
+    }
+    total
+}
+
+// Account for gradients during training
+let training_memory = estimate_model_memory(&model) * 2;
+
+// Account for activations (rough estimate)
+let batch_memory = batch_size * hidden_size * num_layers * 4;
+```
+
+### Memory Budget
+
+```rust
+let available = device.free_memory()?;
+let model_mem = estimate_model_memory(&model);
+let safe_batch_mem = available - model_mem - safety_margin;
+let max_batch_size = safe_batch_mem / mem_per_sample;
+```
+
+## Next Steps
+
+- [GPU Issues](./gpu-issues.md) - GPU-specific problems
+- [Performance Memory](../performance/memory.md) - Memory optimization
+- [FAQ](./faq.md) - Common questions