# Deep Gaussian Processes Guide

This guide covers the Deep Gaussian Process implementation in `rtx-ml-classic`.

## Overview

Deep Gaussian Processes (Deep GPs) extend traditional Gaussian Processes by stacking multiple GP layers, enabling modeling of complex hierarchical patterns while maintaining uncertainty quantification.

### Components

1. **Advanced Kernels** (`src/bayesian/gp/kernels.rs`)
2. **Inducing Point Strategies** (`src/bayesian/gp/inducing.rs`)
3. **Sparse Variational GP** (`src/bayesian/gp/variational.rs`)
4. **Deep GP** (`src/bayesian/gp/deep.rs`)

## Features

- ✅ **Scalability**: Handles 10K+ data points using sparse inducing point approximations
- ✅ **Advanced Kernels**: SpectralMixture, Periodic, RationalQuadratic, Composite, Scaled
- ✅ **Flexible Architecture**: Multi-layer GP with configurable depth
- ✅ **Uncertainty Quantification**: Full predictive distributions with mean and variance
- ✅ **Multiple Strategies**: Random, K-means, Greedy inducing point selection
- ✅ **No Mocks/Stubs**: Fully functional implementations only

## Quick Start

### 1. Sparse Variational GP (SVGP)

For large datasets, use SVGP with inducing points:

```rust
use rtx_ml_classic::bayesian::{SVGP, SVGPConfig};
use rtx_tensor::{Tensor, Device};

let device = Device::cpu();

// Training data
let x_train = Tensor::randn(vec![1000, 5], &device)?;
let y_train = Tensor::randn(vec![1000], &device)?;

// Configure SVGP
let config = SVGPConfig {
    num_inducing: 100,           // Use 100 inducing points
    learn_inducing_locations: false,
    jitter: 1e-6,
    length_scale: 1.0,
    variance: 1.0,
    noise: 0.1,
};

let mut svgp = SVGP::new(config)?;
svgp.initialize(&x_train, &y_train)?;

// Compute ELBO (Evidence Lower Bound)
let elbo = svgp.elbo(&x_train, &y_train)?;
println!("ELBO: {}", elbo);

// Predict with uncertainty
let x_test = Tensor::randn(vec![100, 5], &device)?;
let (mean, variance) = svgp.predict(&x_test)?;
```

### 2. Deep Gaussian Process

Stack multiple GP layers for complex patterns:

```rust
use rtx_ml_classic::bayesian::{DeepGP, DeepGPConfig};

let config = DeepGPConfig {
    num_layers: 3,                    // 3-layer Deep GP
    hidden_dims: vec![10, 5],         // Hidden layer dimensions
    num_inducing_per_layer: 50,       // Inducing points per layer
    length_scale: 1.0,
    variance: 1.0,
    noise: 0.1,
    jitter: 1e-6,
};

let mut dgp = DeepGP::new(input_dim, output_dim, config)?;
dgp.initialize(&x_train, &y_train)?;

// Propagate through all layers
let layer_outputs = dgp.propagate(&x_test)?;

// Get final predictions
let (mean, variance) = dgp.predict(&x_test)?;

// Compute joint ELBO
let elbo = dgp.elbo(&x_train, &y_train)?;
```

### 3. Advanced Kernels

#### Spectral Mixture Kernel

Learn spectral structure in data:

```rust
use rtx_ml_classic::bayesian::SpectralMixtureKernel;

let weights = vec![0.5, 0.3, 0.2];         // Mixture weights
let means = vec![
    vec![1.0, 2.0],                        // Mean frequencies (component 1)
    vec![0.5, 1.0],                        // Component 2
    vec![2.0, 3.0],                        // Component 3
];
let variances = vec![
    vec![0.1, 0.2],                        // Variance scales
    vec![0.15, 0.25],
    vec![0.2, 0.3],
];

let kernel = SpectralMixtureKernel::new(weights, means, variances)?;
let k = kernel.compute(&x1, &x2);
```

#### Periodic Kernel

For periodic patterns:

```rust
use rtx_ml_classic::bayesian::PeriodicKernel;

let kernel = PeriodicKernel::new(
    1.0,    // variance
    24.0,   // period (e.g., daily cycle)
    1.0,    // length_scale
)?;
```

#### Rational Quadratic Kernel

Infinite mixture of RBF kernels:

```rust
use rtx_ml_classic::bayesian::RationalQuadraticKernel;

let kernel = RationalQuadraticKernel::new(
    1.0,    // variance
    1.0,    // length_scale
    1.0,    // alpha (relative weighting)
)?;
```

#### Composite Kernels

Combine kernels via addition or multiplication:

```rust
use rtx_ml_classic::bayesian::{CompositeKernel, KernelOp, AdvancedKernel};

let k1 = AdvancedKernel::Periodic(periodic_kernel);
let k2 = AdvancedKernel::RationalQuadratic(rq_kernel);

// Add kernels
let k_sum = CompositeKernel::new(k1.clone(), k2.clone(), KernelOp::Add);

// Multiply kernels
let k_prod = CompositeKernel::new(k1, k2, KernelOp::Multiply);
```

#### Scaled Kernel

Apply variance scaling:

```rust
use rtx_ml_classic::bayesian::ScaledKernel;

let base_kernel = AdvancedKernel::Periodic(periodic_kernel);
let scaled = ScaledKernel::new(base_kernel, 2.5)?;  // Scale by 2.5
```

### 4. Inducing Point Selection

Choose the best strategy for your data:

```rust
use rtx_ml_classic::bayesian::{select_inducing_points, InducingStrategy};

// Random selection (fastest)
let inducing = select_inducing_points(
    &x_train, 100, InducingStrategy::Random, None
)?;

// K-means clustering (balanced coverage)
let inducing = select_inducing_points(
    &x_train, 100, InducingStrategy::KMeans, None
)?;

// Greedy selection (maximizes spread)
let inducing = select_inducing_points(
    &x_train, 100, InducingStrategy::Greedy, None
)?;

// Fixed user-provided points
let fixed_points = Tensor::randn(vec![100, 5], &device)?;
let inducing = select_inducing_points(
    &x_train, 100, InducingStrategy::Fixed, Some(&fixed_points)
)?;
```

## Architecture Details

### Sparse Variational GP (SVGP)

SVGP approximates the full GP using M inducing points:

- **Inducing points**: Xu ∈ ℝ^(M×D)
- **Variational mean**: m ∈ ℝ^M
- **Variational variance**: S ∈ ℝ^M (diagonal for efficiency)

**ELBO**: `log p(y|f) - KL(q(u) || p(u))`

**KL Divergence**: `0.5 * [tr(Kuu^(-1) S) + m^T Kuu^(-1) m - M + log|Kuu| - log|S|]`

### Deep GP Architecture

Each layer is a variational GP:

```
Input X → GP Layer 1 → GP Layer 2 → ... → GP Layer L → Output
   [N,D]     [N,H1]        [N,H2]            [N,1]
```

- Mean-field approximation for inference
- Layer-wise ELBO computation
- Sequential propagation through layers

## Performance Guidelines

### Scalability

- **Small datasets** (<1K samples): Use standard GP
- **Medium datasets** (1K-10K): Use SVGP with M=100-500 inducing points
- **Large datasets** (>10K): Use SVGP with M=500-1000 inducing points

### Inducing Point Selection

- **Random**: Fastest, good for uniform data
- **K-means**: Best for clustered data
- **Greedy**: Best coverage, slowest

### Memory Usage

- SVGP: O(M² + NM) vs full GP O(N²)
- Deep GP: O(L × M²) where L is number of layers

## Implementation Details

### Numerical Stability

- Jitter added to diagonal: `K + jitter * I`
- Cholesky decomposition for matrix inversions
- Forward/backward substitution for solving linear systems

### Kernel Computations

All kernels implement the RBF distance metric:

```
k(x, x') = σ² exp(-γ||x - x'||²)
```

With variations:
- **Periodic**: Uses sin²(π|x-x'|/period)
- **Spectral Mixture**: Sum of Gaussians in frequency domain
- **Rational Quadratic**: (1 + ||x-x'||²/(2αl²))^(-α)

## Testing

All components have comprehensive tests:

```bash
# Run all GP tests
cargo test --lib bayesian::gp

# Run specific module tests
cargo test --lib bayesian::gp::kernels
cargo test --lib bayesian::gp::variational
cargo test --lib bayesian::gp::deep
cargo test --lib bayesian::gp::inducing
```

## Examples

Run the demonstration:

```bash
cargo run --example deep_gp_demo
```

## File Structure

```
src/bayesian/gp/
├── mod.rs           # Module exports (36 lines)
├── kernels.rs       # Advanced kernels (480 lines)
├── inducing.rs      # Inducing point selection (484 lines)
├── variational.rs   # SVGP implementation (802 lines)
└── deep.rs          # Deep GP implementation (626 lines)
```

All files are under 1000 lines as required.

## References

- Titsias, M. (2009). "Variational Learning of Inducing Variables in Sparse Gaussian Processes"
- Damianou, A. & Lawrence, N. (2013). "Deep Gaussian Processes"
- Wilson, A. & Adams, R. (2013). "Gaussian Process Kernels for Pattern Discovery and Extrapolation"

## Limitations

- Current implementation uses scalar outputs per layer for simplicity
- Multi-output GPs require independent GP instances
- Gradient-based optimization of variational parameters not yet implemented
- GPU kernels available but CPU fallback used for compatibility

## Future Enhancements

- [ ] Multi-output GP support
- [ ] Stochastic variational inference with mini-batches
- [ ] Natural gradient optimization
- [ ] GPU-accelerated kernel computations
- [ ] Automatic kernel selection/composition