rustytorch/integration_tests/README.md

# RustyTorch++ Integration Tests

Comprehensive end-to-end integration testing framework for the entire RustyTorch++ platform using strict Test-Driven Development (TDD) methodology. These tests validate real production workloads and scenarios without mocked dependencies.

## Overview

This integration test suite provides comprehensive validation of:
- Complete ML pipelines (data → training → evaluation → deployment)
- Cross-component integration between all RustyTorch++ modules
- Production deployment scenarios with containerization
- Performance characteristics with SLA validation
- Real-world use cases (BERT, GPT, CLIP, etc.)
- Multi-tenant system capabilities
- Disaster recovery and fault tolerance
- Security and compliance features

## Test Categories

### 1. ML Pipeline Integration Tests
Tests the complete machine learning pipeline from data loading through deployment.

**Components Tested:**
- Data loading and validation (`rtx-data-validation`, `rtx-etl`)
- Model training (`rtx-transformers`, `rtx-autograd`, `rtx-distributed`)
- Model evaluation (`rtx-eval`)
- Model deployment (`rtx-serving-api`, `rtx-inference`)
- Model compression (`rtx-compress`)

**Key Scenarios:**
- End-to-end classification pipeline
- Distributed multi-GPU training
- Model compression and deployment
- Performance optimization validation

### 2. Cross-Component Integration Tests
Validates integration between all major RustyTorch++ components.

**Integration Patterns:**
- Tensor ↔ Autograd (gradient computation)
- Autograd ↔ Transformers (training integration)
- Transformers ↔ Serving (inference pipeline)
- Memory ↔ Runtime (resource management)
- Distributed ↔ Training (multi-GPU coordination)

**Key Scenarios:**
- Gradient flow across component boundaries
- Model serialization and loading
- Error propagation and recovery
- Performance optimization across boundaries

### 3. Production Deployment Tests
Tests complete production deployment scenarios.

**Deployment Patterns:**
- Containerized deployment with Docker/Podman
- Multi-model serving with load balancing
- Hot model swapping and version management
- Monitoring and observability integration
- Fault tolerance and recovery

**Key Scenarios:**
- Blue-green and canary deployments
- Auto-scaling under load
- Service mesh integration
- Database and cache integration

### 4. Performance Integration Tests
Validates performance characteristics meet production requirements.

**Performance Metrics:**
- Latency SLAs (P50, P95, P99)
- Throughput scaling
- Memory utilization efficiency
- GPU utilization optimization
- Concurrent request handling

**Key Scenarios:**
- Load testing under realistic traffic
- Resource scaling validation
- Performance regression detection
- Bottleneck identification

### 5. Real-World Use Case Tests
Tests complete real-world ML use cases end-to-end.

**Use Cases:**
- BERT fine-tuning and deployment
- GPT text generation pipeline
- Computer vision workflows
- CLIP multimodal search
- Recommendation systems
- Time series forecasting

**Key Scenarios:**
- Production-quality model training
- Realistic dataset handling
- API integration validation
- Accuracy and performance benchmarking

### 6. Multi-Tenant System Tests
Validates multi-tenant capabilities and resource isolation.

**Multi-Tenancy Features:**
- Resource isolation (CPU, memory, GPU)
- Fair scheduling and quota enforcement
- Security boundary validation
- Per-tenant configuration management
- Billing and usage tracking

### 7. Disaster Recovery Tests
Tests fault tolerance and disaster recovery procedures.

**Recovery Scenarios:**
- Node failure and recovery
- Data corruption handling
- Network partition tolerance
- Complete system backup/restore
- Graceful degradation modes

### 8. Security Integration Tests
Validates comprehensive security features.

**Security Features:**
- Authentication and authorization
- Input validation and sanitization
- Encryption (data at rest and in transit)
- Audit logging and compliance
- API security (rate limiting, CORS)

## Quick Start

### Prerequisites

**System Requirements:**
- Linux/macOS with Docker or Podman
- NVIDIA GPU with CUDA 12.0+ (recommended)
- 32GB+ RAM for memory-intensive tests
- Network connectivity for distributed tests

**Software Requirements:**
```bash
# Install Rust 2021 edition
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Install Docker (or Podman)
# Ubuntu/Debian:
sudo apt-get install docker.io docker-compose

# Install additional dependencies
sudo apt-get install postgresql-client redis-tools
```

**Environment Setup:**
```bash
# Clone and setup workspace
git clone <rustytorch-repo>
cd rustytorch

# Install development dependencies
cargo install --locked cargo-nextest  # For advanced testing
cargo install --locked cargo-watch    # For continuous testing
```

### Running Tests

#### Run All Integration Tests
```bash
# Run complete integration test suite
cargo test --package rustytorch-integration-tests --features full-integration

# Run with specific backend
RTX_BACKEND=cuda cargo test --package rustytorch-integration-tests

# Run with verbose output
cargo test --package rustytorch-integration-tests -- --nocapture
```

#### Run Specific Test Categories
```bash
# ML Pipeline Tests
cargo run --bin ml_pipeline_tests

# Cross-Component Integration Tests
cargo run --bin cross_component_tests

# Production Deployment Tests
cargo run --bin production_deployment_tests

# Performance Integration Tests
RTX_PERFORMANCE_MODE=1 cargo run --bin performance_integration_tests

# Real-World Use Case Tests
cargo run --bin real_world_validation_tests

# Multi-Tenant Tests
cargo run --bin multi_tenant_tests

# Disaster Recovery Tests
cargo run --bin disaster_recovery_tests

# Security Integration Tests
cargo run --bin security_integration_tests
```

#### Performance Benchmarks
```bash
# Run all performance benchmarks
cargo bench --package rustytorch-integration-tests

# Run specific benchmark categories
cargo bench --bench end_to_end_latency
cargo bench --bench throughput_scaling
cargo bench --bench memory_utilization

# Generate HTML reports
cargo bench --package rustytorch-integration-tests -- --output-format html
```

#### Advanced Testing with Nextest
```bash
# Install nextest for advanced test execution
cargo install cargo-nextest --locked

# Run tests with better output formatting
cargo nextest run --package rustytorch-integration-tests

# Run tests in parallel with custom configuration
cargo nextest run --package rustytorch-integration-tests --config-file nextest.toml
```

## Configuration

### Environment Variables

**Core Configuration:**
```bash
# Backend selection (default: cuda)
export RTX_BACKEND=cuda|rocm|metal|cpu

# Device configuration
export RTX_DEVICE_COUNT=2              # Number of GPUs to use
export RTX_SKIP_GPU_TESTS=1           # Skip GPU-dependent tests

# Test data and timeouts
export RTX_TEST_DATA_PATH=/tmp/rtx_test_data
export RTX_TEST_TIMEOUT=300           # Test timeout in seconds
export RTX_MAX_MEMORY_MB=16384        # Maximum memory usage per test

# Container runtime
export RTX_CONTAINER_RUNTIME=docker   # docker|podman
```

**Database Configuration:**
```bash
# PostgreSQL for metadata storage
export RTX_POSTGRES_URL=postgresql://user:pass@localhost:5432/rtx_test

# Redis for caching
export RTX_REDIS_URL=redis://localhost:6379

# Test database isolation
export RTX_TEST_DB_ISOLATION=1        # Use separate test databases
```

**Performance Testing:**
```bash
# Enable performance mode with strict SLA validation
export RTX_PERFORMANCE_MODE=1

# Performance testing configuration
export RTX_PERFORMANCE_DURATION=300   # Performance test duration (seconds)
export RTX_LOAD_TEST_CONCURRENCY=32   # Concurrent requests for load testing
export RTX_BENCHMARK_WARMUP=10        # Benchmark warmup iterations
```

**Distributed Testing:**
```bash
# Enable distributed testing
export RTX_ENABLE_DISTRIBUTED=1

# Distributed configuration
export RTX_MASTER_ADDR=127.0.0.1
export RTX_MASTER_PORT=29500
export RTX_DISTRIBUTED_BACKEND=nccl   # nccl|gloo|mpi
```

**Security Testing:**
```bash
# Enable security testing features
export RTX_ENABLE_SECURITY_TESTS=1

# Security configuration
export RTX_JWT_SECRET=test-secret-key
export RTX_OAUTH_CLIENT_ID=test-client
export RTX_ENABLE_AUDIT_LOGGING=1
```

### Test Configuration Files

**Main Configuration (`integration_tests/config.toml`):**
```toml
[test_environment]
backend = "cuda"
device_count = 2
timeout_seconds = 300
max_memory_mb = 16384

[databases]
postgres_url = "postgresql://rtx_user:rtx_pass@localhost:5432/rtx_test"
redis_url = "redis://localhost:6379"
use_test_isolation = true

[performance]
enable_performance_mode = true
latency_p99_threshold_ms = 100
throughput_min_rps = 1000
memory_efficiency_threshold = 0.85

[deployment]
container_runtime = "docker"
enable_monitoring = true
enable_distributed = true

[security]
enable_security_tests = true
jwt_expiry_seconds = 3600
rate_limit_per_minute = 1000
```

**Nextest Configuration (`nextest.toml`):**
```toml
[profile.default]
slow-timeout = "300s"
retries = 1
threads-required = 1

[profile.integration]
slow-timeout = "600s"
retries = 0
threads-required = 2
setup-scripts = ["scripts/setup-integration-env.sh"]
teardown-scripts = ["scripts/cleanup-integration-env.sh"]

[profile.performance]
slow-timeout = "1800s"
retries = 0
threads-required = 4

[[profile.default.overrides]]
filter = 'test(gpu_)'
threads-required = 1
setup-scripts = ["scripts/setup-gpu-env.sh"]
```

## Test Data Management

### Test Dataset Generation
```bash
# Generate synthetic datasets for testing
cargo run --bin generate_test_data -- \
  --output-dir ./test_data \
  --dataset-types classification,generation,vision \
  --sizes small,medium,large

# Download real datasets for validation (optional)
cargo run --bin download_datasets -- \
  --datasets cifar10,imdb,squad \
  --output-dir ./test_data/real
```

### Test Data Structure
```
integration_tests/
├── test_data/
│   ├── synthetic/
│   │   ├── classification/
│   │   │   ├── train.jsonl
│   │   │   ├── valid.jsonl
│   │   │   └── test.jsonl
│   │   ├── generation/
│   │   │   └── prompts.jsonl
│   │   └── vision/
│   │       ├── images/
│   │       └── labels.jsonl
│   ├── models/
│   │   ├── pretrained/
│   │   ├── checkpoints/
│   │   └── compressed/
│   └── configs/
│       ├── training/
│       ├── inference/
│       └── deployment/
```

## Continuous Integration

### GitHub Actions Workflow
```yaml
# .github/workflows/integration-tests.yml
name: Integration Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  integration-tests:
    runs-on: ubuntu-latest-gpu  # Custom GPU runner
    timeout-minutes: 120

    strategy:
      matrix:
        backend: [cuda, cpu]
        test-category: [pipeline, cross-component, production, performance]

    steps:
    - uses: actions/checkout@v4

    - name: Setup Rust
      uses: actions-rs/toolchain@v1
      with:
        toolchain: stable
        override: true

    - name: Setup Docker
      run: |
        sudo apt-get update
        sudo apt-get install -y docker.io docker-compose
        sudo systemctl start docker

    - name: Setup Test Environment
      run: |
        docker-compose -f integration_tests/docker-compose.yml up -d
        ./scripts/wait-for-services.sh

    - name: Run Integration Tests
      env:
        RTX_BACKEND: ${{ matrix.backend }}
        RTX_PERFORMANCE_MODE: 1
        RTX_CI_MODE: 1
      run: |
        case "${{ matrix.test-category }}" in
          pipeline)
            cargo run --bin ml_pipeline_tests
            ;;
          cross-component)
            cargo run --bin cross_component_tests
            ;;
          production)
            cargo run --bin production_deployment_tests
            ;;
          performance)
            cargo run --bin performance_integration_tests
            ;;
        esac

    - name: Upload Test Results
      uses: actions/upload-artifact@v3
      if: always()
      with:
        name: test-results-${{ matrix.backend }}-${{ matrix.test-category }}
        path: |
          integration_tests/test-results/
          integration_tests/artifacts/

    - name: Cleanup
      if: always()
      run: |
        docker-compose -f integration_tests/docker-compose.yml down -v
        ./scripts/cleanup-test-env.sh
```

### Docker Compose for CI
```yaml
# integration_tests/docker-compose.yml
version: '3.8'

services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: rtx_integration_test
      POSTGRES_USER: rtx_test_user
      POSTGRES_PASSWORD: rtx_test_pass
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    command: redis-server --maxmemory 256mb --maxmemory-policy allkeys-lru

  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"
      - "14268:14268"
    environment:
      COLLECTOR_OTLP_ENABLED: true

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - ./config/grafana/provisioning:/etc/grafana/provisioning

volumes:
  postgres_data:
```

## Troubleshooting

### Common Issues

**GPU Tests Failing:**
```bash
# Check CUDA/ROCm installation
nvidia-smi  # For NVIDIA GPUs
rocm-smi   # For AMD GPUs

# Run CPU-only tests
RTX_SKIP_GPU_TESTS=1 cargo test --package rustytorch-integration-tests

# Check GPU memory availability
RTX_DEVICE_COUNT=1 cargo run --bin performance_integration_tests
```

**Container Tests Failing:**
```bash
# Check Docker/Podman status
sudo systemctl status docker
docker version

# Use Podman instead
RTX_CONTAINER_RUNTIME=podman cargo run --bin production_deployment_tests

# Check container permissions
sudo usermod -aG docker $USER
newgrp docker
```

**Database Connection Issues:**
```bash
# Start local databases
docker run -d --name rtx-postgres -p 5432:5432 \
  -e POSTGRES_DB=rtx_test -e POSTGRES_USER=rtx_user -e POSTGRES_PASSWORD=rtx_pass \
  postgres:15

docker run -d --name rtx-redis -p 6379:6379 redis:7

# Test connections
pg_isready -h localhost -p 5432 -U rtx_user
redis-cli -h localhost -p 6379 ping
```

**Performance Test Issues:**
```bash
# Reduce test intensity
RTX_PERFORMANCE_DURATION=60 cargo run --bin performance_integration_tests

# Check system resources
free -h     # Memory availability
df -h       # Disk space
lscpu       # CPU information

# Monitor during tests
top -p $(pgrep -f "integration_tests")
```

**Network Issues:**
```bash
# Check port availability
netstat -tlnp | grep -E "(5432|6379|8080)"

# Use alternative ports
RTX_POSTGRES_URL=postgresql://rtx_user:rtx_pass@localhost:5433/rtx_test \
RTX_REDIS_URL=redis://localhost:6380 \
cargo run --bin cross_component_tests
```

### Debug Mode

**Enable Debug Logging:**
```bash
RUST_LOG=debug cargo run --bin ml_pipeline_tests

# Component-specific logging
RUST_LOG=rustytorch_integration_tests=debug,rtx_tensor=info \
cargo run --bin cross_component_tests

# Trace-level logging (very verbose)
RUST_LOG=trace cargo test --package rustytorch-integration-tests -- --nocapture
```

**Generate Debug Artifacts:**
```bash
# Enable artifact generation
RTX_SAVE_ARTIFACTS=1 cargo run --bin production_deployment_tests

# Artifacts saved to:
ls -la integration_tests/artifacts/
```

### Test Isolation

**Run Tests in Isolation:**
```bash
# Run single test
cargo test --package rustytorch-integration-tests test_data_loading_pipeline

# Run with fresh environment
RTX_CLEAN_ENV=1 cargo run --bin ml_pipeline_tests

# Use separate test data directory
RTX_TEST_DATA_PATH=/tmp/rtx_isolated_test_data \
cargo run --bin real_world_validation_tests
```

## Contributing

### Adding New Tests

**1. Choose Test Category:**
```bash
# For ML pipeline tests
edit integration_tests/src/pipeline.rs

# For cross-component tests
edit integration_tests/src/cross_component.rs

# For new test category
create integration_tests/src/my_new_category.rs
```

**2. Follow TDD Pattern:**
```rust
#[tokio::test]
async fn test_my_new_feature() -> Result<()> {
    info!("Testing my new feature...");
    let mut ctx = TestContext::new();

    // Setup test environment
    let test_data = setup_test_data().await?;

    // Execute test scenario
    let result = execute_test_scenario(&test_data).await?;

    // Validate results
    assert!(result.is_valid(), "Test result validation failed");
    assert!(result.performance_meets_sla(), "Performance SLA not met");

    ctx.cleanup().await?;
    info!("Test passed");
    Ok(())
}
```

**3. Add Integration Test Macro:**
```rust
crate::integration_test!("my_new_feature_test",
    || self.test_my_new_feature(), &mut results);
```

**4. Create Binary (if needed):**
```rust
// integration_tests/src/bin/my_category_tests.rs
use anyhow::Result;
use rustytorch_integration_tests::{initialize_test_environment, my_category::MyCategoryTests};
use tracing::{info, error};

#[tokio::main]
async fn main() -> Result<()> {
    let config = initialize_test_environment().await?;
    let tests = MyCategoryTests::new(config);

    match tests.run_all_tests().await {
        Ok(results) => {
            results.print_summary();
            std::process::exit(if results.failed > 0 { 1 } else { 0 });
        }
        Err(e) => {
            error!("Test execution failed: {}", e);
            std::process::exit(1);
        }
    }
}
```

### Testing Guidelines

**Test Naming:**
- Use descriptive test names: `test_bert_fine_tuning_with_distributed_training`
- Group related tests: `test_model_compression_*`, `test_deployment_*`
- Follow pattern: `test_<component>_<scenario>_<expected_outcome>`

**Test Structure:**
- Use `TestContext` for resource management
- Always call `ctx.cleanup().await?` before returning
- Use `integration_test!` macro for consistent error handling
- Validate both functionality and performance

**Error Handling:**
- Use `Result<()>` return type for all test functions
- Provide descriptive error messages with context
- Test both success and failure scenarios
- Validate error propagation across components

**Performance Requirements:**
- Include performance assertions for critical paths
- Use realistic load patterns and data sizes
- Test scaling behavior with multiple configurations
- Measure and validate resource utilization

**Documentation:**
- Document test purpose and expected behavior
- Include setup requirements and dependencies
- Provide troubleshooting guidance
- Update README when adding new test categories

## License

This integration test framework is part of RustyTorch++ and is licensed under the same terms as the main project (MIT OR Apache-2.0).