rustytorch/integration_tests/README.md

RustyTorch++ Integration Tests

Comprehensive end-to-end integration testing framework for the entire RustyTorch++ platform using strict Test-Driven Development (TDD) methodology. These tests validate real production workloads and scenarios without mocked dependencies.

Overview

This integration test suite provides comprehensive validation of:

• Complete ML pipelines (data → training → evaluation → deployment)
• Cross-component integration between all RustyTorch++ modules
• Production deployment scenarios with containerization
• Performance characteristics with SLA validation
• Real-world use cases (BERT, GPT, CLIP, etc.)
• Multi-tenant system capabilities
• Disaster recovery and fault tolerance
• Security and compliance features

Test Categories

1. ML Pipeline Integration Tests

Tests the complete machine learning pipeline from data loading through deployment.

Components Tested:

• Data loading and validation (rtx-data-validation, rtx-etl)
• Model training (rtx-transformers, rtx-autograd, rtx-distributed)
• Model evaluation (rtx-eval)
• Model deployment (rtx-serving-api, rtx-inference)
• Model compression (rtx-compress)

Key Scenarios:

• End-to-end classification pipeline
• Distributed multi-GPU training
• Model compression and deployment
• Performance optimization validation

2. Cross-Component Integration Tests

Validates integration between all major RustyTorch++ components.

Integration Patterns:

• Tensor ↔ Autograd (gradient computation)
• Autograd ↔ Transformers (training integration)
• Transformers ↔ Serving (inference pipeline)
• Memory ↔ Runtime (resource management)
• Distributed ↔ Training (multi-GPU coordination)

Key Scenarios:

• Gradient flow across component boundaries
• Model serialization and loading
• Error propagation and recovery
• Performance optimization across boundaries

3. Production Deployment Tests

Tests complete production deployment scenarios.

Deployment Patterns:

• Containerized deployment with Docker/Podman
• Multi-model serving with load balancing
• Hot model swapping and version management
• Monitoring and observability integration
• Fault tolerance and recovery

Key Scenarios:

• Blue-green and canary deployments
• Auto-scaling under load
• Service mesh integration
• Database and cache integration

4. Performance Integration Tests

Validates performance characteristics meet production requirements.

Performance Metrics:

• Latency SLAs (P50, P95, P99)
• Throughput scaling
• Memory utilization efficiency
• GPU utilization optimization
• Concurrent request handling

Key Scenarios:

• Load testing under realistic traffic
• Resource scaling validation
• Performance regression detection
• Bottleneck identification

5. Real-World Use Case Tests

Tests complete real-world ML use cases end-to-end.

Use Cases:

• BERT fine-tuning and deployment
• GPT text generation pipeline
• Computer vision workflows
• CLIP multimodal search
• Recommendation systems
• Time series forecasting

Key Scenarios:

• Production-quality model training
• Realistic dataset handling
• API integration validation
• Accuracy and performance benchmarking

6. Multi-Tenant System Tests

Validates multi-tenant capabilities and resource isolation.

Multi-Tenancy Features:

• Resource isolation (CPU, memory, GPU)
• Fair scheduling and quota enforcement
• Security boundary validation
• Per-tenant configuration management
• Billing and usage tracking

7. Disaster Recovery Tests

Tests fault tolerance and disaster recovery procedures.

Recovery Scenarios:

• Node failure and recovery
• Data corruption handling
• Network partition tolerance
• Complete system backup/restore
• Graceful degradation modes

8. Security Integration Tests

Validates comprehensive security features.

Security Features:

• Authentication and authorization
• Input validation and sanitization
• Encryption (data at rest and in transit)
• Audit logging and compliance
• API security (rate limiting, CORS)

Quick Start

Prerequisites

System Requirements:

• Linux/macOS with Docker or Podman
• NVIDIA GPU with CUDA 12.0+ (recommended)
• 32GB+ RAM for memory-intensive tests
• Network connectivity for distributed tests

Software Requirements:

# Install Rust 2021 edition
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Install Docker (or Podman)
# Ubuntu/Debian:
sudo apt-get install docker.io docker-compose

# Install additional dependencies
sudo apt-get install postgresql-client redis-tools

Environment Setup:

# Clone and setup workspace
git clone <rustytorch-repo>
cd rustytorch

# Install development dependencies
cargo install --locked cargo-nextest  # For advanced testing
cargo install --locked cargo-watch    # For continuous testing

Running Tests

Run All Integration Tests

# Run complete integration test suite
cargo test --package rustytorch-integration-tests --features full-integration

# Run with specific backend
RTX_BACKEND=cuda cargo test --package rustytorch-integration-tests

# Run with verbose output
cargo test --package rustytorch-integration-tests -- --nocapture

Run Specific Test Categories

# ML Pipeline Tests
cargo run --bin ml_pipeline_tests

# Cross-Component Integration Tests
cargo run --bin cross_component_tests

# Production Deployment Tests
cargo run --bin production_deployment_tests

# Performance Integration Tests
RTX_PERFORMANCE_MODE=1 cargo run --bin performance_integration_tests

# Real-World Use Case Tests
cargo run --bin real_world_validation_tests

# Multi-Tenant Tests
cargo run --bin multi_tenant_tests

# Disaster Recovery Tests
cargo run --bin disaster_recovery_tests

# Security Integration Tests
cargo run --bin security_integration_tests

Performance Benchmarks

# Run all performance benchmarks
cargo bench --package rustytorch-integration-tests

# Run specific benchmark categories
cargo bench --bench end_to_end_latency
cargo bench --bench throughput_scaling
cargo bench --bench memory_utilization

# Generate HTML reports
cargo bench --package rustytorch-integration-tests -- --output-format html

Advanced Testing with Nextest

# Install nextest for advanced test execution
cargo install cargo-nextest --locked

# Run tests with better output formatting
cargo nextest run --package rustytorch-integration-tests

# Run tests in parallel with custom configuration
cargo nextest run --package rustytorch-integration-tests --config-file nextest.toml

Configuration

Environment Variables

Core Configuration:

# Backend selection (default: cuda)
export RTX_BACKEND=cuda|rocm|metal|cpu

# Device configuration
export RTX_DEVICE_COUNT=2              # Number of GPUs to use
export RTX_SKIP_GPU_TESTS=1           # Skip GPU-dependent tests

# Test data and timeouts
export RTX_TEST_DATA_PATH=/tmp/rtx_test_data
export RTX_TEST_TIMEOUT=300           # Test timeout in seconds
export RTX_MAX_MEMORY_MB=16384        # Maximum memory usage per test

# Container runtime
export RTX_CONTAINER_RUNTIME=docker   # docker|podman

Database Configuration:

# PostgreSQL for metadata storage
export RTX_POSTGRES_URL=postgresql://user:pass@localhost:5432/rtx_test

# Redis for caching
export RTX_REDIS_URL=redis://localhost:6379

# Test database isolation
export RTX_TEST_DB_ISOLATION=1        # Use separate test databases

Performance Testing:

# Enable performance mode with strict SLA validation
export RTX_PERFORMANCE_MODE=1

# Performance testing configuration
export RTX_PERFORMANCE_DURATION=300   # Performance test duration (seconds)
export RTX_LOAD_TEST_CONCURRENCY=32   # Concurrent requests for load testing
export RTX_BENCHMARK_WARMUP=10        # Benchmark warmup iterations

Distributed Testing:

# Enable distributed testing
export RTX_ENABLE_DISTRIBUTED=1

# Distributed configuration
export RTX_MASTER_ADDR=127.0.0.1
export RTX_MASTER_PORT=29500
export RTX_DISTRIBUTED_BACKEND=nccl   # nccl|gloo|mpi

Security Testing:

# Enable security testing features
export RTX_ENABLE_SECURITY_TESTS=1

# Security configuration
export RTX_JWT_SECRET=test-secret-key
export RTX_OAUTH_CLIENT_ID=test-client
export RTX_ENABLE_AUDIT_LOGGING=1

Test Configuration Files

Main Configuration (integration_tests/config.toml):

[test_environment]
backend = "cuda"
device_count = 2
timeout_seconds = 300
max_memory_mb = 16384

[databases]
postgres_url = "postgresql://rtx_user:rtx_pass@localhost:5432/rtx_test"
redis_url = "redis://localhost:6379"
use_test_isolation = true

[performance]
enable_performance_mode = true
latency_p99_threshold_ms = 100
throughput_min_rps = 1000
memory_efficiency_threshold = 0.85

[deployment]
container_runtime = "docker"
enable_monitoring = true
enable_distributed = true

[security]
enable_security_tests = true
jwt_expiry_seconds = 3600
rate_limit_per_minute = 1000

Nextest Configuration (nextest.toml):

[profile.default]
slow-timeout = "300s"
retries = 1
threads-required = 1

[profile.integration]
slow-timeout = "600s"
retries = 0
threads-required = 2
setup-scripts = ["scripts/setup-integration-env.sh"]
teardown-scripts = ["scripts/cleanup-integration-env.sh"]

[profile.performance]
slow-timeout = "1800s"
retries = 0
threads-required = 4

[[profile.default.overrides]]
filter = 'test(gpu_)'
threads-required = 1
setup-scripts = ["scripts/setup-gpu-env.sh"]

Test Data Management

Test Dataset Generation

# Generate synthetic datasets for testing
cargo run --bin generate_test_data -- \
  --output-dir ./test_data \
  --dataset-types classification,generation,vision \
  --sizes small,medium,large

# Download real datasets for validation (optional)
cargo run --bin download_datasets -- \
  --datasets cifar10,imdb,squad \
  --output-dir ./test_data/real

Test Data Structure

integration_tests/
├── test_data/
│   ├── synthetic/
│   │   ├── classification/
│   │   │   ├── train.jsonl
│   │   │   ├── valid.jsonl
│   │   │   └── test.jsonl
│   │   ├── generation/
│   │   │   └── prompts.jsonl
│   │   └── vision/
│   │       ├── images/
│   │       └── labels.jsonl
│   ├── models/
│   │   ├── pretrained/
│   │   ├── checkpoints/
│   │   └── compressed/
│   └── configs/
│       ├── training/
│       ├── inference/
│       └── deployment/

Continuous Integration

GitHub Actions Workflow

# .github/workflows/integration-tests.yml
name: Integration Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  integration-tests:
    runs-on: ubuntu-latest-gpu  # Custom GPU runner
    timeout-minutes: 120

    strategy:
      matrix:
        backend: [cuda, cpu]
        test-category: [pipeline, cross-component, production, performance]

    steps:
    - uses: actions/checkout@v4

    - name: Setup Rust
      uses: actions-rs/toolchain@v1
      with:
        toolchain: stable
        override: true

    - name: Setup Docker
      run: |
        sudo apt-get update
        sudo apt-get install -y docker.io docker-compose
        sudo systemctl start docker

    - name: Setup Test Environment
      run: |
        docker-compose -f integration_tests/docker-compose.yml up -d
        ./scripts/wait-for-services.sh

    - name: Run Integration Tests
      env:
        RTX_BACKEND: ${{ matrix.backend }}
        RTX_PERFORMANCE_MODE: 1
        RTX_CI_MODE: 1
      run: |
        case "${{ matrix.test-category }}" in
          pipeline)
            cargo run --bin ml_pipeline_tests
            ;;
          cross-component)
            cargo run --bin cross_component_tests
            ;;
          production)
            cargo run --bin production_deployment_tests
            ;;
          performance)
            cargo run --bin performance_integration_tests
            ;;
        esac

    - name: Upload Test Results
      uses: actions/upload-artifact@v3
      if: always()
      with:
        name: test-results-${{ matrix.backend }}-${{ matrix.test-category }}
        path: |
          integration_tests/test-results/
          integration_tests/artifacts/

    - name: Cleanup
      if: always()
      run: |
        docker-compose -f integration_tests/docker-compose.yml down -v
        ./scripts/cleanup-test-env.sh

Docker Compose for CI

# integration_tests/docker-compose.yml
version: '3.8'

services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: rtx_integration_test
      POSTGRES_USER: rtx_test_user
      POSTGRES_PASSWORD: rtx_test_pass
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    command: redis-server --maxmemory 256mb --maxmemory-policy allkeys-lru

  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"
      - "14268:14268"
    environment:
      COLLECTOR_OTLP_ENABLED: true

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - ./config/grafana/provisioning:/etc/grafana/provisioning

volumes:
  postgres_data:

Troubleshooting

Common Issues

GPU Tests Failing:

# Check CUDA/ROCm installation
nvidia-smi  # For NVIDIA GPUs
rocm-smi   # For AMD GPUs

# Run CPU-only tests
RTX_SKIP_GPU_TESTS=1 cargo test --package rustytorch-integration-tests

# Check GPU memory availability
RTX_DEVICE_COUNT=1 cargo run --bin performance_integration_tests

Container Tests Failing:

# Check Docker/Podman status
sudo systemctl status docker
docker version

# Use Podman instead
RTX_CONTAINER_RUNTIME=podman cargo run --bin production_deployment_tests

# Check container permissions
sudo usermod -aG docker $USER
newgrp docker

Database Connection Issues:

# Start local databases
docker run -d --name rtx-postgres -p 5432:5432 \
  -e POSTGRES_DB=rtx_test -e POSTGRES_USER=rtx_user -e POSTGRES_PASSWORD=rtx_pass \
  postgres:15

docker run -d --name rtx-redis -p 6379:6379 redis:7

# Test connections
pg_isready -h localhost -p 5432 -U rtx_user
redis-cli -h localhost -p 6379 ping

Performance Test Issues:

# Reduce test intensity
RTX_PERFORMANCE_DURATION=60 cargo run --bin performance_integration_tests

# Check system resources
free -h     # Memory availability
df -h       # Disk space
lscpu       # CPU information

# Monitor during tests
top -p $(pgrep -f "integration_tests")

Network Issues:

# Check port availability
netstat -tlnp | grep -E "(5432|6379|8080)"

# Use alternative ports
RTX_POSTGRES_URL=postgresql://rtx_user:rtx_pass@localhost:5433/rtx_test \
RTX_REDIS_URL=redis://localhost:6380 \
cargo run --bin cross_component_tests

Debug Mode

Enable Debug Logging:

RUST_LOG=debug cargo run --bin ml_pipeline_tests

# Component-specific logging
RUST_LOG=rustytorch_integration_tests=debug,rtx_tensor=info \
cargo run --bin cross_component_tests

# Trace-level logging (very verbose)
RUST_LOG=trace cargo test --package rustytorch-integration-tests -- --nocapture

Generate Debug Artifacts:

# Enable artifact generation
RTX_SAVE_ARTIFACTS=1 cargo run --bin production_deployment_tests

# Artifacts saved to:
ls -la integration_tests/artifacts/

Test Isolation

Run Tests in Isolation:

# Run single test
cargo test --package rustytorch-integration-tests test_data_loading_pipeline

# Run with fresh environment
RTX_CLEAN_ENV=1 cargo run --bin ml_pipeline_tests

# Use separate test data directory
RTX_TEST_DATA_PATH=/tmp/rtx_isolated_test_data \
cargo run --bin real_world_validation_tests

Contributing

Adding New Tests

1. Choose Test Category:

# For ML pipeline tests
edit integration_tests/src/pipeline.rs

# For cross-component tests
edit integration_tests/src/cross_component.rs

# For new test category
create integration_tests/src/my_new_category.rs

2. Follow TDD Pattern:

#[tokio::test]
async fn test_my_new_feature() -> Result<()> {
    info!("Testing my new feature...");
    let mut ctx = TestContext::new();

    // Setup test environment
    let test_data = setup_test_data().await?;

    // Execute test scenario
    let result = execute_test_scenario(&test_data).await?;

    // Validate results
    assert!(result.is_valid(), "Test result validation failed");
    assert!(result.performance_meets_sla(), "Performance SLA not met");

    ctx.cleanup().await?;
    info!("Test passed");
    Ok(())
}

3. Add Integration Test Macro:

crate::integration_test!("my_new_feature_test",
    || self.test_my_new_feature(), &mut results);

4. Create Binary (if needed):

// integration_tests/src/bin/my_category_tests.rs
use anyhow::Result;
use rustytorch_integration_tests::{initialize_test_environment, my_category::MyCategoryTests};
use tracing::{info, error};

#[tokio::main]
async fn main() -> Result<()> {
    let config = initialize_test_environment().await?;
    let tests = MyCategoryTests::new(config);

    match tests.run_all_tests().await {
        Ok(results) => {
            results.print_summary();
            std::process::exit(if results.failed > 0 { 1 } else { 0 });
        }
        Err(e) => {
            error!("Test execution failed: {}", e);
            std::process::exit(1);
        }
    }
}

Testing Guidelines

Test Naming:

• Use descriptive test names: test_bert_fine_tuning_with_distributed_training
• Group related tests: test_model_compression_*, test_deployment_*
• Follow pattern: test_<component>_<scenario>_<expected_outcome>

Test Structure:

• Use TestContext for resource management
• Always call ctx.cleanup().await? before returning
• Use integration_test! macro for consistent error handling
• Validate both functionality and performance

Error Handling:

• Use Result<()> return type for all test functions
• Provide descriptive error messages with context
• Test both success and failure scenarios
• Validate error propagation across components

Performance Requirements:

• Include performance assertions for critical paths
• Use realistic load patterns and data sizes
• Test scaling behavior with multiple configurations
• Measure and validate resource utilization

Documentation:

• Document test purpose and expected behavior
• Include setup requirements and dependencies
• Provide troubleshooting guidance
• Update README when adding new test categories

License

This integration test framework is part of RustyTorch++ and is licensed under the same terms as the main project (MIT OR Apache-2.0).