rustytorch/docs/book/src/deployment/production.md

Production Deployment

Deploy RustyTorch++ models to production environments.

Deployment Options

Option	Latency	Throughput	Complexity
Direct Binary	~1ms	High	Low
HTTP API	~5-10ms	Medium	Medium
gRPC	~2-5ms	High	Medium
Kubernetes	~10-20ms	Scalable	High

Direct Binary Deployment

Building for Production

# Optimized release build
cargo build --release -p rtx-inference

# Strip symbols for smaller binary
strip target/release/rtx-inference

Production Cargo.toml

[profile.release]
lto = "thin"           # Link-time optimization
codegen-units = 1      # Better optimization
panic = "abort"        # Smaller binary
strip = true           # Remove debug symbols
opt-level = 3          # Maximum optimization

HTTP API Deployment

Using rtx-serving-api

use rtx_serving_api::{Server, ServerConfig, ModelRegistry};
use rtx_inference::InferenceEngine;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Load model
    let engine = InferenceEngine::load("model.bin")?;

    // Configure server
    let config = ServerConfig {
        host: "0.0.0.0".to_string(),
        port: 8080,
        max_batch_size: 32,
        timeout_ms: 5000,
        workers: num_cpus::get(),
    };

    // Start server
    let server = Server::new(config, engine)?;
    server.run().await
}

API Endpoints

POST /v1/predict       - Single prediction
POST /v1/batch         - Batch predictions
GET  /v1/health        - Health check
GET  /v1/metrics       - Prometheus metrics
WS   /v1/stream        - Streaming predictions

Model Optimization

Quantization

use rtx_compress::{Quantizer, QuantConfig};

let config = QuantConfig {
    dtype: DType::I8,           // INT8 quantization
    calibration_samples: 1000,  // Calibration dataset size
    per_channel: true,          // Per-channel quantization
};

let quantizer = Quantizer::new(config)?;
let quantized_model = quantizer.quantize(&model)?;

// 4x smaller, ~2x faster on INT8 capable hardware
quantized_model.save("model_int8.bin")?;

ONNX Export

use rtx_bindings::OnnxExporter;

let exporter = OnnxExporter::new();
exporter.export(&model, "model.onnx")?;

Request Batching

use rtx_serving_api::BatchProcessor;

let processor = BatchProcessor::new(BatchConfig {
    max_batch_size: 64,
    max_wait_ms: 10,        // Wait up to 10ms to fill batch
    dynamic_batching: true,  // Adaptive batch sizes
});

Load Balancing

Multiple Instances

# docker-compose.yml
services:
  rtx-api-1:
    image: rustytorch/serving-api
    deploy:
      replicas: 3
      resources:
        reservations:
          devices:
            - capabilities: [gpu]

  nginx:
    image: nginx
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf

Health Checks

use rtx_serving_api::HealthCheck;

// Configure health checks
let health = HealthCheck::new()
    .with_gpu_check()           // Verify GPU available
    .with_model_check(&model)   // Verify model loaded
    .with_memory_threshold(0.9); // Alert at 90% memory

// Expose at /health
server.add_health_check(health);

Monitoring

Prometheus Metrics

use rtx_monitoring::{MetricsExporter, PrometheusConfig};

let exporter = MetricsExporter::prometheus(PrometheusConfig {
    endpoint: "/metrics",
    buckets: vec![0.001, 0.005, 0.01, 0.025, 0.05, 0.1],
});

// Exposed metrics:
// - rtx_inference_latency_seconds
// - rtx_inference_throughput_requests
// - rtx_gpu_memory_used_bytes
// - rtx_batch_size_histogram

Logging

use tracing_subscriber::fmt;

// Structured JSON logging for production
fmt()
    .json()
    .with_env_filter("rtx=info,tower_http=warn")
    .init();

Security

Input Validation

use rtx_serving_api::validation::InputValidator;

let validator = InputValidator::new()
    .max_sequence_length(2048)
    .max_batch_size(64)
    .allowed_dtypes(&[DType::F32, DType::F16]);

server.add_validator(validator);

Rate Limiting

use tower::limit::RateLimitLayer;

let rate_limit = RateLimitLayer::new(100, Duration::from_secs(1));
// 100 requests per second

Next Steps

• Edge Deployment - Deploy to edge devices
• Docker & Kubernetes - Container orchestration
• Serving API - API customization