# Continuous Batching

RustyTorch++ implements **SLA-aware continuous batching** for production LLM serving.

## Overview

Continuous batching dynamically manages requests to maximize throughput while meeting latency SLAs.

## Usage

```rust
use rtx_inference::{BatchScheduler, SlaLane};

let scheduler = BatchScheduler::new(config)
    .with_sla_lane(SlaLane::Interactive, Duration::from_millis(100))
    .with_sla_lane(SlaLane::Batch, Duration::from_secs(5));

// Requests are automatically batched and prioritized
scheduler.submit(request).await?;
```

## Features

- **Iteration-level preemption** for priority requests
- **Chunked prefill** for reduced time-to-first-token
- **Multi-LoRA serving** in a single batch
- **Automatic routing** based on context length

## Performance

| Max Batch Size | Requests/sec | P99 Latency | GPU Util |
|----------------|--------------|-------------|----------|
| 32 | 45 | 85ms | 78% |
| 64 | 82 | 95ms | 89% |
| 128 | 140 | 120ms | 94% |

## Next Steps

- [KV-Cache Optimization](./kv-cache.md)
- [Quantization](./quantization.md)