rustytorch/docs/book/src/llm/continuous-batching.md

Continuous Batching

RustyTorch++ implements SLA-aware continuous batching for production LLM serving.

Overview

Continuous batching dynamically manages requests to maximize throughput while meeting latency SLAs.

Usage

use rtx_inference::{BatchScheduler, SlaLane};

let scheduler = BatchScheduler::new(config)
    .with_sla_lane(SlaLane::Interactive, Duration::from_millis(100))
    .with_sla_lane(SlaLane::Batch, Duration::from_secs(5));

// Requests are automatically batched and prioritized
scheduler.submit(request).await?;

Features

• Iteration-level preemption for priority requests
• Chunked prefill for reduced time-to-first-token
• Multi-LoRA serving in a single batch
• Automatic routing based on context length

Performance

Max Batch Size	Requests/sec	P99 Latency	GPU Util
32	45	85ms	78%
64	82	95ms	89%
128	140	120ms	94%

Next Steps

• KV-Cache Optimization
• Quantization