fix(production): wire real inference path through engine, serving, and streaming

- rtx-inference: sample_next_token now copies the actual logits from the
  forward pass (Tensor::to_vec, last-token slice) instead of sampling
  from a fabricated all-zero vector; request metrics report measured
  queue/processing times instead of hardcoded constants.
- rtx-serving-api: depends on rtx-inference; /v1/completions dispatches
  to a shared InferenceEngine (byte-level tokenization until a real
  tokenizer is threaded through) and returns 503 when no engine is
  loaded instead of mock text. ServingServer::with_engine attaches one.
- rtx-streaming: depends on rtx-inference; generate_tokens delegates to
  an attached backend engine and errors without one instead of emitting
  "token_N" placeholders; tokenization is byte-level, not position-mod.
- speculative decoding: document the orchestration (speculative/) vs
  implementation (medusa.rs/lookahead.rs) layering; CLAUDE.md no longer
  claims a standalone rtx-speculative-decoding crate.

Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
osobh
2026-07-09 19:05:27 -07:00
co-authored by Claude Fable 5
parent 522400a72b
commit 64ade03ab9
9 changed files with 179 additions and 81 deletions
@@ -9,6 +9,19 @@
//! - Configurable acceptance thresholds and draft lengths
//! - Performance tracking and adaptive optimization
//! - Safe error handling throughout the pipeline
//!
//! # Layering
//!
//! This module is the *orchestration* layer: strategy traits
//! ([`MedusaDraftModel`], [`SelfSpeculativeModel`]), configs, candidate
//! trees, and the [`AdvancedSpeculativeDecoder`] that combines them. The
//! concrete numeric implementations live at the crate root:
//! [`crate::medusa`] (real multi-head FFN prediction over hidden states,
//! arXiv:2401.10774) and [`crate::lookahead`] (n-gram lookahead decoding).
//! The configs here (`MedusaConfig`, `LookaheadConfig`) parameterize the
//! orchestration strategies; they are distinct from the implementation
//! configs (`crate::medusa::MedusaConfig` is re-exported from the crate
//! root as `MedusaHeadsConfig` to disambiguate).
mod advanced;
mod config;