fix(production): wire real inference path through engine, serving, and streaming
- rtx-inference: sample_next_token now copies the actual logits from the forward pass (Tensor::to_vec, last-token slice) instead of sampling from a fabricated all-zero vector; request metrics report measured queue/processing times instead of hardcoded constants. - rtx-serving-api: depends on rtx-inference; /v1/completions dispatches to a shared InferenceEngine (byte-level tokenization until a real tokenizer is threaded through) and returns 503 when no engine is loaded instead of mock text. ServingServer::with_engine attaches one. - rtx-streaming: depends on rtx-inference; generate_tokens delegates to an attached backend engine and errors without one instead of emitting "token_N" placeholders; tokenization is byte-level, not position-mod. - speculative decoding: document the orchestration (speculative/) vs implementation (medusa.rs/lookahead.rs) layering; CLAUDE.md no longer claims a standalone rtx-speculative-decoding crate. Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
@@ -9,6 +9,19 @@
|
||||
//! - Configurable acceptance thresholds and draft lengths
|
||||
//! - Performance tracking and adaptive optimization
|
||||
//! - Safe error handling throughout the pipeline
|
||||
//!
|
||||
//! # Layering
|
||||
//!
|
||||
//! This module is the *orchestration* layer: strategy traits
|
||||
//! ([`MedusaDraftModel`], [`SelfSpeculativeModel`]), configs, candidate
|
||||
//! trees, and the [`AdvancedSpeculativeDecoder`] that combines them. The
|
||||
//! concrete numeric implementations live at the crate root:
|
||||
//! [`crate::medusa`] (real multi-head FFN prediction over hidden states,
|
||||
//! arXiv:2401.10774) and [`crate::lookahead`] (n-gram lookahead decoding).
|
||||
//! The configs here (`MedusaConfig`, `LookaheadConfig`) parameterize the
|
||||
//! orchestration strategies; they are distinct from the implementation
|
||||
//! configs (`crate::medusa::MedusaConfig` is re-exported from the crate
|
||||
//! root as `MedusaHeadsConfig` to disambiguate).
|
||||
|
||||
mod advanced;
|
||||
mod config;
|
||||
|
||||
Reference in New Issue
Block a user