feat(demos,inference): wire simulation demos to real compute; fix embedding lookup and weight-name aliases
GPU Tests / Check GPU Availability (push) Successful in 0s
GPU Tests / Metal Tests (push) Has been skipped
CI / Format Check (push) Failing after 6s
CI / Clippy Check (push) Failing after 7s
Performance Benchmarks / Run Benchmarks (push) Failing after 7s
GPU Tests / CUDA Tests (11.8) (push) Has been skipped
GPU Tests / CUDA Tests (12.1) (push) Has been skipped
CI / Build (ubuntu-latest) (push) Failing after 7s
Documentation / Build User Guide (push) Successful in 8s
CI / Build (macos-latest) (push) Failing after 9s
CI / Test (macos-latest) (push) Has been skipped
CI / Test (ubuntu-latest) (push) Has been skipped
CI / Python Bindings (maturin) (macos-latest) (push) Has been skipped
CI / Python Bindings (maturin) (ubuntu-latest) (push) Has been skipped
CI / WASM Build + Size Check (push) Has been skipped
CI / Distributed Training Tests (push) Has been skipped
CI / Build CPU-Only (Explicit) (push) Failing after 43s
Documentation / Build API Documentation (push) Failing after 48s
CI / CI Success (push) Failing after 0s
GPU Tests / Check GPU Availability (push) Successful in 0s
GPU Tests / Metal Tests (push) Has been skipped
CI / Format Check (push) Failing after 6s
CI / Clippy Check (push) Failing after 7s
Performance Benchmarks / Run Benchmarks (push) Failing after 7s
GPU Tests / CUDA Tests (11.8) (push) Has been skipped
GPU Tests / CUDA Tests (12.1) (push) Has been skipped
CI / Build (ubuntu-latest) (push) Failing after 7s
Documentation / Build User Guide (push) Successful in 8s
CI / Build (macos-latest) (push) Failing after 9s
CI / Test (macos-latest) (push) Has been skipped
CI / Test (ubuntu-latest) (push) Has been skipped
CI / Python Bindings (maturin) (macos-latest) (push) Has been skipped
CI / Python Bindings (maturin) (ubuntu-latest) (push) Has been skipped
CI / WASM Build + Size Check (push) Has been skipped
CI / Distributed Training Tests (push) Has been skipped
CI / Build CPU-Only (Explicit) (push) Failing after 43s
Documentation / Build API Documentation (push) Failing after 48s
CI / CI Success (push) Failing after 0s
Demos: - rtx-distllm-demo: real rtx-tensor weights per shard, real scaled-dot-product attention forward, metrics measured (Instant) instead of hardcoded constants; network topology remains a documented simulation fed by real tensor byte sizes. - rtx-model-zoo: MockInferenceEngine deleted; RealInferenceEngine loads a tiny real transformer into rtx_inference::InferenceEngine and runs genuine engine.infer per request; domain outputs are explicitly- labeled toy proxies derived from real output tokens. - rtx-inference-profiler: mock models deleted; profiles real matmul/softmax pipelines on rtx-tensor with measured latency/memory. Inference-path bugs the demos surfaced (fixed here): - ForwardPass::apply_embedding misused Tensor::gather for the embedding lookup — gather returns the indices' shape, silently dropping the hidden dim and breaking every downstream broadcast. Now uses the existing Tensor::embedding_lookup ([vocab,hidden] x [batch,seq] -> [batch,seq,hidden]). - Attention weight lookup accepts both self_attn. (HF-LLaMA) and attention. prefixes; final layer norm accepts norm.weight / model.norm.weight / ln_f.weight aliases. - Integration fixture gains the final norm weight; the previously always-failing engine tests now pass (8/8 model_loading_test). End-to-end inference through the real engine now works for the first time — verified via model_zoo_demo producing real forward-pass outputs across all categories. Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
@@ -0,0 +1,26 @@
|
||||
//! Runnable entry point for the DistributedLLM demo.
|
||||
//!
|
||||
//! Prints the `InferenceResult` from `rtx_distllm_demo::run_demo()`,
|
||||
//! including real measured latencies from actual tensor compute (attention
|
||||
//! matmul/softmax) mixed with the simulated network/tokenizer pieces
|
||||
//! documented in `rtx_distllm_demo`'s crate-level docs.
|
||||
|
||||
fn main() {
|
||||
let result = rtx_distllm_demo::run_demo();
|
||||
|
||||
println!("=== DistributedLLM Demo ===");
|
||||
println!("Prompt tokens: {}", result.prompt_tokens);
|
||||
println!("Completion tokens: {}", result.completion_tokens);
|
||||
println!("Finish reason: {:?}", result.finish_reason);
|
||||
println!();
|
||||
println!("Total latency (ms): {:.3}", result.latency_ms);
|
||||
println!("Time to first token (ms): {:.3}", result.time_to_first_token_ms);
|
||||
println!("Tokens/sec: {:.3}", result.tokens_per_second);
|
||||
println!(
|
||||
"Per-token latencies (ms): {:?}",
|
||||
&result.token_latencies_ms[..result.token_latencies_ms.len().min(10)]
|
||||
);
|
||||
println!();
|
||||
println!("Generated text:");
|
||||
println!("{}", result.text);
|
||||
}
|
||||
Reference in New Issue
Block a user