feat(demos,inference): wire simulation demos to real compute; fix embedding lookup and weight-name aliases
GPU Tests / Check GPU Availability (push) Successful in 0s
GPU Tests / Metal Tests (push) Has been skipped
CI / Format Check (push) Failing after 6s
CI / Clippy Check (push) Failing after 7s
Performance Benchmarks / Run Benchmarks (push) Failing after 7s
GPU Tests / CUDA Tests (11.8) (push) Has been skipped
GPU Tests / CUDA Tests (12.1) (push) Has been skipped
CI / Build (ubuntu-latest) (push) Failing after 7s
Documentation / Build User Guide (push) Successful in 8s
CI / Build (macos-latest) (push) Failing after 9s
CI / Test (macos-latest) (push) Has been skipped
CI / Test (ubuntu-latest) (push) Has been skipped
CI / Python Bindings (maturin) (macos-latest) (push) Has been skipped
CI / Python Bindings (maturin) (ubuntu-latest) (push) Has been skipped
CI / WASM Build + Size Check (push) Has been skipped
CI / Distributed Training Tests (push) Has been skipped
CI / Build CPU-Only (Explicit) (push) Failing after 43s
Documentation / Build API Documentation (push) Failing after 48s
CI / CI Success (push) Failing after 0s

Demos:
- rtx-distllm-demo: real rtx-tensor weights per shard, real
  scaled-dot-product attention forward, metrics measured (Instant)
  instead of hardcoded constants; network topology remains a documented
  simulation fed by real tensor byte sizes.
- rtx-model-zoo: MockInferenceEngine deleted; RealInferenceEngine loads
  a tiny real transformer into rtx_inference::InferenceEngine and runs
  genuine engine.infer per request; domain outputs are explicitly-
  labeled toy proxies derived from real output tokens.
- rtx-inference-profiler: mock models deleted; profiles real
  matmul/softmax pipelines on rtx-tensor with measured latency/memory.

Inference-path bugs the demos surfaced (fixed here):
- ForwardPass::apply_embedding misused Tensor::gather for the embedding
  lookup — gather returns the indices' shape, silently dropping the
  hidden dim and breaking every downstream broadcast. Now uses the
  existing Tensor::embedding_lookup ([vocab,hidden] x [batch,seq] ->
  [batch,seq,hidden]).
- Attention weight lookup accepts both self_attn. (HF-LLaMA) and
  attention. prefixes; final layer norm accepts norm.weight /
  model.norm.weight / ln_f.weight aliases.
- Integration fixture gains the final norm weight; the previously
  always-failing engine tests now pass (8/8 model_loading_test).

End-to-end inference through the real engine now works for the first
time — verified via model_zoo_demo producing real forward-pass outputs
across all categories.

Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
osobh
2026-07-09 22:06:29 -07:00
co-authored by Claude Fable 5
parent 733b02cd8b
commit e080748d88
18 changed files with 1390 additions and 754 deletions
@@ -0,0 +1,26 @@
//! Runnable entry point for the DistributedLLM demo.
//!
//! Prints the `InferenceResult` from `rtx_distllm_demo::run_demo()`,
//! including real measured latencies from actual tensor compute (attention
//! matmul/softmax) mixed with the simulated network/tokenizer pieces
//! documented in `rtx_distllm_demo`'s crate-level docs.
fn main() {
let result = rtx_distllm_demo::run_demo();
println!("=== DistributedLLM Demo ===");
println!("Prompt tokens: {}", result.prompt_tokens);
println!("Completion tokens: {}", result.completion_tokens);
println!("Finish reason: {:?}", result.finish_reason);
println!();
println!("Total latency (ms): {:.3}", result.latency_ms);
println!("Time to first token (ms): {:.3}", result.time_to_first_token_ms);
println!("Tokens/sec: {:.3}", result.tokens_per_second);
println!(
"Per-token latencies (ms): {:?}",
&result.token_latencies_ms[..result.token_latencies_ms.len().min(10)]
);
println!();
println!("Generated text:");
println!("{}", result.text);
}