feat(demos,inference): wire simulation demos to real compute; fix embedding lookup and weight-name aliases
GPU Tests / Check GPU Availability (push) Successful in 0s
GPU Tests / Metal Tests (push) Has been skipped
CI / Format Check (push) Failing after 6s
CI / Clippy Check (push) Failing after 7s
Performance Benchmarks / Run Benchmarks (push) Failing after 7s
GPU Tests / CUDA Tests (11.8) (push) Has been skipped
GPU Tests / CUDA Tests (12.1) (push) Has been skipped
CI / Build (ubuntu-latest) (push) Failing after 7s
Documentation / Build User Guide (push) Successful in 8s
CI / Build (macos-latest) (push) Failing after 9s
CI / Test (macos-latest) (push) Has been skipped
CI / Test (ubuntu-latest) (push) Has been skipped
CI / Python Bindings (maturin) (macos-latest) (push) Has been skipped
CI / Python Bindings (maturin) (ubuntu-latest) (push) Has been skipped
CI / WASM Build + Size Check (push) Has been skipped
CI / Distributed Training Tests (push) Has been skipped
CI / Build CPU-Only (Explicit) (push) Failing after 43s
Documentation / Build API Documentation (push) Failing after 48s
CI / CI Success (push) Failing after 0s

Demos:
- rtx-distllm-demo: real rtx-tensor weights per shard, real
  scaled-dot-product attention forward, metrics measured (Instant)
  instead of hardcoded constants; network topology remains a documented
  simulation fed by real tensor byte sizes.
- rtx-model-zoo: MockInferenceEngine deleted; RealInferenceEngine loads
  a tiny real transformer into rtx_inference::InferenceEngine and runs
  genuine engine.infer per request; domain outputs are explicitly-
  labeled toy proxies derived from real output tokens.
- rtx-inference-profiler: mock models deleted; profiles real
  matmul/softmax pipelines on rtx-tensor with measured latency/memory.

Inference-path bugs the demos surfaced (fixed here):
- ForwardPass::apply_embedding misused Tensor::gather for the embedding
  lookup — gather returns the indices' shape, silently dropping the
  hidden dim and breaking every downstream broadcast. Now uses the
  existing Tensor::embedding_lookup ([vocab,hidden] x [batch,seq] ->
  [batch,seq,hidden]).
- Attention weight lookup accepts both self_attn. (HF-LLaMA) and
  attention. prefixes; final layer norm accepts norm.weight /
  model.norm.weight / ln_f.weight aliases.
- Integration fixture gains the final norm weight; the previously
  always-failing engine tests now pass (8/8 model_loading_test).

End-to-end inference through the real engine now works for the first
time — verified via model_zoo_demo producing real forward-pass outputs
across all categories.

Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
osobh
2026-07-09 22:06:29 -07:00
co-authored by Claude Fable 5
parent 733b02cd8b
commit e080748d88
18 changed files with 1390 additions and 754 deletions
@@ -0,0 +1,73 @@
//! End-to-end demo: drives the real `rtx-inference` engine through `RealInferenceEngine`
//! for a couple of `ModelCategory` variants and prints the resulting
//! `InferenceResult` (real measured latency, real device, real-compute-backed output).
use model_zoo_shared::{InferenceRequest, ModelCategory};
use rtx_model_zoo::RealInferenceEngine;
fn main() {
println!("Building RealInferenceEngine (loads a tiny real transformer)...");
let engine = RealInferenceEngine::new().expect("failed to build RealInferenceEngine");
println!("Engine ready.\n");
match engine.model_info() {
Ok(info) => {
println!("Real loaded model metadata (from the real rtx-inference engine):");
println!(" name: {}", info.name);
println!(" parameter_count: {}", info.parameter_count);
println!(" memory_usage: {} bytes", info.memory_usage);
println!(" layer_count: {}", info.layer_count);
println!();
}
Err(e) => println!("model_info() failed: {e}\n"),
}
let cases = [
(
"gpt2_small",
"Machine learning",
"text/plain",
ModelCategory::TextGeneration,
),
(
"resnet50",
"base64encodedimage",
"image/jpeg",
ModelCategory::ImageClassification,
),
(
"yolov8n",
"base64encodedimage",
"image/jpeg",
ModelCategory::ObjectDetection,
),
(
"bert_base_uncased",
"This is a great product",
"text/plain",
ModelCategory::NLP,
),
];
for (model_id, input_data, input_type, category) in cases {
let request = InferenceRequest {
model_id: model_id.to_string(),
input_data: input_data.to_string(),
input_type: input_type.to_string(),
};
println!("category: {category:?}");
match engine.run_inference(&request, category) {
Ok(result) => {
println!(" model_id: {}", result.model_id);
println!(" device: {}", result.device);
println!(" inference_time_ms: {:.4}", result.inference_time_ms);
println!(" output: {}", result.output);
}
Err(e) => {
println!(" real engine attempt failed (see module docs for known upstream bug):");
println!(" {e}");
}
}
println!();
}
}