feat(demos,inference): wire simulation demos to real compute; fix embedding lookup and weight-name aliases
GPU Tests / Check GPU Availability (push) Successful in 0s
GPU Tests / Metal Tests (push) Has been skipped
CI / Format Check (push) Failing after 6s
CI / Clippy Check (push) Failing after 7s
Performance Benchmarks / Run Benchmarks (push) Failing after 7s
GPU Tests / CUDA Tests (11.8) (push) Has been skipped
GPU Tests / CUDA Tests (12.1) (push) Has been skipped
CI / Build (ubuntu-latest) (push) Failing after 7s
Documentation / Build User Guide (push) Successful in 8s
CI / Build (macos-latest) (push) Failing after 9s
CI / Test (macos-latest) (push) Has been skipped
CI / Test (ubuntu-latest) (push) Has been skipped
CI / Python Bindings (maturin) (macos-latest) (push) Has been skipped
CI / Python Bindings (maturin) (ubuntu-latest) (push) Has been skipped
CI / WASM Build + Size Check (push) Has been skipped
CI / Distributed Training Tests (push) Has been skipped
CI / Build CPU-Only (Explicit) (push) Failing after 43s
Documentation / Build API Documentation (push) Failing after 48s
CI / CI Success (push) Failing after 0s
GPU Tests / Check GPU Availability (push) Successful in 0s
GPU Tests / Metal Tests (push) Has been skipped
CI / Format Check (push) Failing after 6s
CI / Clippy Check (push) Failing after 7s
Performance Benchmarks / Run Benchmarks (push) Failing after 7s
GPU Tests / CUDA Tests (11.8) (push) Has been skipped
GPU Tests / CUDA Tests (12.1) (push) Has been skipped
CI / Build (ubuntu-latest) (push) Failing after 7s
Documentation / Build User Guide (push) Successful in 8s
CI / Build (macos-latest) (push) Failing after 9s
CI / Test (macos-latest) (push) Has been skipped
CI / Test (ubuntu-latest) (push) Has been skipped
CI / Python Bindings (maturin) (macos-latest) (push) Has been skipped
CI / Python Bindings (maturin) (ubuntu-latest) (push) Has been skipped
CI / WASM Build + Size Check (push) Has been skipped
CI / Distributed Training Tests (push) Has been skipped
CI / Build CPU-Only (Explicit) (push) Failing after 43s
Documentation / Build API Documentation (push) Failing after 48s
CI / CI Success (push) Failing after 0s
Demos: - rtx-distllm-demo: real rtx-tensor weights per shard, real scaled-dot-product attention forward, metrics measured (Instant) instead of hardcoded constants; network topology remains a documented simulation fed by real tensor byte sizes. - rtx-model-zoo: MockInferenceEngine deleted; RealInferenceEngine loads a tiny real transformer into rtx_inference::InferenceEngine and runs genuine engine.infer per request; domain outputs are explicitly- labeled toy proxies derived from real output tokens. - rtx-inference-profiler: mock models deleted; profiles real matmul/softmax pipelines on rtx-tensor with measured latency/memory. Inference-path bugs the demos surfaced (fixed here): - ForwardPass::apply_embedding misused Tensor::gather for the embedding lookup — gather returns the indices' shape, silently dropping the hidden dim and breaking every downstream broadcast. Now uses the existing Tensor::embedding_lookup ([vocab,hidden] x [batch,seq] -> [batch,seq,hidden]). - Attention weight lookup accepts both self_attn. (HF-LLaMA) and attention. prefixes; final layer norm accepts norm.weight / model.norm.weight / ln_f.weight aliases. - Integration fixture gains the final norm weight; the previously always-failing engine tests now pass (8/8 model_loading_test). End-to-end inference through the real engine now works for the first time — verified via model_zoo_demo producing real forward-pass outputs across all categories. Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
@@ -0,0 +1,73 @@
|
||||
//! End-to-end demo: drives the real `rtx-inference` engine through `RealInferenceEngine`
|
||||
//! for a couple of `ModelCategory` variants and prints the resulting
|
||||
//! `InferenceResult` (real measured latency, real device, real-compute-backed output).
|
||||
|
||||
use model_zoo_shared::{InferenceRequest, ModelCategory};
|
||||
use rtx_model_zoo::RealInferenceEngine;
|
||||
|
||||
fn main() {
|
||||
println!("Building RealInferenceEngine (loads a tiny real transformer)...");
|
||||
let engine = RealInferenceEngine::new().expect("failed to build RealInferenceEngine");
|
||||
println!("Engine ready.\n");
|
||||
|
||||
match engine.model_info() {
|
||||
Ok(info) => {
|
||||
println!("Real loaded model metadata (from the real rtx-inference engine):");
|
||||
println!(" name: {}", info.name);
|
||||
println!(" parameter_count: {}", info.parameter_count);
|
||||
println!(" memory_usage: {} bytes", info.memory_usage);
|
||||
println!(" layer_count: {}", info.layer_count);
|
||||
println!();
|
||||
}
|
||||
Err(e) => println!("model_info() failed: {e}\n"),
|
||||
}
|
||||
|
||||
let cases = [
|
||||
(
|
||||
"gpt2_small",
|
||||
"Machine learning",
|
||||
"text/plain",
|
||||
ModelCategory::TextGeneration,
|
||||
),
|
||||
(
|
||||
"resnet50",
|
||||
"base64encodedimage",
|
||||
"image/jpeg",
|
||||
ModelCategory::ImageClassification,
|
||||
),
|
||||
(
|
||||
"yolov8n",
|
||||
"base64encodedimage",
|
||||
"image/jpeg",
|
||||
ModelCategory::ObjectDetection,
|
||||
),
|
||||
(
|
||||
"bert_base_uncased",
|
||||
"This is a great product",
|
||||
"text/plain",
|
||||
ModelCategory::NLP,
|
||||
),
|
||||
];
|
||||
|
||||
for (model_id, input_data, input_type, category) in cases {
|
||||
let request = InferenceRequest {
|
||||
model_id: model_id.to_string(),
|
||||
input_data: input_data.to_string(),
|
||||
input_type: input_type.to_string(),
|
||||
};
|
||||
println!("category: {category:?}");
|
||||
match engine.run_inference(&request, category) {
|
||||
Ok(result) => {
|
||||
println!(" model_id: {}", result.model_id);
|
||||
println!(" device: {}", result.device);
|
||||
println!(" inference_time_ms: {:.4}", result.inference_time_ms);
|
||||
println!(" output: {}", result.output);
|
||||
}
|
||||
Err(e) => {
|
||||
println!(" real engine attempt failed (see module docs for known upstream bug):");
|
||||
println!(" {e}");
|
||||
}
|
||||
}
|
||||
println!();
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user