bench: make the CUDA embedding path discoverable when it is unavailable
CI / test (push) Failing after 2s

The GPU path worked but was effectively hidden. cudarc's build script shells out
to `nvcc`, which ships in /usr/local/cuda/bin — a directory the reference host
had installed but never exported to the login shell, so `--features
embeddings-cuda` failed with a bare "`nvcc --version` failed" panic from a
dependency's build script, and the runtime fallback then reported only
"Embedder: CPU (...)" before spending hours on work a GPU does in minutes.

Two changes, both about making the failure legible rather than changing what the
code does:

  - The CPU fallback now says why it fell back and what that costs, with the
    concrete fix. A run that silently takes two orders of magnitude longer reads
    as a hang, not as a configuration choice.
  - BENCHMARKS.md states the build-time nvcc requirement, where the toolkit
    actually installs, and that a shell file read non-interactively is the place
    to export it — `~/.zshenv` rather than `~/.zshrc`, because build scripts do
    not run in an interactive shell.

Host-side, the reference machine's CUDA exports lived in ~/.bashrc below its
non-interactive guard while the login shell is zsh, so they never applied to
anything. Moved to ~/.zshenv with duplicate-prepend guards; `nvcc --version`
and `cargo build --features embeddings-cuda` now both work over a plain
non-interactive ssh with no manual export.
This commit is contained in:
Omar Sobh
2026-08-07 08:13:59 -07:00
parent c913cd1cbf
commit 12d9d8462f
2 changed files with 18 additions and 3 deletions
+11 -2
View File
@@ -300,8 +300,17 @@ matching, and MiniLM at 384 dimensions is a small embedding model.
> **Run:** `cargo run --release --bin longmemeval_bench --features embeddings -- \ > **Run:** `cargo run --release --bin longmemeval_bench --features embeddings -- \
> benchmarks/longmemeval/longmemeval_s_cleaned.json --embeddings weights/all-minilm-l6-v2` > benchmarks/longmemeval/longmemeval_s_cleaned.json --embeddings weights/all-minilm-l6-v2`
> Add `--features embeddings-cuda` (and put `nvcc` on `PATH`) for the GPU path. > For the GPU path use `--features embeddings-cuda`. That requires `nvcc` on
> Weights: `huggingface.co/sentence-transformers/all-MiniLM-L6-v2`. > `PATH` at *build* time — cudarc's build script shells out to it. The toolkit
> installs to `/usr/local/cuda/bin`, which many distributions do not export;
> check with `nvcc --version` and, if it is missing, add it somewhere every
> shell reads (for zsh that is `~/.zshenv`, not `~/.zshrc`, since build tooling
> runs non-interactively). The device is selected at runtime with a CPU
> fallback, so a machine without CUDA still produces correct numbers — just far
> more slowly, and the bench says so on startup.
>
> Weights: `huggingface.co/sentence-transformers/all-MiniLM-L6-v2` — place
> `model.safetensors` and `tokenizer.json` in the `--embeddings` directory.
### Oracle variant — `longmemeval_oracle`, n=500 (easier corpus, kept for continuity) ### Oracle variant — `longmemeval_oracle`, n=500 (easier corpus, kept for continuity)
@@ -43,7 +43,13 @@ impl Embedder {
d d
} }
Err(e) => { Err(e) => {
eprintln!("Embedder: CPU ({e})"); // Loud, because the CPU path is correct but ~100x slower: the
// full longmemeval_s haystack is minutes on a GPU and most of a
// day on 8 cores. Silently falling back looks like a hang.
eprintln!("Embedder: CPU — CUDA unavailable ({e})");
eprintln!(
" WARNING: CPU embedding is roughly two orders of magnitude slower.\n Expect minutes for longmemeval_oracle and many hours for the full\n longmemeval_s haystack. For the GPU path, rebuild with\n `--features embeddings-cuda` and make sure `nvcc` is on PATH\n (it ships in /usr/local/cuda/bin, which is often not exported)."
);
Device::Cpu Device::Cpu
} }
}; };