docs(uno-q): measure on-board inference; retire the LiteRT spike
Replaces the never-executed "LiteRT-LM on UNO Q 4GB" spike plan with a measured record of the local-fallback path we actually run. Measured on board 65301572 (Qwen2.5-0.5B-Instruct, -c 8192, 4 threads), using llama-server's own timings rather than wall clock: prefill ~17-20 tok/s (linear), decode ~6-11 tok/s (degrades with KV) warm prefix-cached tool call: 3.8s, 6/6 correct structured calls Two findings that changed the deployment: 1. The board had drifted onto Qwen2.5-Coder-1.5B - larger and tuned for the wrong task. Reverting to the repo's 0.5B made tool calls ~6x faster (24s -> 3.8s) and freed ~700MB. The repo was right. 2. The harness, not the model, was the bottleneck. The default agent profile sent a 4718-token prompt (~4.6 min prefill) and the client cancelled before the model could answer. A lean runtime profile cuts that to 706 tokens, lifts prefix-cache match 0.435 -> 0.966, and completes a full agentic turn with a real tool call in 11s warm. The ZeroClaw text parser was never at fault. Prompt cost model for budgeting profiles: ~706 base (1 tool), ~244/additional tool, +315 for uno_q_flash (schema + flash imperative), ~53/skill in compact mode. Also standardises context on -c 8192 across all three provisioning paths (a 16k window costs ~16 min to fill at this speed and doubles KV for nothing), and fixes stale references to the deleted src/lib/harness.ts. Adds bench-prefill.sh and bench-tools.py as reproducible baselines. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
e2ce67f069
commit
a23a525aef
@@ -0,0 +1,305 @@
|
|||||||
|
# Local fallback — what we run, and why not LiteRT
|
||||||
|
|
||||||
|
How an APESS Uno Q node answers when the cloud is slow, rate-limited, or gone.
|
||||||
|
Supersedes the *"LiteRT-LM on UNO Q 4GB — Technical Spike Plan v1.0"* (July 2026),
|
||||||
|
which was never executed. See **Why not LiteRT** for the decision, and
|
||||||
|
**Lessons carried forward** for the design constraints it got wrong.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. What we actually run
|
||||||
|
|
||||||
|
**Cloud-first, on-board Qwen as fallback.** The local model is a safety net for a
|
||||||
|
guaranteed-offline condition — not the primary reasoner.
|
||||||
|
|
||||||
|
### Agent → provider routing
|
||||||
|
|
||||||
|
From [`config.template.toml`](./config.template.toml):
|
||||||
|
|
||||||
|
| alias | provider | behaviour |
|
||||||
|
|---|---|---|
|
||||||
|
| `default` | `custom.claude` | cloud first, `fallback = ["llamacpp.local"]` — the workshop default |
|
||||||
|
| `cloud` | `custom.cloud` | cloud only, no fallback |
|
||||||
|
| `local` | `llamacpp.local` | on-board Qwen only (fully offline) |
|
||||||
|
| `chaos` | `custom.dead` | points at a dead `:9099` so it *always* fails over — the outage demo |
|
||||||
|
|
||||||
|
`chaos` exists to make resilience demonstrable on command: nothing listens on
|
||||||
|
`127.0.0.1:9099`, so any prompt routed there is guaranteed to fall through to the
|
||||||
|
local model. It is the only honest way to show failover without unplugging WiFi.
|
||||||
|
|
||||||
|
The `fallback = ["llamacpp.local"]` array on `custom.claude` and `custom.dead` is the
|
||||||
|
**only** fallback configured on the board. There is no multi-cloud chain locally.
|
||||||
|
|
||||||
|
### Local serving
|
||||||
|
|
||||||
|
From [`systemd/zeroclaw-llama.service`](./systemd/zeroclaw-llama.service):
|
||||||
|
|
||||||
|
```
|
||||||
|
llama-server -m /home/arduino/models/qwen.gguf \
|
||||||
|
--host 127.0.0.1 --port 8083 -np 1 -c 8192 --jinja --mlock
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Persistent server, loopback-only.** The model is loaded once and stays resident.
|
||||||
|
It is never spawned per request.
|
||||||
|
- **`-np 1` is mandatory** — a larger slot count makes the tiny model 500 on context.
|
||||||
|
- **`--mlock`** pins the weights so they aren't swapped out under memory pressure.
|
||||||
|
- Health at `:8083/health`. The ZeroClaw daemon is on `:8080` and is ordered
|
||||||
|
`After=zeroclaw-llama.service`.
|
||||||
|
|
||||||
|
> ### Config drift — repo vs. dev board (2026-07-19)
|
||||||
|
>
|
||||||
|
> The dev board (serial 65301572) had been hand-edited away from the repo and never
|
||||||
|
> propagated back. Benchmarking settled it: **the repo was right.**
|
||||||
|
>
|
||||||
|
> | | Repo before | Board before | Both now |
|
||||||
|
> |---|---|---|---|
|
||||||
|
> | Model | `qwen.gguf` | `qwen-coder.gguf` (1.1 GB) | ✅ `qwen.gguf` (409 MB) |
|
||||||
|
> | Context | `-c 16384` | `-c 8192` | ✅ `-c 8192` |
|
||||||
|
>
|
||||||
|
> **Model — resolved in the repo's favour.** The board ran Qwen2.5-**Coder**-1.5B, which is
|
||||||
|
> both larger and specialized for the wrong task. Swapping to Qwen2.5-0.5B-Instruct made tool
|
||||||
|
> calls **~6× faster** (24 s → 3.8 s warm) at **6/6 correctness**, and freed ~700 MB.
|
||||||
|
> A pre-swap backup sits at `~/zeroclaw-supervisor.sh.bak-preswap` on the board.
|
||||||
|
>
|
||||||
|
> **Context — resolved in the board's favour: standardised on `-c 8192`** across
|
||||||
|
> `zeroclaw-llama.service`, `zeroclaw-supervisor.sh`, and `provision-uno-q.sh`. At ~17 tok/s
|
||||||
|
> prefill, filling even an 8k window costs ~8 minutes, so a 16k window was capacity that
|
||||||
|
> could never be afforded and doubled KV footprint for nothing.
|
||||||
|
>
|
||||||
|
> **Gotcha for whoever changes this:** the supervisor reads `MODEL` into a shell variable
|
||||||
|
> at launch. Editing the file does **not** affect the running process — you must restart
|
||||||
|
> the supervisor itself, not just llama-server, or it will relaunch the old model.
|
||||||
|
|
||||||
|
### Keeping it alive
|
||||||
|
|
||||||
|
[`zeroclaw-supervisor.sh`](./zeroclaw-supervisor.sh) is the no-root watchdog for boards
|
||||||
|
where systemd isn't usable. Two properties matter:
|
||||||
|
|
||||||
|
- It polls **`/health` endpoints, not `pgrep`** — a wedged process passes `pgrep` but
|
||||||
|
fails a health check. Process liveness is not service liveness.
|
||||||
|
- It respects a **warmup grace** (`LLAMA_WARMUP=300`). A cold GGUF load takes **3–5
|
||||||
|
minutes**; reaping mid-load produces an infinite restart loop that never converges.
|
||||||
|
|
||||||
|
Children start under `setsid` so they survive the launching shell closing — plain
|
||||||
|
`nohup … &` inside an `adb shell` does **not** give you that, which is why services
|
||||||
|
died on a previous USB disconnect.
|
||||||
|
|
||||||
|
### Workshop-proxy variant
|
||||||
|
|
||||||
|
[`../workshop-llm/zeroclaw-board.toml`](../workshop-llm/zeroclaw-board.toml) points the
|
||||||
|
`cloud` agent at a LiteLLM proxy on `:4000`. The board asks for one model name
|
||||||
|
(`workshop`) and never sees the routing: the proxy pools Kimi + GLM and falls back to
|
||||||
|
Groq → Gemini. **Cloud-side fallback lives in `litellm-config.yaml`, not on the board.**
|
||||||
|
|
||||||
|
### Secrets
|
||||||
|
|
||||||
|
Provider config is **file-based**; the only `ZEROCLAW_`-prefixed variable in the repo is
|
||||||
|
`ZEROCLAW_BIN` (a binary path). The Claude Max setup-token is **env-only at runtime and
|
||||||
|
must never be written to disk or committed** — not into `config.toml`, not into a unit
|
||||||
|
file, not into a provisioning script. Same for attendee virtual keys and paired tokens.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Why not LiteRT
|
||||||
|
|
||||||
|
The spike proposed LiteRT-LM + FunctionGemma 270M, invoked by shelling out to the
|
||||||
|
`litert-lm` CLI per classification. We did not pursue it:
|
||||||
|
|
||||||
|
- **Subprocess-per-call reloads the model every request.** On a quad-A53 that cost
|
||||||
|
dominates the entire latency budget. We already had a persistent server working.
|
||||||
|
- **Fleet provisioning burden.** It required an export pipeline (`litert-torch`, run
|
||||||
|
off-board) plus a per-board `huggingface-cli login` with a personal token.
|
||||||
|
- **The strategy inverted.** The spike framed the local model as a primary classifier.
|
||||||
|
The shipped decision is cloud-default with local reserved for guaranteed-offline —
|
||||||
|
so the bar the spike was measuring against stopped being the bar.
|
||||||
|
- **Its premise was pre-pivot.** Every prompt and pass gate assumed
|
||||||
|
`nominal`/`anomalous`/`critical` structural-health classification. The workshop now
|
||||||
|
teaches domain-driven design (Domain → Skills → Policies → Harness → Loops) over an
|
||||||
|
open, team-chosen domain, so a fixed three-label classifier is not the target.
|
||||||
|
|
||||||
|
None of this says LiteRT is a bad runtime. It says it solved a problem we'd already
|
||||||
|
solved, in a shape that cost more to operate.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Measured baseline
|
||||||
|
|
||||||
|
**Measured 2026-07-19** on board 65301572 at `-c 8192`, warm, 4 threads
|
||||||
|
(NEON + ARM_FMA + OPENMP + REPACK, `n_batch=2048`). Figures come from llama-server's own
|
||||||
|
`timings` block, not wall clock. Prefix caching was **deliberately defeated** with a unique
|
||||||
|
leading nonce, so these are honest uncached numbers — the cached case is far better and is
|
||||||
|
what real workloads see (see below).
|
||||||
|
|
||||||
|
Two models compared: **`qwen.gguf`** = Qwen2.5-0.5B-Instruct (630 M params, 24 layers,
|
||||||
|
n_embd 896) and **`qwen-coder.gguf`** = Qwen2.5-Coder-1.5B-Instruct (1.78 B, 28 layers,
|
||||||
|
n_embd 1536).
|
||||||
|
|
||||||
|
| prompt tok | 0.5B prefill | 1.5B-coder prefill | 0.5B decode | coder decode |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| ~260 | **19.6 tok/s** | 5.30 | 10.9 | 3.5 |
|
||||||
|
| ~484 | **19.3 tok/s** | 5.28 | 10.0 | 3.3 |
|
||||||
|
| ~933 | **18.5 tok/s** | 5.14 | 9.4 | 3.0 |
|
||||||
|
| ~1833 | **17.1 tok/s** | 4.94 | 6.3 | 2.33 |
|
||||||
|
|
||||||
|
- **Prefill is linear** in both — cost is simply proportional to context loaded.
|
||||||
|
- **The 0.5B is ~3.6× faster at prefill, ~3× at decode.** That slightly exceeds the 2.8×
|
||||||
|
parameter ratio because cost tracks compute per layer (layers × n_embd² predicts ~3.4×),
|
||||||
|
not parameter count.
|
||||||
|
- **Decode degrades as KV grows** in both (0.5B: 10.9 → 6.3 tok/s). Output cost is not
|
||||||
|
constant; it worsens with how much context you loaded.
|
||||||
|
- Full-window extrapolation (0.5B): 4k ctx ≈ 4 min, a full 8k ≈ 8 min.
|
||||||
|
- RAM with the 0.5B resident: ~3.0 GB of 3.67 GB available. The coder model costs ~700 MB more.
|
||||||
|
|
||||||
|
### Tool calling — the offline task class that matters
|
||||||
|
|
||||||
|
Warm, prefix-cached function calling on the **0.5B**, stable system+tools preamble,
|
||||||
|
3 tools (`record_reading`, `update_document`, `set_led`), 6 requests
|
||||||
|
(reproduce: `bench-tools.py`):
|
||||||
|
|
||||||
|
| | result |
|
||||||
|
|---|---|
|
||||||
|
| Correct tool + valid JSON args | **6 / 6** |
|
||||||
|
| First call (cold preamble, 397 tok) | 24.3 s |
|
||||||
|
| Warm calls | **avg 3.8 s** (3.0–4.5 s) |
|
||||||
|
| `prompt_n` warm / `cache_n` | 14 / 377 |
|
||||||
|
|
||||||
|
`prompt_n` collapsing from 397 → ~14 once the cache fills is the whole story: the preamble
|
||||||
|
prefills once, later calls pay only for the user's sentence. Calls came back as **native
|
||||||
|
`tool_calls`**, not scraped from prose.
|
||||||
|
|
||||||
|
**Caveats:** 6 unambiguous single-tool cases is a floor check, not a ceiling — it says nothing
|
||||||
|
about ambiguous or multi-step requests. And this exercised llama-server's *native* tool path,
|
||||||
|
whereas the board config sets `native_tools = false` and uses ZeroClaw's text parser.
|
||||||
|
**The path ZeroClaw actually takes is still unvalidated.**
|
||||||
|
|
||||||
|
### Prefix caching is the dominant lever
|
||||||
|
|
||||||
|
The table above is the *uncached* worst case. In real tool-calling workloads the system
|
||||||
|
prompt and tool schemas are byte-identical on every call, so llama.cpp prefills them once
|
||||||
|
and later calls pay only for the delta. A ~600-token preamble + ~60-token request + ~50-token
|
||||||
|
tool call is **~2 minutes uncached but ~25–30 s warm**.
|
||||||
|
|
||||||
|
Three ways to destroy that advantage — all easy to do by accident:
|
||||||
|
|
||||||
|
- **Varying the preamble.** A timestamp, session id, or rotating skill list injected into the
|
||||||
|
system prompt invalidates the cache every call.
|
||||||
|
- **Unbounded `max_tokens`.** At ~3 tok/s, a 500-token answer is 2.5 minutes of pure decode.
|
||||||
|
- **`max_tool_iterations = 6`.** Each iteration is a full model call — ~3 min per task.
|
||||||
|
Acceptable for a background loop, not for anything interactive.
|
||||||
|
|
||||||
|
### What this means for offline work
|
||||||
|
|
||||||
|
On the **0.5B** (measured or extrapolated from the curve above):
|
||||||
|
|
||||||
|
| task | cost | verdict |
|
||||||
|
|---|---|---|
|
||||||
|
| Function/tool call (warm, cached preamble) | **3.8 s** *(measured)* | viable, near-interactive |
|
||||||
|
| Process a result → decide → emit | ~4–6 s | viable |
|
||||||
|
| Update / retag / summarize one short doc | ~25–35 s | viable |
|
||||||
|
| Q&A retrieving several notes (1.5k ctx) | ~1.5 min | marginal, usable offline |
|
||||||
|
| Code generation | — | not viable (quality, not speed) |
|
||||||
|
|
||||||
|
An 8-hour overnight loop processes roughly **500k prefill tokens** on the 0.5B — several
|
||||||
|
hundred to a thousand short notes.
|
||||||
|
|
||||||
|
**The offline task class Omar specified — function calls, processing a specific result,
|
||||||
|
updating documents, nothing needing complex reasoning — is comfortably served by the 0.5B.**
|
||||||
|
Sub-4-second tool calls are fast enough that this is not merely a background queue worker.
|
||||||
|
|
||||||
|
### ⚠️ The harness, not the model, is the bottleneck
|
||||||
|
|
||||||
|
Raw model speed is *not* what makes or breaks offline. **Prompt size is.** Measured
|
||||||
|
2026-07-19 driving the real ZeroClaw agent loop (`native_tools = false`, the text parser):
|
||||||
|
|
||||||
|
| | default profile | lean `offline` profile |
|
||||||
|
|---|---|---|
|
||||||
|
| Prompt tokens | **4,718** | **706** (−85%) |
|
||||||
|
| Prefix-cache match (`sim_best`) | 0.435 | **0.966** |
|
||||||
|
| Tokens re-prefilled when warm | ~2,600 | **24** |
|
||||||
|
| Cold turn | cancelled mid-prefill | **~29 s** |
|
||||||
|
| Warm turn | never completed | **11 s** |
|
||||||
|
| Tool actually fired | no | **yes** |
|
||||||
|
|
||||||
|
The 11 s warm figure is a *complete agentic turn* — two LLM calls plus tool execution —
|
||||||
|
ending in a real answer (`"The I2C devices on the bus are: i2c-0, i2c-1, and i2c-2."`).
|
||||||
|
|
||||||
|
**The text parser was never the problem.** With the default profile the agent sent a
|
||||||
|
4,718-token prompt that took ~4.6 min to prefill at 17 tok/s, so the client cancelled before
|
||||||
|
the model could respond. Disabling skills injection alone made it *worse* (10,901 tokens —
|
||||||
|
past the 8k window, hard failure) because 84 accumulated memories in `brain.db` flooded the
|
||||||
|
prompt. Flipping `native_tools = true` changed nothing (still 4,718): tool schemas were never
|
||||||
|
the bulk.
|
||||||
|
|
||||||
|
### The lean offline profile
|
||||||
|
|
||||||
|
Everything that controls prompt size lives on the **runtime profile**, not the agent
|
||||||
|
(`effective_skills_prompt_mode`, `schema.rs:4192`, lets a runtime profile override the global
|
||||||
|
`[skills] prompt_injection_mode`):
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[runtime_profiles.offline]
|
||||||
|
agentic = true
|
||||||
|
max_tool_iterations = 3
|
||||||
|
compact_context = true # 6000 chars / 2 RAG chunks; also drops Channel Capabilities
|
||||||
|
prompt_injection_mode = "compact"
|
||||||
|
max_system_prompt_chars = 2000
|
||||||
|
max_context_tokens = 3000
|
||||||
|
max_history_messages = 2
|
||||||
|
memory_recall_limit = 1
|
||||||
|
parallel_tools = false
|
||||||
|
|
||||||
|
[risk_profiles.sense_only] # narrow tools ⇒ also drops the hardware block +
|
||||||
|
level = "supervised" # Uno-Q flash imperative from the system prompt
|
||||||
|
allowed_tools = ["i2cdetect"]
|
||||||
|
auto_approve = ["i2cdetect"]
|
||||||
|
|
||||||
|
[agents.sense]
|
||||||
|
model_provider = "llamacpp.local"
|
||||||
|
risk_profile = "sense_only"
|
||||||
|
runtime_profile = "offline"
|
||||||
|
skill_bundles = [] # the 12 bundled skills were most of the 4,718
|
||||||
|
mcp_bundles = []
|
||||||
|
```
|
||||||
|
|
||||||
|
**Two zero-value footguns:**
|
||||||
|
- `memory_recall_limit = 0` means **unlimited** (`usize::MAX`), not disabled. Use `1`.
|
||||||
|
- `max_actions_per_hour = 0` is a **hard zero** that blocks everything; unlimited is `u32::MAX`.
|
||||||
|
|
||||||
|
### Per-step prompts for multi-step operations
|
||||||
|
|
||||||
|
Cron `JobType::Agent` carries its own `prompt`, `allowed_tools`, and `uses_memory` per job
|
||||||
|
(`crates/zeroclaw-runtime/src/cron/types.rs:32,143`), bound to an agent via
|
||||||
|
`AliasedAgentConfig.cron_jobs`. So a multi-step offline operation is a sequence of jobs, each
|
||||||
|
with the prompt and tool surface for its own step — and `uses_memory = false` skips memory
|
||||||
|
injection entirely for steps that don't need it. Agent-to-agent `delegates` is the other path.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Lessons carried forward
|
||||||
|
|
||||||
|
Design constraints the spike document got wrong. Worth keeping even though the spike
|
||||||
|
itself is dead.
|
||||||
|
|
||||||
|
- **A fail-safe must never be "nominal".** The spike degraded both error paths to a
|
||||||
|
synthetic `nominal` response. In any monitoring context that is a silent all-clear
|
||||||
|
manufactured by an outage. Degrade to `unknown` / `escalate` — never to "fine".
|
||||||
|
- **Fallback must trigger on more than "network unavailable".** Timeouts, 429s, DNS
|
||||||
|
failures, and auth errors are the realistic degradations — a flaky workshop LAN far
|
||||||
|
more often than a cleanly absent one. Matching only a `NetworkUnavailable` variant
|
||||||
|
routes the common cases straight past the local model.
|
||||||
|
- **Health-check endpoints, not processes.** A wedged process passes `pgrep`.
|
||||||
|
- **Persistent server, never per-call subprocess.** Model load time dominates.
|
||||||
|
- **Check exit status and stderr on any subprocess.** Otherwise a crashed child
|
||||||
|
surfaces as an unrelated JSON parse error, which is miserable to debug on a board.
|
||||||
|
- **Don't gate on accuracy against undefined thresholds.** The spike asked a model to
|
||||||
|
choose among three labels without ever defining their boundaries, then scored it
|
||||||
|
against the author's own intuition. That measures prompt underspecification, not
|
||||||
|
model capability. Define the thresholds or don't call it accuracy.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## See also
|
||||||
|
|
||||||
|
- [`README.md`](./README.md) — provisioning, the three modalities, open → locked lifecycle
|
||||||
|
- [`config.template.toml`](./config.template.toml) — providers, agents, risk profile
|
||||||
|
- [`zeroclaw-supervisor.sh`](./zeroclaw-supervisor.sh) · [`recover-uno-q.sh`](./recover-uno-q.sh)
|
||||||
@@ -14,13 +14,14 @@ proxies each board and streams its activity to participants and instructors.
|
|||||||
- **`llama-server`** on `:8083` — on-board Qwen for offline/fallback inference.
|
- **`llama-server`** on `:8083` — on-board Qwen for offline/fallback inference.
|
||||||
- **`zeroclaw-reload-watcher`** (loopback) — applies a team's browser config
|
- **`zeroclaw-reload-watcher`** (loopback) — applies a team's browser config
|
||||||
edits (e.g. enabling their Telegram bot) via `/admin/reload`. See *modalities*.
|
edits (e.g. enabling their Telegram bot) via `/admin/reload`. See *modalities*.
|
||||||
- **Three agents**, selected per request via `?agent=` (APESS maps the harness
|
- **Four agents**, selected per request via `?agent=` (callers pass the alias
|
||||||
provider choice → alias; see `src/lib/harness.ts` `harnessToAgent`):
|
explicitly — see `sendPrompt(teamId, message, agent?)` in `src/lib/api.ts`):
|
||||||
| alias | provider | behaviour |
|
| alias | provider | behaviour |
|
||||||
|-------|----------|-----------|
|
|-------|----------|-----------|
|
||||||
| `default` | `custom.claude` | cloud, **fallback to on-board Qwen** |
|
| `default` | `custom.claude` | cloud, **fallback to on-board Qwen** |
|
||||||
| `cloud` | `custom.cloud` | cloud only |
|
| `cloud` | `custom.cloud` | cloud only |
|
||||||
| `local` | `llamacpp.local`| on-board Qwen only (fully offline) |
|
| `local` | `llamacpp.local`| on-board Qwen only (fully offline) |
|
||||||
|
| `chaos` | `custom.dead` | dead endpoint → **always** fails over to Qwen (outage demo) |
|
||||||
|
|
||||||
## Talking to the node — the three modalities
|
## Talking to the node — the three modalities
|
||||||
|
|
||||||
|
|||||||
Executable
+42
@@ -0,0 +1,42 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# bench-prefill.sh — measure llama-server prefill/decode scaling on :8083
|
||||||
|
#
|
||||||
|
# Uses the server's own `timings` block (prompt_ms / predicted_ms), which is far
|
||||||
|
# more accurate than wall clock. A unique nonce is placed FIRST in every prompt to
|
||||||
|
# defeat llama.cpp prefix caching — otherwise cache_n>0 and prefill is not measured.
|
||||||
|
|
||||||
|
URL=http://127.0.0.1:8083/v1/chat/completions
|
||||||
|
OUT=/home/arduino/bench-prefill.json
|
||||||
|
UNIT="The structural sensor node recorded a nominal vibration reading during the overnight monitoring window and logged it. "
|
||||||
|
|
||||||
|
: > "$OUT"
|
||||||
|
echo "size_label,prompt_n,prompt_ms,prefill_tps,predicted_n,predicted_ms,decode_tps,cache_n,wall_s"
|
||||||
|
|
||||||
|
for reps in 12 25 50 100; do
|
||||||
|
# build filler
|
||||||
|
filler=""
|
||||||
|
i=0
|
||||||
|
while [ $i -lt $reps ]; do filler="$filler$UNIT"; i=$((i+1)); done
|
||||||
|
|
||||||
|
nonce="$(date +%s%N)$$"
|
||||||
|
python3 - "$nonce" "$filler" > /tmp/payload.json <<'PY'
|
||||||
|
import json,sys
|
||||||
|
nonce, filler = sys.argv[1], sys.argv[2]
|
||||||
|
content = f"[req-{nonce}] {filler}\n\nIn one short sentence, what is this text about?"
|
||||||
|
print(json.dumps({"model":"qwen","max_tokens":16,"temperature":0,
|
||||||
|
"messages":[{"role":"user","content":content}]}))
|
||||||
|
PY
|
||||||
|
|
||||||
|
start=$(date +%s)
|
||||||
|
resp=$(curl -s --max-time 1800 "$URL" -H 'Content-Type: application/json' -d @/tmp/payload.json)
|
||||||
|
end=$(date +%s)
|
||||||
|
wall=$((end-start))
|
||||||
|
|
||||||
|
echo "$resp" >> "$OUT"
|
||||||
|
echo "$resp" | jq -r --arg r "reps=$reps" --arg w "$wall" '
|
||||||
|
[$r, .timings.prompt_n, (.timings.prompt_ms|round),
|
||||||
|
(.timings.prompt_per_second*100|round/100),
|
||||||
|
.timings.predicted_n, (.timings.predicted_ms|round),
|
||||||
|
(.timings.predicted_per_second*100|round/100),
|
||||||
|
.timings.cache_n, $w] | @csv'
|
||||||
|
done
|
||||||
Executable
+155
@@ -0,0 +1,155 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""bench-tools.py — warm prefix-cached tool-calling benchmark for llama-server :8083.
|
||||||
|
|
||||||
|
Measures the offline task class we actually care about: function calls, processing a
|
||||||
|
specific result, updating a document. Two things are under test:
|
||||||
|
|
||||||
|
SPEED — with a byte-identical system+tools preamble, llama.cpp should cache the
|
||||||
|
prefix so each call pays only for the delta. cache_n in the output shows
|
||||||
|
whether that is happening.
|
||||||
|
CORRECT — did the model emit a well-formed tool call, pick the right tool, and
|
||||||
|
extract the right arguments? For function calling this matters more than
|
||||||
|
speed: a malformed call is a hard failure, a slow one is just slow.
|
||||||
|
|
||||||
|
Run ON the board: python3 /home/arduino/bench-tools.py
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
URL = "http://127.0.0.1:8083/v1/chat/completions"
|
||||||
|
|
||||||
|
# --- STABLE PREAMBLE: must be byte-identical every call or the prefix cache is lost ---
|
||||||
|
SYSTEM = (
|
||||||
|
"You are an offline node agent. You call exactly one tool per request. "
|
||||||
|
"Never explain. Never answer in prose. Always call a tool."
|
||||||
|
)
|
||||||
|
|
||||||
|
TOOLS = [
|
||||||
|
{
|
||||||
|
"type": "function",
|
||||||
|
"function": {
|
||||||
|
"name": "record_reading",
|
||||||
|
"description": "Record a sensor reading.",
|
||||||
|
"parameters": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"sensor": {"type": "string", "description": "Sensor id, e.g. A3"},
|
||||||
|
"metric": {"type": "string", "description": "temperature | humidity | pressure"},
|
||||||
|
"value": {"type": "number", "description": "Numeric reading"},
|
||||||
|
},
|
||||||
|
"required": ["sensor", "metric", "value"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "function",
|
||||||
|
"function": {
|
||||||
|
"name": "update_document",
|
||||||
|
"description": "Append a line to a named document.",
|
||||||
|
"parameters": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"document": {"type": "string", "description": "Document name"},
|
||||||
|
"text": {"type": "string", "description": "Line to append"},
|
||||||
|
},
|
||||||
|
"required": ["document", "text"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "function",
|
||||||
|
"function": {
|
||||||
|
"name": "set_led",
|
||||||
|
"description": "Set the status LED colour.",
|
||||||
|
"parameters": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"colour": {"type": "string", "description": "red | green | blue | off"},
|
||||||
|
},
|
||||||
|
"required": ["colour"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
# (request, expected_tool, expected_arg_substrings)
|
||||||
|
CASES = [
|
||||||
|
("Log a temperature reading of 22.5 from sensor A3.", "record_reading", ["A3", "22.5"]),
|
||||||
|
("Append 'checked bearings' to the maintenance log.", "update_document", ["bearings"]),
|
||||||
|
("Turn the status LED red.", "set_led", ["red"]),
|
||||||
|
("Record humidity 61 from sensor B1.", "record_reading", ["B1", "61"]),
|
||||||
|
("Add a note saying calibration completed to the maintenance log.", "update_document", ["calibration"]),
|
||||||
|
("Set the status LED to green.", "set_led", ["green"]),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def call(user_msg):
|
||||||
|
payload = {
|
||||||
|
"model": "qwen",
|
||||||
|
"messages": [{"role": "system", "content": SYSTEM},
|
||||||
|
{"role": "user", "content": user_msg}],
|
||||||
|
"tools": TOOLS,
|
||||||
|
"tool_choice": "auto",
|
||||||
|
"max_tokens": 128,
|
||||||
|
"temperature": 0,
|
||||||
|
}
|
||||||
|
req = urllib.request.Request(
|
||||||
|
URL, data=json.dumps(payload).encode(),
|
||||||
|
headers={"Content-Type": "application/json"})
|
||||||
|
t0 = time.time()
|
||||||
|
with urllib.request.urlopen(req, timeout=1800) as r:
|
||||||
|
body = json.loads(r.read())
|
||||||
|
return body, time.time() - t0
|
||||||
|
|
||||||
|
|
||||||
|
def evaluate(body, expected_tool, expected_args):
|
||||||
|
"""Return (ok, tool_name, detail). Accepts native tool_calls or a JSON blob in content."""
|
||||||
|
msg = body.get("choices", [{}])[0].get("message", {}) or {}
|
||||||
|
tcs = msg.get("tool_calls") or []
|
||||||
|
if tcs:
|
||||||
|
fn = tcs[0].get("function", {})
|
||||||
|
name = fn.get("name", "")
|
||||||
|
raw = fn.get("arguments", "")
|
||||||
|
try:
|
||||||
|
args = json.loads(raw) if isinstance(raw, str) else raw
|
||||||
|
args_ok = True
|
||||||
|
except Exception:
|
||||||
|
args, args_ok = {}, False
|
||||||
|
blob = json.dumps(args).lower()
|
||||||
|
hit = all(s.lower() in blob for s in expected_args)
|
||||||
|
return (name == expected_tool and args_ok and hit), name, ("native", args)
|
||||||
|
# fallback: model emitted text instead of a tool call
|
||||||
|
content = (msg.get("content") or "").strip()
|
||||||
|
return False, "(none)", ("text", content[:120])
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
print(f"{'#':<3} {'ok':<4} {'tool':<16} {'p_tok':>6} {'cache':>6} {'pre_s':>7} {'dec_s':>7} {'wall_s':>7}")
|
||||||
|
results = []
|
||||||
|
for i, (msg, want, want_args) in enumerate(CASES, 1):
|
||||||
|
body, wall = call(msg)
|
||||||
|
ok, got, detail = evaluate(body, want, want_args)
|
||||||
|
t = body.get("timings", {}) or {}
|
||||||
|
print(f"{i:<3} {'PASS' if ok else 'FAIL':<4} {got:<16} "
|
||||||
|
f"{t.get('prompt_n', 0):>6} {t.get('cache_n', 0):>6} "
|
||||||
|
f"{t.get('prompt_ms', 0)/1000:>7.1f} {t.get('predicted_ms', 0)/1000:>7.1f} {wall:>7.1f}")
|
||||||
|
results.append((ok, got, detail, t, wall))
|
||||||
|
|
||||||
|
passed = sum(1 for r in results if r[0])
|
||||||
|
warm = results[1:] # first call pays uncached preamble
|
||||||
|
if warm:
|
||||||
|
avg = sum(r[4] for r in warm) / len(warm)
|
||||||
|
avg_ptok = sum(r[3].get("prompt_n", 0) for r in warm) / len(warm)
|
||||||
|
avg_cache = sum(r[3].get("cache_n", 0) for r in warm) / len(warm)
|
||||||
|
print(f"\ncorrect: {passed}/{len(results)}")
|
||||||
|
print(f"first call (cold preamble): {results[0][4]:.1f}s")
|
||||||
|
print(f"warm calls: avg {avg:.1f}s | avg prompt_n {avg_ptok:.0f} avg cache_n {avg_cache:.0f}")
|
||||||
|
print("\nfailures:")
|
||||||
|
for i, (ok, got, detail, _t, _w) in enumerate(results, 1):
|
||||||
|
if not ok:
|
||||||
|
print(f" case {i}: got tool={got} via {detail[0]} -> {detail[1]}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -24,7 +24,7 @@ fallback = ["llamacpp.local"]
|
|||||||
# A deliberately-dead cloud endpoint (nothing listens on :9099) that fails over
|
# A deliberately-dead cloud endpoint (nothing listens on :9099) that fails over
|
||||||
# to the on-board Qwen. Used by the `chaos` agent to DEMONSTRATE resilience: a
|
# to the on-board Qwen. Used by the `chaos` agent to DEMONSTRATE resilience: a
|
||||||
# prompt routed here always finds the cloud unreachable and answers locally —
|
# prompt routed here always finds the cloud unreachable and answers locally —
|
||||||
# the "simulate cloud outage" button in APESS (Module 2, failure modes / L4).
|
# the "simulate cloud outage" demo (ADD L4 Harness: reasoning + tiering).
|
||||||
[providers.models.custom.dead]
|
[providers.models.custom.dead]
|
||||||
uri = "http://127.0.0.1:9099/v1"
|
uri = "http://127.0.0.1:9099/v1"
|
||||||
model = "__CLOUD_MODEL__"
|
model = "__CLOUD_MODEL__"
|
||||||
@@ -91,8 +91,8 @@ max_tool_iterations = 6
|
|||||||
strict_tool_parsing = false
|
strict_tool_parsing = false
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Agents — one per provider strategy. The APESS harness choice routes here via
|
# Agents — one per provider strategy. APESS routes here via ?agent=; callers
|
||||||
# ?agent= (see harnessToAgent in src/lib/harness.ts).
|
# pass the alias explicitly (see sendPrompt in src/lib/api.ts).
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
[agents.default] # cloud + on-board Qwen fallback — the workshop default
|
[agents.default] # cloud + on-board Qwen fallback — the workshop default
|
||||||
enabled = true
|
enabled = true
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ a push /tmp/apess-node-config.toml /home/arduino/.zeroclaw/config.toml >/dev/nul
|
|||||||
echo "==> [2/5] start on-board Qwen (llama-server :8083)"
|
echo "==> [2/5] start on-board Qwen (llama-server :8083)"
|
||||||
a shell 'pgrep -f llama-server >/dev/null 2>&1 || (cd /home/arduino/llama && \
|
a shell 'pgrep -f llama-server >/dev/null 2>&1 || (cd /home/arduino/llama && \
|
||||||
LD_LIBRARY_PATH=/home/arduino/llama nohup ./llama-server -m /home/arduino/models/qwen.gguf \
|
LD_LIBRARY_PATH=/home/arduino/llama nohup ./llama-server -m /home/arduino/models/qwen.gguf \
|
||||||
--host 127.0.0.1 --port 8083 -np 1 -c 16384 --jinja --mlock >/tmp/llama8083.log 2>&1 &)'
|
--host 127.0.0.1 --port 8083 -np 1 -c 8192 --jinja --mlock >/tmp/llama8083.log 2>&1 &)'
|
||||||
|
|
||||||
echo "==> [3/5] start zeroclaw daemon (:8080)"
|
echo "==> [3/5] start zeroclaw daemon (:8080)"
|
||||||
a shell 'pkill -f "zeroclaw daemon" 2>/dev/null; pkill -f "zeroclaw gateway" 2>/dev/null; sleep 2; \
|
a shell 'pkill -f "zeroclaw daemon" 2>/dev/null; pkill -f "zeroclaw gateway" 2>/dev/null; sleep 2; \
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ User=arduino
|
|||||||
Environment=LD_LIBRARY_PATH=/home/arduino/llama
|
Environment=LD_LIBRARY_PATH=/home/arduino/llama
|
||||||
# -np 1 is mandatory (a bigger slot count makes the tiny model 500 on ctx).
|
# -np 1 is mandatory (a bigger slot count makes the tiny model 500 on ctx).
|
||||||
ExecStart=/home/arduino/llama/llama-server -m /home/arduino/models/qwen.gguf \
|
ExecStart=/home/arduino/llama/llama-server -m /home/arduino/models/qwen.gguf \
|
||||||
--host 127.0.0.1 --port 8083 -np 1 -c 16384 --jinja --mlock
|
--host 127.0.0.1 --port 8083 -np 1 -c 8192 --jinja --mlock
|
||||||
Restart=on-failure
|
Restart=on-failure
|
||||||
RestartSec=5
|
RestartSec=5
|
||||||
|
|
||||||
|
|||||||
@@ -56,7 +56,7 @@ daemon_ok_after=0
|
|||||||
start_llama() {
|
start_llama() {
|
||||||
log "starting llama-server :8083"
|
log "starting llama-server :8083"
|
||||||
setsid sh -c "cd '$LLAMA_DIR' && LD_LIBRARY_PATH='$LLAMA_DIR' exec ./llama-server \
|
setsid sh -c "cd '$LLAMA_DIR' && LD_LIBRARY_PATH='$LLAMA_DIR' exec ./llama-server \
|
||||||
-m '$MODEL' --host 127.0.0.1 --port 8083 -np 1 -c 16384 --jinja --mlock" \
|
-m '$MODEL' --host 127.0.0.1 --port 8083 -np 1 -c 8192 --jinja --mlock" \
|
||||||
>> "$HOME_DIR/llama8083.log" 2>&1 < /dev/null &
|
>> "$HOME_DIR/llama8083.log" 2>&1 < /dev/null &
|
||||||
llama_ok_after=$(( $(now) + LLAMA_WARMUP ))
|
llama_ok_after=$(( $(now) + LLAMA_WARMUP ))
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user