Omar SobhandClaude Opus 4.8
a23a525aef
Replaces the never-executed "LiteRT-LM on UNO Q 4GB" spike plan with a measured record of the local-fallback path we actually run. Measured on board 65301572 (Qwen2.5-0.5B-Instruct, -c 8192, 4 threads), using llama-server's own timings rather than wall clock: prefill ~17-20 tok/s (linear), decode ~6-11 tok/s (degrades with KV) warm prefix-cached tool call: 3.8s, 6/6 correct structured calls Two findings that changed the deployment: 1. The board had drifted onto Qwen2.5-Coder-1.5B - larger and tuned for the wrong task. Reverting to the repo's 0.5B made tool calls ~6x faster (24s -> 3.8s) and freed ~700MB. The repo was right. 2. The harness, not the model, was the bottleneck. The default agent profile sent a 4718-token prompt (~4.6 min prefill) and the client cancelled before the model could answer. A lean runtime profile cuts that to 706 tokens, lifts prefix-cache match 0.435 -> 0.966, and completes a full agentic turn with a real tool call in 11s warm. The ZeroClaw text parser was never at fault. Prompt cost model for budgeting profiles: ~706 base (1 tool), ~244/additional tool, +315 for uno_q_flash (schema + flash imperative), ~53/skill in compact mode. Also standardises context on -c 8192 across all three provisioning paths (a 16k window costs ~16 min to fill at this speed and doubles KV for nothing), and fixes stale references to the deleted src/lib/harness.ts. Adds bench-prefill.sh and bench-tools.py as reproducible baselines. Co-Authored-By: Claude Opus 4.8 <[email protected]>
43 lines
1.6 KiB
Bash
Executable File
43 lines
1.6 KiB
Bash
Executable File
#!/bin/sh
|
|
# bench-prefill.sh — measure llama-server prefill/decode scaling on :8083
|
|
#
|
|
# Uses the server's own `timings` block (prompt_ms / predicted_ms), which is far
|
|
# more accurate than wall clock. A unique nonce is placed FIRST in every prompt to
|
|
# defeat llama.cpp prefix caching — otherwise cache_n>0 and prefill is not measured.
|
|
|
|
URL=http://127.0.0.1:8083/v1/chat/completions
|
|
OUT=/home/arduino/bench-prefill.json
|
|
UNIT="The structural sensor node recorded a nominal vibration reading during the overnight monitoring window and logged it. "
|
|
|
|
: > "$OUT"
|
|
echo "size_label,prompt_n,prompt_ms,prefill_tps,predicted_n,predicted_ms,decode_tps,cache_n,wall_s"
|
|
|
|
for reps in 12 25 50 100; do
|
|
# build filler
|
|
filler=""
|
|
i=0
|
|
while [ $i -lt $reps ]; do filler="$filler$UNIT"; i=$((i+1)); done
|
|
|
|
nonce="$(date +%s%N)$$"
|
|
python3 - "$nonce" "$filler" > /tmp/payload.json <<'PY'
|
|
import json,sys
|
|
nonce, filler = sys.argv[1], sys.argv[2]
|
|
content = f"[req-{nonce}] {filler}\n\nIn one short sentence, what is this text about?"
|
|
print(json.dumps({"model":"qwen","max_tokens":16,"temperature":0,
|
|
"messages":[{"role":"user","content":content}]}))
|
|
PY
|
|
|
|
start=$(date +%s)
|
|
resp=$(curl -s --max-time 1800 "$URL" -H 'Content-Type: application/json' -d @/tmp/payload.json)
|
|
end=$(date +%s)
|
|
wall=$((end-start))
|
|
|
|
echo "$resp" >> "$OUT"
|
|
echo "$resp" | jq -r --arg r "reps=$reps" --arg w "$wall" '
|
|
[$r, .timings.prompt_n, (.timings.prompt_ms|round),
|
|
(.timings.prompt_per_second*100|round/100),
|
|
.timings.predicted_n, (.timings.predicted_ms|round),
|
|
(.timings.predicted_per_second*100|round/100),
|
|
.timings.cache_n, $w] | @csv'
|
|
done
|