Omar SobhandClaude Opus 4.8
a23a525aef
Replaces the never-executed "LiteRT-LM on UNO Q 4GB" spike plan with a measured record of the local-fallback path we actually run. Measured on board 65301572 (Qwen2.5-0.5B-Instruct, -c 8192, 4 threads), using llama-server's own timings rather than wall clock: prefill ~17-20 tok/s (linear), decode ~6-11 tok/s (degrades with KV) warm prefix-cached tool call: 3.8s, 6/6 correct structured calls Two findings that changed the deployment: 1. The board had drifted onto Qwen2.5-Coder-1.5B - larger and tuned for the wrong task. Reverting to the repo's 0.5B made tool calls ~6x faster (24s -> 3.8s) and freed ~700MB. The repo was right. 2. The harness, not the model, was the bottleneck. The default agent profile sent a 4718-token prompt (~4.6 min prefill) and the client cancelled before the model could answer. A lean runtime profile cuts that to 706 tokens, lifts prefix-cache match 0.435 -> 0.966, and completes a full agentic turn with a real tool call in 11s warm. The ZeroClaw text parser was never at fault. Prompt cost model for budgeting profiles: ~706 base (1 tool), ~244/additional tool, +315 for uno_q_flash (schema + flash imperative), ~53/skill in compact mode. Also standardises context on -c 8192 across all three provisioning paths (a 16k window costs ~16 min to fill at this speed and doubles KV for nothing), and fixes stale references to the deleted src/lib/harness.ts. Adds bench-prefill.sh and bench-tools.py as reproducible baselines. Co-Authored-By: Claude Opus 4.8 <[email protected]>
17 lines
448 B
Desktop File
17 lines
448 B
Desktop File
[Unit]
|
|
Description=ZeroClaw on-board Qwen (llama-server)
|
|
After=network.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=arduino
|
|
Environment=LD_LIBRARY_PATH=/home/arduino/llama
|
|
# -np 1 is mandatory (a bigger slot count makes the tiny model 500 on ctx).
|
|
ExecStart=/home/arduino/llama/llama-server -m /home/arduino/models/qwen.gguf \
|
|
--host 127.0.0.1 --port 8083 -np 1 -c 8192 --jinja --mlock
|
|
Restart=on-failure
|
|
RestartSec=5
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|