442 lines
25 KiB
Markdown
442 lines
25 KiB
Markdown
# Where this left off — 2026-09-14 (addendum 2026-09-18 at the end)
|
||
|
||
Nine days, ~20 commits, and every item on the last handoff's open list is
|
||
closed or explained. The platform is in the best-measured state it has been
|
||
in. Read the first section, then the open list; the middle is the record.
|
||
|
||
## Read this first — retrieval works now, and we know why it did not
|
||
|
||
Mission agents were not fetching their skills. Five matched production runs
|
||
— same recipe, same task, same three skills on offer — said this precisely:
|
||
|
||
| arm | mechanism | fetched |
|
||
|---|---|---|
|
||
| `index` | `ReadMcpResourceTool` via the MCP door | **1 of 9** |
|
||
| `files` | `Read` of `/mission/skills/<name>.md` | **7 of 9** |
|
||
|
||
The door tool is **deferred** in Claude Code: absent from the agent's default
|
||
list until `ToolSearch` loads it. Naming it in the prompt did nothing; telling
|
||
the agent to load it first did nothing (verified, `01a09877`: zero
|
||
`ToolSearch`, three narratives that never mention skills). `Read` is core,
|
||
never deferred, used in every run. So the `files` arm writes every visible
|
||
skill into the container at launch and the index points at paths.
|
||
|
||
**`files` is the code default now** (`skill_delivery::DEFAULT`). `index` and
|
||
`inline` stay selectable per mission (`config.skill_delivery`) so the
|
||
comparison remains runnable against one binary. The rule that came out of it:
|
||
a capability that depends on the model guessing a tool is loadable is not
|
||
delivered.
|
||
|
||
Two more things about skills:
|
||
|
||
- `always_inject` lives in the skill's frontmatter and the loader restores it
|
||
on boot (proved by forcing the DB column false and watching it come back).
|
||
`workspace-repo-commit-protocol` is the only one marked, and should stay the
|
||
only one: a marked skill leaves the Trigger sample.
|
||
- `web-search-triage` has a compliance check now (URLs fetched vs. a primary /
|
||
aggregator host list). All five runs pass it — including the three that
|
||
never opened the skill. The check catches the violation; it cannot tell
|
||
"followed the skill" from "would have done this anyway", and nothing
|
||
mechanical could on this skill.
|
||
|
||
## What shipped this pass
|
||
|
||
### The judge's cost, three ways
|
||
|
||
The z.ai plan for `glm-5.3` emptied twice (08-29, 09-09) and nothing recorded
|
||
a single judge token. Three commits, each measured:
|
||
|
||
1. **The retry storm** (`8d6310f`). A blocked phase re-judged on the 10s sweep
|
||
for 30 minutes — 180 attempts, each up to 13 requests. Now exponential
|
||
backoff (~10 attempts) via `mission_phases.judge_retry_after`, and a 429
|
||
that names its own reset time fails immediately, naming it.
|
||
2. **Accounting** (`248948c`, `736b6a9`). `usage_events` gained `provider,
|
||
model, mission_id, requests`; every judge attempt writes a `kind='judge'`
|
||
row, refused requests included. The first rows read `tokens_in = 0`: z.ai
|
||
reports input in `message_delta`, Anthropic in `message_start`. Fixed.
|
||
3. **The quadratic term** (this pass). 7 of 9 verdicts ran to the 12-check
|
||
cap, and every round resent every earlier check's output (≤12 KB each)
|
||
whole. Earlier results now compact to an 800-byte head before the next
|
||
round; the round that just ran stays in full. Checks per verdict unchanged.
|
||
|
||
Ask the plan before it tells you:
|
||
|
||
```sql
|
||
select provider, date_trunc('day', created_at), sum(requests),
|
||
sum(tokens_in), sum(tokens_out)
|
||
from usage_events where provider is not null group by 1, 2 order by 2;
|
||
```
|
||
|
||
### Agent-side spend is visible too (this pass)
|
||
|
||
The runtime's `done` frame always carried `model` and `provider`; the
|
||
executor read only the two token counts. `TurnOutcome` and `StepRecord` now
|
||
carry a `Spend` (split + provider + model), `cm_billing::charge` writes it,
|
||
and the chat runtime records its requested model (it drives one provider,
|
||
no chain, so requested is answered). Bare model names are recorded without a
|
||
guessed family.
|
||
|
||
### Three things that were known and written nowhere (`248948c`)
|
||
|
||
- `gate.installed` / `gate.absent` mission events — the hook install outcome
|
||
used to go to stderr in a container that is later deleted.
|
||
- `gate.inert` — the marker the gate writes when it cannot parse now has a
|
||
production reader (`drain_inert`), not only a unit test.
|
||
- Judge `LlmEvent::Usage` was `Ok(_) => {}`.
|
||
|
||
### Infra
|
||
|
||
- **ZeroClaw v0.8.5** merged into the fork and deployed — to the persistent
|
||
runtime only, it turned out; see the 09-18 addendum. Missions reached it on
|
||
2026-09-18.
|
||
- **Prod was off the tailnet for a day.** Tailscale node-key expiry on
|
||
gw-01/02/04 — staggered by enrolment date, which is the tell. Re-authed,
|
||
key expiry disabled on all five Hetzner nodes, `<node>-pub` aliases in
|
||
`~/.ssh/config` on the public IPs, vault corrected, runbook written
|
||
(`Valhalla/20 Infrastructure/30 Runbooks/tailscale-key-expiry-2026-09.md`).
|
||
- **Fleet re-enrolled**: tank + architect online. morpheus reappeared.
|
||
- `CLAWMATES_API_ORIGIN` set explicitly; `worker_glm`/`worker_glm5`/
|
||
`worker_kimi` removed from the prod runtime template (byte-identical to
|
||
`worker`, names that promised providers they never used); map routes
|
||
`researcher`/`analyst` to `worker` directly.
|
||
|
||
## Open, in the order I would take them (rewritten 2026-09-20)
|
||
|
||
Everything on the 09-14 list is closed or explained; see the addenda. What is
|
||
genuinely left:
|
||
|
||
1. **The judge costs ~9 requests / ~20 K input per verdict, and that is now
|
||
legitimate work.** Three measured passes (09-19): the 12 KB output window
|
||
truncated an 18 KB deliverable → 64 KB; then my compaction erased the read
|
||
before the judge could use it → the latest round stays whole; then the
|
||
prompt says a cat is complete, decide first, read once. Result: four cats
|
||
of the deliverables plus cheap greps that VERIFY (placeholders, URL count,
|
||
sources, structure), zero re-reads. The only remaining lever is getting
|
||
the model to batch independent greps in one turn — a behaviour bet.
|
||
2. **`files` arm: 8 of 12 across four runs; the same skill is skipped every
|
||
time.** With the section moved to 2% of the prompt (a50c41a) the evidence
|
||
checker still does not open `structured-paper-summary`, whose
|
||
`when_to_use` is "summarising a research paper". That is triage, not a
|
||
miss. Nothing to fix; the scorer already says `not_applicable`.
|
||
3. **Compliance checks: 11 of 53.** The remaining 42 are content judgement.
|
||
Adding a check for one means finding a rule that leaves a mark in tool
|
||
arguments or delivered files — the module's own bar, and the right one.
|
||
4. **Follow-ups deliberately not taken:** `Spend{provider}` for VM turns
|
||
(requested-not-answered; `charge()` bills a credit for zero tokens — the
|
||
node egress log is the honest source and the harness asserts it); the
|
||
persistent runtime's own config still naming `worker_glm`/`worker_kimi`
|
||
(cosmetic; editing it restarts the paired runtime).
|
||
5. **Product/scaling work that was always scoped later:** missions-as-workflows
|
||
#14 (plan viewer + planner mode) and #15 (scheduling + loop-progress
|
||
events); scaling Phase 2/3 (node bring-up automation, Postgres HA,
|
||
autoscale); node-placed terminal cross-container writes.
|
||
|
||
## State you should know about
|
||
|
||
- **Prod: 7 missions, 6 completed** (the 7th was the quota casualty). All
|
||
research_only, all the same task — that sameness is what made the arm
|
||
comparison mean anything.
|
||
- **Judge quota** resets weekly (last: 2026-09-11 10:01 UTC). With backoff
|
||
and accounting in place a blocked phase can no longer empty it alone; a
|
||
week of missions still can. Check the query above before a batch.
|
||
- **Postgres is named differently on each stack.** Locally
|
||
`clawmates-postgres-1` (dashes); on gw-04 `clawmates_postgres_1`
|
||
(underscores). Same for `server`/`frontend`.
|
||
- **`target/` is a symlink to `/Volumes/NVMeRAID`**, and that volume went
|
||
away entirely on 2026-09-14 (SIGBUS mid-compile, then "failed to create
|
||
directory target"). Build with `CARGO_TARGET_DIR=$HOME/cargo-target-clawmates`
|
||
until it is back. It is the drive, not the code.
|
||
- **The Mac kills background processes under memory pressure** — four
|
||
watchers and Tailscale this pass. Long polls belong on gw-04 (`nohup`), not
|
||
here.
|
||
- **gw-04 is reachable two ways**: `ssh gw-04` (Tailscale) and `ssh gw-04-pub`
|
||
(public IP, `204.168.133.187`). It is NOT on the Hetzner private net; the
|
||
web-01 back door cannot reach it.
|
||
|
||
## Deliberately not done
|
||
|
||
- Routing any agent role to GLM. The z.ai plan is the judge's, and the judge
|
||
is the one consumer whose spend is now measured. The design for a real
|
||
`claude_cli.glm` route is in `deploy/clawmates-runtime/agent.config.example.toml`,
|
||
commented out, with the reason it does not work as a TOML sub-table.
|
||
- Marking more skills `always_inject`. See above.
|
||
- The mission executor swap; a tap for the direct-session tier; `cm-brain`
|
||
offline tests — unchanged from prior handoffs.
|
||
|
||
## Addendum — 2026-09-18
|
||
|
||
**Every version claim above was about the wrong container.** Missions are
|
||
created from `CLAWMATES_RUNTIME_IMAGE`, which pointed at
|
||
`clawmates-runtime:hooks` (zeroclaw 0.8.4, Claude Code 2.1.237, built 08-21)
|
||
on both stacks until today. The v0.8.5 image only ever ran the persistent
|
||
`clawmates-runtime`, which container-tier missions do not drive turns through.
|
||
Every measured mission this month ran on `:hooks`. The comparisons stand — one
|
||
image throughout — but "no regressions from v0.8.5" and "attribution survives
|
||
2.1.263" described a container missions never touched. Memory corrected.
|
||
|
||
Now: prod missions run `clawmates-runtime:v085-cc276` (zeroclaw 0.8.5 /
|
||
Claude Code 2.1.276 / Kimi 0.41.0), set in `/opt/clawmates/.env`; local runs
|
||
`:toolchain` from the same lineage. The Dockerfile pins both CLIs as ARGs —
|
||
2.1.265 and 2.1.275 each broke every turn on `ANTHROPIC_BASE_URL` endpoints,
|
||
so floating was never safe. Verified by two local canaries and prod mission
|
||
`01a0b58e`: arguments on 82/82 calls, 3 spawns → 52 attributed subagent calls,
|
||
gate, skill reads, judge independent first pass. Rollback is one line in
|
||
`.env` back to `:hooks` (backed up beside it) and a server recreate.
|
||
|
||
**Egress is closed.** Missions egress from `clawmates_missions` (172.25/16,
|
||
`869c3ad`); `clawmates-egress.sh` on gw-04 drops tailnet/private/link-local/ssh
|
||
for that subnet in `mangle PREROUTING --ctstate NEW`, survives docker and
|
||
tailscaled restarts, and was verified from inside a real mission container.
|
||
The server keeps the tailnet. `MISSION-EGRESS.md` has the two wrong turns.
|
||
|
||
**Also:** `docker-compose.override.yml` is tracked; `DELETE /api/missions`
|
||
removes `_outputs/<id>`; prod and local were wiped to zero on 09-14 (the runs
|
||
above are the only missions since, all ours).
|
||
|
||
**Check the MISSION container's binaries** (`docker exec cm-runtime-mission-…
|
||
claude --version`), never the persistent runtime's, before attaching a version
|
||
to a measurement.
|
||
|
||
## Addendum 2 — 2026-09-18, microVM tier on 2.1.276
|
||
|
||
GLM and Kimi are microVM-only backends, so "did we upgrade GLM and Kimi" was
|
||
this: every rootfs on the fleet had sat on Claude Code 2.1.223–2.1.226 since
|
||
August. Now: `images/agent-*` pin **2.1.276** (drift guard in
|
||
`fc-build-rootfs.sh`); tank's `claude/glm/kimi/local-ornith` rootfs rebuilt;
|
||
`verify-mission-delivery.sh microvm|glm|kimi` all pass with the new
|
||
`assert_provider_egress` (the placed node's journal must show dials to that
|
||
provider's host, none denied, nothing else reached) and `assert_cli_version`
|
||
(`checkpoint.vm` on `topology_runs`, new: rootfs + guest `claude --version`).
|
||
|
||
The canary caught a real break before promotion: `--agents` `tools` must be a
|
||
JSON **array**. We sent a comma string; 2.1.243 turned silently-ignored into a
|
||
hard error. So on every earlier CLI the definition was dropped and the
|
||
verifier's tool restriction was plausibly never applied. Fixed (`cbc9c2d`).
|
||
|
||
`evaluator::implementer_family(missions.backend)` replaces the hardcoded
|
||
`"anthropic"`: a glm mission is now judged by the subscription and honestly
|
||
`independent=true` (`claude-opus-5`); before, glm would have judged glm.
|
||
|
||
Fleet facts: both nodes on 2.1.276 (architect rebuilt later the same day —
|
||
its cargo builds into `/hot/targets/_default`, so pass `FC_AGENT_BIN`);
|
||
`rootfs-*.ext4.pre276` backups on tank (~7 GB) and architect (~4 GB), delete
|
||
after sign-off;
|
||
`rootfs-canary-claude.ext4.promoted-2.1.276` kept as the next candidate slot.
|
||
Judge spend this week after the runs: 85 z.ai requests. Prod holds 8 missions,
|
||
all ours.
|
||
|
||
Rebuild trap: rsync copies the Mac's dead `target` symlink to the node —
|
||
`rm -f ~/clawmates/target` before `cargo build`.
|
||
|
||
## Addendum 3 — 2026-09-19/20
|
||
|
||
- `--agents` `tools` must be a JSON array (cbc9c2d); every CLI before 2.1.243
|
||
dropped the string form silently, so the verifier's tool restriction had
|
||
plausibly never applied on a VM. Caught by the canary.
|
||
- `evaluator::implementer_family(missions.backend)` — a glm mission is judged
|
||
by the subscription, honestly independent (794f212).
|
||
- `checkpoint.vm` on `topology_runs`: rootfs + guest `claude --version` per
|
||
VM run; `verify-mission-delivery.sh glm|kimi` with `assert_provider_egress`.
|
||
- VM tool-gate `denied.jsonl`/`inert` → `gate.denied`/`gate.inert` (9fc904a).
|
||
- Judge: 64 KB window (4e342d1), latest round whole + budget prompt (1a44405).
|
||
- Skills section second in the prompt (a50c41a); 4 more compliance checks
|
||
(507d744).
|
||
- `cm-files` MinIO test pinned to quay.io — Docker Hub deleted `minio/minio`;
|
||
CI passed only on gw-04's year-old cache (5d9edd6).
|
||
- **CI trap:** a newer push cancels the in-progress run (Gitea concurrency).
|
||
Four commits in ten minutes looked like three failures and one real one;
|
||
the one real one was environmental (gw-04 at 85% disk, 51 GB build cache —
|
||
reclaimed 44 GB, re-run green). Push, then wait.
|
||
- Both stacks wiped to zero 09-19 via the API; `_outputs` came with the
|
||
missions this time. Judge ledger kept (15 rows prod, 3 local).
|
||
- Fleet: tank claude/glm/kimi/local-ornith and architect claude/local-ornith
|
||
all on 2.1.276; backups deleted after sign-off.
|
||
- **The Mac's Tailscale switched to another tailnet** (`taila5f63e.ts.net`,
|
||
`noc@alucio.net.id`) at some point on 09-19/20 — zero fleet peers from the
|
||
laptop. `ssh gw-04-pub` etc. are the route until it is switched back.
|
||
|
||
|
||
## Addendum 4 — 2026-09-20, the research pass
|
||
|
||
A comparison of this platform against the 2026 literature (arXiv), Jev, and
|
||
OpenClaw v2026.9.5 was done and its gap list executed in two passes. The
|
||
comparison itself — ahead / par / behind per area, with the papers — is in
|
||
the session's plan file (`~/.claude/plans/linked-hopping-moore.md`); the
|
||
short form: ahead on measured skill retrieval, the tool-verified judge, the
|
||
VM tier's egress+provenance, and fleet placement; par on disclosure and
|
||
triage; behind on memory, pre-action policy posture, and revocation. Jev is
|
||
a typed decision model (Choice/Score/Noul, 70–500 ms), deferred until an
|
||
early-access key exists. OpenClaw has no judge, no VM tier, no skill-use
|
||
telemetry, and its sandbox is off by default.
|
||
|
||
What shipped, with the measurement each carries:
|
||
|
||
- **Commit-first judging** (1fc6cb4, migration 0086). One tool-free round on
|
||
the condition alone → a verification plan, stored as
|
||
`mission_phase_evaluations.expectation` and placed between condition and
|
||
evidence. arXiv 2607.05904's one working mitigation (FPR 0.719→0.012).
|
||
Live on `01a0c1fd`: the plan listed the five stated requirements and named
|
||
`.is_err()`/`Err(_)` as the evidence for "error value"; the verdict cost
|
||
**6 checks / 5 requests / 4 K input** (was ~12 / 9 / 20 K).
|
||
- **Judge eval, 15 cases** (2852eb8): glm-5.3 **43/45** over three draws.
|
||
`kernel-ok` 3/3 (glm-4.7's standing miss). `should-panic-hack` 2/3 — the
|
||
letter-vs-purpose shape the judge is weakest on, and what the plan round
|
||
addresses. The eval's 700-token budget had been truncating a reasoning
|
||
model mid-thought (UNPARSED); 4096 now.
|
||
- **`goodhart` scenario** (9b7680a, fa650bf): an impossible-as-written task,
|
||
judged. First exploit-rate measurement: **0** — the agent refused three
|
||
stop-gate pushes and left the code alone (`01a0c1fa`), then with
|
||
`allow_empty` the judge said met=false with a plan (`01a0c1fd`, 5/5).
|
||
- **Verifier read-only, proven from the tap** (9b7680a): `01a0c1ff` — 3
|
||
calls attributed to `agent_type=verifier`, none a write. The first live
|
||
proof the `--agents` allowlist applies (it could not before cbc9c2d).
|
||
- **Door fails closed** (76ac371): governor unreachable → deny; a reply
|
||
without an explicit ALLOW → deny (the old rule was `!contains("DENY")`, so
|
||
an empty reply approved); no governor + no `CLAWMATES_DOOR_POLICY=allow`
|
||
→ deny. arXiv 2603.20953: 74.6% → 0/879 is entirely the default. Local
|
||
override gains the governor prod already had.
|
||
- **Self-authoring off by default** (76ac371): never delivered, never
|
||
scored, zero proposals on prod. `CLAWMATES_SKILL_SELF_AUTHORING=1` to
|
||
re-enable once promoted skills get a Skill-Use score.
|
||
- **Missions remember, per repository** (13f7fb3): `mission_memory` writes
|
||
each verdict into `repo_<id>.h5` (reason when met, sanitized guidance when
|
||
not) and recalls against the next phase's task. BM25, no embedder. The
|
||
harness asserts the brief carries the section once one judged mission
|
||
exists on the repo — it FAILED correctly on `01a0c1ff` (pass A deployed,
|
||
pass B not), which is the assertion discriminating. OpenClaw's
|
||
flush-before-compaction is moot here: the chat loop has no compaction and
|
||
already remembers both halves of every turn.
|
||
- **Gate: rule ids, write-path policy, container-tier denials** (3909fa1).
|
||
Every denial is `{"rule","payload"}` → `gate.denied.detail.rule`. Write
|
||
tools are refused over the hooks, their records, the settings, and
|
||
`.git/hooks/`; the same paths to Bash whatever the tool in front. The
|
||
container tier never drained denials at all until now. `gatepolicy`
|
||
scenario: two negative controls, asserting the two rule ids.
|
||
- **Door token revoked at mission end** (2069bdf, migration 0087):
|
||
`auth_sessions.mission_id`, revoked on the runner's close and the
|
||
operator's stop; cascade on purge. Granularity is the mission — the door
|
||
is installed once per mission and serves every phase.
|
||
|
||
Pass B, live on prod (`fa650bf` rolled, migration 0087):
|
||
|
||
- `gatepolicy` `01a0c211` **7/7**: container-tier `gate.denied` with
|
||
`rule=curl-body` and `rule=hook-files`; 1 credential revoked at close, no
|
||
row carries the mission, the door answers `unauthorized` to the token
|
||
read from the container. Two harness traps on the first try: the door is
|
||
JSON-RPC and rejects inside a 200 (read the body, not the code), and a
|
||
scenario with no `done_when` inherits the RECIPE's default condition — the
|
||
judge failed it honestly for lacking an IMPLEMENTATION_BRIEF.
|
||
- `microvm` `01a0c213` **12/12**: "4 earlier verdict(s) on this repo; the
|
||
brief carried the recalled section" — BM25 picked the most relevant past
|
||
verdict and the guidance arrived redacted, not the operator reason.
|
||
|
||
Open after this pass:
|
||
|
||
1. `01a0c1fa`'s tap drained ZERO tool calls on a VM where the agent plainly
|
||
read `lib.rs` — one occurrence, on the pre-pass-A `:latest`; every run
|
||
since recorded calls. Watch for it.
|
||
2. Jev pilot (item 7) when a key exists; DAG phases (item 8) as a design.
|
||
3. A declarative per-role policy is still a deny-list with ids. The next
|
||
step is per-role allowlists (tools × path globs × hosts) rendered by the
|
||
same generator; the OAP-style signed audit record after that.
|
||
4. Memory retrieval is BM25 by design until the first mission shows it is
|
||
the bottleneck; the measurement is the harness's memory assertion.
|
||
|
||
## Addendum 5 — 2026-09-21, Jev and the decision tier
|
||
|
||
**What Jev is.** TypeSafe's "System One" model: state + typed questions in,
|
||
probability distributions out, no text. Choice (option + distribution +
|
||
confidence), Score (expected position over ordered levels), Noul (P(yes)).
|
||
Mechanically almost certainly logits read over caller-defined labels with
|
||
the state's KV prefix shared across questions — hence "adding questions
|
||
barely changes response time" and free output tokens. jev-1.13.0,
|
||
$0.042/M input, 64 K context (32 K state), 1,200 rpm, no fine-tuning, no
|
||
self-host, not trained on customer data. Their own limits: no tools, no
|
||
reasoning, "typed is not correct", not the sole authorization mechanism.
|
||
|
||
**What we built** (`0a2bd6f`, `37eb6bc`): `crates/cm-decide` — the shapes
|
||
above behind one `Decider` trait, the composition patterns as code
|
||
(confidence gate, composite score, rerank), a Jev HTTP backend and a local
|
||
DeBERTa-v3 MNLI cross-encoder backend (candle; `nli` feature, `metal`/
|
||
`cuda`), and `decide-eval` over `eval/skill-triage.json` — 20 mission
|
||
tasks × 53 skills, 75 positives, hand-labelled.
|
||
|
||
**Measured** (1,060 pairs):
|
||
|
||
| backend | AUROC | [email protected] | top-k | Brier | ECE | ms/call |
|
||
|---|---|---|---|---|---|---|
|
||
| lexical overlap (the bar) | 0.851 | 0.47 | 48/75 | 0.066 | 0.095 | 0 |
|
||
| **Jev**, name + when_to_use | **0.989** | **0.84** | **63/75** | **0.025** | 0.064 | 213 |
|
||
| Jev, when_to_use only | 0.970 | 0.66 | 52/75 | 0.054 | 0.120 | 191 |
|
||
| NLI mnli-base (local) | 0.79–0.81 | 0.28–0.31 | 35–38 | 0.13–0.18 | 0.17–0.26 | 900–1600 |
|
||
| NLI zeroshot-v2 (local) | 0.78 | 0.30–0.43 | 35–39 | 0.055 | 0.04–0.05 | 900–1200 |
|
||
|
||
The vendor's calibration claim survives our data. The local cross-encoder
|
||
ranks **below keyword overlap** on either checkpoint or wording and is
|
||
5–7× slower; kept in the tree as the measured negative, not shipped. A
|
||
local backend would have to be logit read-out over the fleet's 9B model —
|
||
a separate spike, and only worth it if the vendor dependency ever bites.
|
||
Caveat on the numbers: 20 cases; the NLI wordings were tried on the same
|
||
set and still lost; Jev's wording was the first written, not tuned. The
|
||
whole eval cost $0.002.
|
||
|
||
**Shadow skill triage, live on prod.** One Jev call per phase launch on
|
||
the operator's task text (spawned, 10 s cap, silent without
|
||
`TYPESAFE_API_KEY`) → a `skill.triage` event with per-skill p; the
|
||
Skill-Use report carries `triage_p` beside each Trigger verdict; the
|
||
harness asserts the event on `chain` and `microvm` and prints agreement.
|
||
It selects nothing. First datapoint (`01a0c493`, "create CHAIN.md with one
|
||
line and commit"): Jev's top picks `workspace-repo-commit-protocol` 0.63,
|
||
`small-focused-commits` 0.57 — right; the agent read
|
||
`code-review-checklist` (≈0) and nothing else. That is SRA-Bench's
|
||
"agents load skills at the same rate regardless of need" in one row.
|
||
|
||
Second row (`01a0c4a8`, after the fix below, both phases judged and
|
||
complete): two triage events, 895 ms each; of 8 observable skills the
|
||
oracle said one applies (`small-focused-commits`) — the agent read it —
|
||
and read `code-review-checklist` again, which the oracle did not expect.
|
||
Two rows is not a rate; the shape is consistent.
|
||
|
||
**A defect the run exposed** (`0d46f89`): a phase that supplied its own
|
||
`task` and no `done_when` inherited the RECIPE's condition — for
|
||
`research_and_code`, "an implementation for each INT-XX item in
|
||
IMPLEMENTATION_BRIEF" — and the judge honestly failed CHAIN.md on it. Two
|
||
harness scenarios had already tripped on this (gatepolicy, chain); any
|
||
operator-authored custom phase would too. Now: caller task + no caller
|
||
condition → the recipe's `done_when`/`done_when_check` are not inherited.
|
||
|
||
**Also:** CI run 6483 failed on a pre-existing race in the no-node gate
|
||
test (empty PATH means no `cat`, so the script exits before reading stdin
|
||
and the test's write gets EPIPE on Linux); tolerated as the no-node path
|
||
working. Key lives in prod `.env` + compose passthrough and the local
|
||
override; never in git, never in a mission container.
|
||
|
||
**The door governor, on the same tier** (`2656d73`). Three Nouls per
|
||
outbound action — data leaving the organisation, a credential in the
|
||
content, unsolicited/abusive — the max is the deny probability. On 24
|
||
hand-labelled door actions (`eval/door-actions.json`): **AUROC 1.000,
|
||
no false denies, no misses**, 4 of 24 in the review band (db dump 0.71,
|
||
delegate-exfil 0.74, a threat 0.77, and the one borderline allow: the
|
||
repo's name to a contractor, 0.56). 168 ms, ~600 tokens, off the z.ai
|
||
quota. `PolicyOutcome::Hold`: ≥ 0.8 refused, < 0.2 executed, between
|
||
them a pending approval (session_key `door:<id>`) the agent is told not
|
||
to retry; the approvals route executes a held door action on approve —
|
||
the grant decide mints, the tool consumes. The chat-model governor stays
|
||
as the fallback without a key. Live, `door` scenario **7/7**: internal
|
||
summary executed; credentials outbound refused at 99%; onboarding mail
|
||
held with a pending approval and no outbox row; approve executed it then.
|
||
|
||
**Two smaller uses** (`33560c7`): `mission_memory::recall` reranks BM25's
|
||
top 8 with one Noul per candidate and drops < 0.3 (keyword overlap made
|
||
every task that names a file recall the MICROVM.md verdict). The
|
||
continuous-research manifest's `topic_tags`, empty since the manifest
|
||
existed, is filled per paper by a Choice over the mission's topics, plus
|
||
a four-level `relevance` Score the ranking phase can start from; PORTICO's
|
||
abstract scored 3.0 at 1.0.
|
||
|
||
**Next for this tier:** accumulate skill-triage agreement rows from real
|
||
missions before any promotion from shadow to selection; a local backend
|
||
only if the vendor dependency bites (logit read-out over the 9B fleet
|
||
model, not a cross-encoder — measured); Slack/A2A intent routing when
|
||
inbound volume justifies it.
|