Runs 490 and 491 both failed `test`. Neither failure was in the code.
Runs 490 and 491 started 16 minutes apart and a full suite takes longer
than that, so they overlapped. The first thing a run does is
`docker rm -fv cm-ci-pg` — a name every run shared — so the newer run
deleted the older run's database mid-suite. Both failed, and the failures
read as test failures.
Verified before changing anything: the exact CI command, on gw-04, against
the same warm cargo volumes and a Postgres started exactly as CI starts it,
passes on 128b423 — as do `npm ci`, `typecheck` and `vitest` on that host.
The code was never the problem.
- `concurrency: deploy-${{ gitea.ref }}` with cancel-in-progress, so runs
on a ref serialize. A superseded run tests a commit that is no longer
the tip; finishing it costs 20 minutes to learn something that no longer
matters.
- the test Postgres is named per run, so overlap cannot corrupt a run even
if the concurrency guard is later removed. Impossible rather than
unlikely.
- `--shm-size=1g` on it. Docker defaults /dev/shm to 64MB and cm-testkit
creates a database per test; Postgres exhausts its parallel-query
segments mid-run and reports `could not resize shared memory segment`
DURING MIGRATIONS, which reads like a schema fault. Hit locally on
2026-08-19; scripts/test-server.sh already carries the same flag.
The lesson is the session's own: I twice inferred a cause from a red build
without reading the failure — first node, then dash — and both were wrong.
The answer came from running the job on the runner's own host.
Co-Authored-By: Claude Opus 5 <[email protected]>
7.3 KiB
Where this left off — 2026-08-20
Read CAPABILITY-REVIEW.md for the system picture,
TOOL-CALL-ARCHITECTURE.md for the current investigation, and
SKILL-USE-BASELINE.md for the measurement.
State of the tree
Everything is pushed. main is at 128b423; the fork's
merge/upstream-v0.8.4 is at db1c50966. The local suite is green — 106 test
binaries, zero build errors, frontend builds.
⚠ CI IS RED — READ THIS FIRST
Pushing main auto-deploys (.gitea/workflows/deploy.yml: push → test → build
→ registry → gw-04's 60s timer). The test job is failing, so the build job
has not run and NOTHING FROM THESE COMMITS IS DEPLOYED. Production is still
serving the previous image. That is a safe state, not a broken one — but it
means the work is pushed and not live.
Two runs failed: 490 (b653dbf) and 491 (128b423).
Known and fixed: run 490 failed because vm_tool_gate's shell tests execute
the generated hook, which parses its payload with node, and CI's
rust:1.96-slim has no node. deploy.yml now installs nodejs, and the gate
records an inert marker when node is missing rather than silently allowing.
Verified in a rust:1.96-slim container: without node the force-push payload
returns 0, with node it returns 2.
Still unexplained: run 491 carried that fix and failed anyway. The cause was NOT determined. Do not guess at it — that is what produced two bad diagnoses already this session.
To get the actual failure, reproduce the CI job locally (the Gitea logs API
returns 403 with the token in deploy/compose/.env, which lacks the actions
scope):
docker network create cm-ci-net
docker run -d --name cm-ci-pg-local --network cm-ci-net \
-e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres --shm-size=1g postgres:16-alpine
docker run --rm --network cm-ci-net -v "$PWD":/w -w /w \
-e SQLX_OFFLINE=true -e CARGO_NET_GIT_FETCH_WITH_CLI=true \
-e FORGE_TOKEN=<GITEA_TOKEN from deploy/compose/.env> \
-e CM_TEST_DATABASE_URL=postgres://postgres:postgres@cm-ci-pg-local:5432/postgres \
rust:1.96-slim sh -c 'apt-get update -qq &&
apt-get install -y -qq pkg-config libssl-dev cmake git nodejs &&
git config --global url."https://oauth2:[email protected]/".insteadOf "https://git.redclaw.dev/" &&
cargo test --workspace --no-fail-fast'
A run of exactly this was in flight when the session ended; its output was going
to /tmp/cilocal.log.
Candidates worth checking first, none confirmed:
- something in the new tests that is Linux- or dash-specific beyond the node issue (the gate script itself IS verified under dash);
--shm-size— the local test Postgres needed 1g because the suite exhausts Docker's 64MB default mid-run, and CI'scm-ci-pgsets no--shm-size. This surfaces ascould not resize shared memory segment … No space left on deviceduring migrations, which reads like a schema fault and is not one. If that is the cause, the fix is--shm-size=1gon CI's postgres, not on our code.
The thing to do first
A runtime image with the stream-json fix is built and NOT deployed.
clawmates-runtime:streamjson # built 2026-08-19, never run
Nothing has yet confirmed end to end that real tool.call rows land in
mission_events from a live mission. That confirmation is the whole point of
the change and it is the one step not taken. Until it runs, treat the fix as
plausible rather than proven — the parser is unit-tested against a captured
real stream, but no mission has driven it.
To do it: deploy the image as the mission runtime, launch a mission, then
SELECT kind, target, count(*) FROM mission_events
WHERE mission_id = '<id>' GROUP BY 1,2;
Expect tool.call rows on the container tier for the first time. If they do not
appear, the next suspects are topology_exec's frame parser and whether the
gateway forwards the provider's tool_calls at all — neither has been exercised
with a non-empty list.
Then, in order
-
Prove the PreToolUse gate in a real VM.
vm_tool_gateis unit-tested and shell-tested on the host; it has never run in a guest. Four of its bugs were found only by executing the generated script, and two of them (syntax error,IFS) produce a gate that looks installed while being fully closed or fully open. Check/root/toolgate/denied.jsonlafter a mission that tries something denied. -
Give the direct-session tier a tap. It is the one tier with no observability at all —
session_executor::run_sessionis a singlecontainer_exec::execreturning a string. Same settings document, written into a container instead of a VM. -
Deploy the door we already built (
docs/TOOL-CALL-ARCHITECTURE.md§3). Create/zeroclaw-data/clawmates-mcp.json, add a door-shaped provider alias, bind mission claws to it. This is config, not code — the provider feature is ours and shipped. It would also makeclawmates_skillsgenuinely reachable, which is the precondition for moving skills from inlined bodies to progressive disclosure and making Skill-Use Trigger measurable the way the paper defines it. -
Pull upstream's egress policy —
0db7d999a feat(plugins): add shared egress policy foundation (#9137). We are 218 commits behind; this is the one item identified as worth taking, and it is defence for a problem we have not solved.
Open decisions that are yours
- Self-authoring scope. Agents now apply their own
skill_candidateitems with no human click (CLAWMATES_SKILL_SELF_AUTHORING=0restores the gate).identity_refinementandbrain_consolidationstill wait for a human, because they change what an agent IS rather than adding a procedure it can consult. Say if you want those autonomous too. - Skill-Use Compliance coverage. Most skills still score
not_applicable— we cannot tell whether they changed anything.small-focused-commitsandtdd-red-green-refactorare the next candidates and both need the repository diff rather than the turn text.
Deliberately not done
- The mission executor swap (running turns through
ProviderExecutoror the chatRuntime). The blockers are structural, not wiring:cm-runtime'sfilestool rejects absolute paths by construction,shellruns in a per-agent sandbox with no mission mount,ToolContextcarries no path or VM handle, and approvals key on(session_id, message_id). The cheap fixes deliver what it was wanted for. cm-brainoffline tests — 6 of 9 need liveclawbrainhub.com. Stubbing means reproducing an external registry protocol we have no spec for.- Graph memory /
clawhdf5-agent— in the workspace manifest, used by no crate. Measure against a baseline before migrating.
Two corrections made this session, worth remembering
- "Missions can't call tools at all" was wrong. They call
BashandWritewith permissions pre-accepted. The gap was observing and gating, not having. - Raw test counts are a bad coverage metric. They pointed at
cm-safety, whose seven tests already covered its critical paths, and missed a Slack replay hole that let one captured request authenticate forever.
The recurring shape, now seven times over: a claim in a comment or a doc, believed and never checked. Every significant finding this session came from running the thing rather than reading about it.