Files
clawmates/docs/NEXT-SESSION.md
T
Omar SobhandClaude Opus 5 72ba4ba523
deploy / test (push) Failing after 8s
deploy / build (push) Skipped
fix(ci): two runs stomped each other, and the logs blamed the tests
Runs 490 and 491 both failed `test`. Neither failure was in the code.

Runs 490 and 491 started 16 minutes apart and a full suite takes longer
than that, so they overlapped. The first thing a run does is
`docker rm -fv cm-ci-pg` — a name every run shared — so the newer run
deleted the older run's database mid-suite. Both failed, and the failures
read as test failures.

Verified before changing anything: the exact CI command, on gw-04, against
the same warm cargo volumes and a Postgres started exactly as CI starts it,
passes on 128b423 — as do `npm ci`, `typecheck` and `vitest` on that host.
The code was never the problem.

  - `concurrency: deploy-${{ gitea.ref }}` with cancel-in-progress, so runs
    on a ref serialize. A superseded run tests a commit that is no longer
    the tip; finishing it costs 20 minutes to learn something that no longer
    matters.
  - the test Postgres is named per run, so overlap cannot corrupt a run even
    if the concurrency guard is later removed. Impossible rather than
    unlikely.
  - `--shm-size=1g` on it. Docker defaults /dev/shm to 64MB and cm-testkit
    creates a database per test; Postgres exhausts its parallel-query
    segments mid-run and reports `could not resize shared memory segment`
    DURING MIGRATIONS, which reads like a schema fault. Hit locally on
    2026-08-19; scripts/test-server.sh already carries the same flag.

The lesson is the session's own: I twice inferred a cause from a red build
without reading the failure — first node, then dash — and both were wrong.
The answer came from running the job on the runner's own host.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-20 07:00:10 -07:00

7.3 KiB

Where this left off — 2026-08-20

Read CAPABILITY-REVIEW.md for the system picture, TOOL-CALL-ARCHITECTURE.md for the current investigation, and SKILL-USE-BASELINE.md for the measurement.

State of the tree

Everything is pushed. main is at 128b423; the fork's merge/upstream-v0.8.4 is at db1c50966. The local suite is green — 106 test binaries, zero build errors, frontend builds.

⚠ CI IS RED — READ THIS FIRST

Pushing main auto-deploys (.gitea/workflows/deploy.yml: push → test → build → registry → gw-04's 60s timer). The test job is failing, so the build job has not run and NOTHING FROM THESE COMMITS IS DEPLOYED. Production is still serving the previous image. That is a safe state, not a broken one — but it means the work is pushed and not live.

Two runs failed: 490 (b653dbf) and 491 (128b423).

Known and fixed: run 490 failed because vm_tool_gate's shell tests execute the generated hook, which parses its payload with node, and CI's rust:1.96-slim has no node. deploy.yml now installs nodejs, and the gate records an inert marker when node is missing rather than silently allowing. Verified in a rust:1.96-slim container: without node the force-push payload returns 0, with node it returns 2.

Still unexplained: run 491 carried that fix and failed anyway. The cause was NOT determined. Do not guess at it — that is what produced two bad diagnoses already this session.

To get the actual failure, reproduce the CI job locally (the Gitea logs API returns 403 with the token in deploy/compose/.env, which lacks the actions scope):

docker network create cm-ci-net
docker run -d --name cm-ci-pg-local --network cm-ci-net \
  -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres --shm-size=1g postgres:16-alpine
docker run --rm --network cm-ci-net -v "$PWD":/w -w /w \
  -e SQLX_OFFLINE=true -e CARGO_NET_GIT_FETCH_WITH_CLI=true \
  -e FORGE_TOKEN=<GITEA_TOKEN from deploy/compose/.env> \
  -e CM_TEST_DATABASE_URL=postgres://postgres:postgres@cm-ci-pg-local:5432/postgres \
  rust:1.96-slim sh -c 'apt-get update -qq &&
    apt-get install -y -qq pkg-config libssl-dev cmake git nodejs &&
    git config --global url."https://oauth2:[email protected]/".insteadOf "https://git.redclaw.dev/" &&
    cargo test --workspace --no-fail-fast'

A run of exactly this was in flight when the session ended; its output was going to /tmp/cilocal.log.

Candidates worth checking first, none confirmed:

  • something in the new tests that is Linux- or dash-specific beyond the node issue (the gate script itself IS verified under dash);
  • --shm-size — the local test Postgres needed 1g because the suite exhausts Docker's 64MB default mid-run, and CI's cm-ci-pg sets no --shm-size. This surfaces as could not resize shared memory segment … No space left on device during migrations, which reads like a schema fault and is not one. If that is the cause, the fix is --shm-size=1g on CI's postgres, not on our code.

The thing to do first

A runtime image with the stream-json fix is built and NOT deployed.

clawmates-runtime:streamjson     # built 2026-08-19, never run

Nothing has yet confirmed end to end that real tool.call rows land in mission_events from a live mission. That confirmation is the whole point of the change and it is the one step not taken. Until it runs, treat the fix as plausible rather than proven — the parser is unit-tested against a captured real stream, but no mission has driven it.

To do it: deploy the image as the mission runtime, launch a mission, then

SELECT kind, target, count(*) FROM mission_events
 WHERE mission_id = '<id>' GROUP BY 1,2;

Expect tool.call rows on the container tier for the first time. If they do not appear, the next suspects are topology_exec's frame parser and whether the gateway forwards the provider's tool_calls at all — neither has been exercised with a non-empty list.

Then, in order

  1. Prove the PreToolUse gate in a real VM. vm_tool_gate is unit-tested and shell-tested on the host; it has never run in a guest. Four of its bugs were found only by executing the generated script, and two of them (syntax error, IFS) produce a gate that looks installed while being fully closed or fully open. Check /root/toolgate/denied.jsonl after a mission that tries something denied.

  2. Give the direct-session tier a tap. It is the one tier with no observability at all — session_executor::run_session is a single container_exec::exec returning a string. Same settings document, written into a container instead of a VM.

  3. Deploy the door we already built (docs/TOOL-CALL-ARCHITECTURE.md §3). Create /zeroclaw-data/clawmates-mcp.json, add a door-shaped provider alias, bind mission claws to it. This is config, not code — the provider feature is ours and shipped. It would also make clawmates_skills genuinely reachable, which is the precondition for moving skills from inlined bodies to progressive disclosure and making Skill-Use Trigger measurable the way the paper defines it.

  4. Pull upstream's egress policy0db7d999a feat(plugins): add shared egress policy foundation (#9137). We are 218 commits behind; this is the one item identified as worth taking, and it is defence for a problem we have not solved.

Open decisions that are yours

  • Self-authoring scope. Agents now apply their own skill_candidate items with no human click (CLAWMATES_SKILL_SELF_AUTHORING=0 restores the gate). identity_refinement and brain_consolidation still wait for a human, because they change what an agent IS rather than adding a procedure it can consult. Say if you want those autonomous too.
  • Skill-Use Compliance coverage. Most skills still score not_applicable — we cannot tell whether they changed anything. small-focused-commits and tdd-red-green-refactor are the next candidates and both need the repository diff rather than the turn text.

Deliberately not done

  • The mission executor swap (running turns through ProviderExecutor or the chat Runtime). The blockers are structural, not wiring: cm-runtime's files tool rejects absolute paths by construction, shell runs in a per-agent sandbox with no mission mount, ToolContext carries no path or VM handle, and approvals key on (session_id, message_id). The cheap fixes deliver what it was wanted for.
  • cm-brain offline tests — 6 of 9 need live clawbrainhub.com. Stubbing means reproducing an external registry protocol we have no spec for.
  • Graph memory / clawhdf5-agent — in the workspace manifest, used by no crate. Measure against a baseline before migrating.

Two corrections made this session, worth remembering

  • "Missions can't call tools at all" was wrong. They call Bash and Write with permissions pre-accepted. The gap was observing and gating, not having.
  • Raw test counts are a bad coverage metric. They pointed at cm-safety, whose seven tests already covered its critical paths, and missed a Slack replay hole that let one captured request authenticate forever.

The recurring shape, now seven times over: a claim in a comment or a doc, believed and never checked. Every significant finding this session came from running the thing rather than reading about it.