Omar SobhandClaude Opus 5
72ba4ba523
Runs 490 and 491 both failed `test`. Neither failure was in the code.
Runs 490 and 491 started 16 minutes apart and a full suite takes longer
than that, so they overlapped. The first thing a run does is
`docker rm -fv cm-ci-pg` — a name every run shared — so the newer run
deleted the older run's database mid-suite. Both failed, and the failures
read as test failures.
Verified before changing anything: the exact CI command, on gw-04, against
the same warm cargo volumes and a Postgres started exactly as CI starts it,
passes on 128b423 — as do `npm ci`, `typecheck` and `vitest` on that host.
The code was never the problem.
- `concurrency: deploy-${{ gitea.ref }}` with cancel-in-progress, so runs
on a ref serialize. A superseded run tests a commit that is no longer
the tip; finishing it costs 20 minutes to learn something that no longer
matters.
- the test Postgres is named per run, so overlap cannot corrupt a run even
if the concurrency guard is later removed. Impossible rather than
unlikely.
- `--shm-size=1g` on it. Docker defaults /dev/shm to 64MB and cm-testkit
creates a database per test; Postgres exhausts its parallel-query
segments mid-run and reports `could not resize shared memory segment`
DURING MIGRATIONS, which reads like a schema fault. Hit locally on
2026-08-19; scripts/test-server.sh already carries the same flag.
The lesson is the session's own: I twice inferred a cause from a red build
without reading the failure — first node, then dash — and both were wrong.
The answer came from running the job on the runner's own host.
Co-Authored-By: Claude Opus 5 <[email protected]>
153 lines
7.3 KiB
Markdown
153 lines
7.3 KiB
Markdown
# Where this left off — 2026-08-20
|
|
|
|
Read `CAPABILITY-REVIEW.md` for the system picture,
|
|
`TOOL-CALL-ARCHITECTURE.md` for the current investigation, and
|
|
`SKILL-USE-BASELINE.md` for the measurement.
|
|
|
|
## State of the tree
|
|
|
|
**Everything is pushed.** `main` is at `128b423`; the fork's
|
|
`merge/upstream-v0.8.4` is at `db1c50966`. The local suite is green — 106 test
|
|
binaries, zero build errors, frontend builds.
|
|
|
|
## ⚠ CI IS RED — READ THIS FIRST
|
|
|
|
Pushing `main` auto-deploys (`.gitea/workflows/deploy.yml`: push → test → build
|
|
→ registry → gw-04's 60s timer). **The `test` job is failing, so the `build` job
|
|
has not run and NOTHING FROM THESE COMMITS IS DEPLOYED.** Production is still
|
|
serving the previous image. That is a safe state, not a broken one — but it
|
|
means the work is pushed and not live.
|
|
|
|
Two runs failed: 490 (`b653dbf`) and 491 (`128b423`).
|
|
|
|
**Known and fixed:** run 490 failed because `vm_tool_gate`'s shell tests execute
|
|
the generated hook, which parses its payload with `node`, and CI's
|
|
`rust:1.96-slim` has no node. `deploy.yml` now installs `nodejs`, and the gate
|
|
records an `inert` marker when node is missing rather than silently allowing.
|
|
Verified in a `rust:1.96-slim` container: without node the force-push payload
|
|
returns 0, with node it returns 2.
|
|
|
|
**Still unexplained:** run 491 carried that fix and failed anyway. The cause was
|
|
NOT determined. Do not guess at it — that is what produced two bad diagnoses
|
|
already this session.
|
|
|
|
To get the actual failure, reproduce the CI job locally (the Gitea logs API
|
|
returns 403 with the token in `deploy/compose/.env`, which lacks the actions
|
|
scope):
|
|
|
|
```
|
|
docker network create cm-ci-net
|
|
docker run -d --name cm-ci-pg-local --network cm-ci-net \
|
|
-e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres --shm-size=1g postgres:16-alpine
|
|
docker run --rm --network cm-ci-net -v "$PWD":/w -w /w \
|
|
-e SQLX_OFFLINE=true -e CARGO_NET_GIT_FETCH_WITH_CLI=true \
|
|
-e FORGE_TOKEN=<GITEA_TOKEN from deploy/compose/.env> \
|
|
-e CM_TEST_DATABASE_URL=postgres://postgres:postgres@cm-ci-pg-local:5432/postgres \
|
|
rust:1.96-slim sh -c 'apt-get update -qq &&
|
|
apt-get install -y -qq pkg-config libssl-dev cmake git nodejs &&
|
|
git config --global url."https://oauth2:[email protected]/".insteadOf "https://git.redclaw.dev/" &&
|
|
cargo test --workspace --no-fail-fast'
|
|
```
|
|
|
|
A run of exactly this was in flight when the session ended; its output was going
|
|
to `/tmp/cilocal.log`.
|
|
|
|
Candidates worth checking first, none confirmed:
|
|
- something in the new tests that is Linux- or dash-specific beyond the node
|
|
issue (the gate script itself IS verified under dash);
|
|
- `--shm-size` — the local test Postgres needed 1g because the suite exhausts
|
|
Docker's 64MB default mid-run, and CI's `cm-ci-pg` sets no `--shm-size`. This
|
|
surfaces as `could not resize shared memory segment … No space left on device`
|
|
**during migrations**, which reads like a schema fault and is not one. If that
|
|
is the cause, the fix is `--shm-size=1g` on CI's postgres, not on our code.
|
|
|
|
## The thing to do first
|
|
|
|
**A runtime image with the `stream-json` fix is built and NOT deployed.**
|
|
|
|
```
|
|
clawmates-runtime:streamjson # built 2026-08-19, never run
|
|
```
|
|
|
|
Nothing has yet confirmed end to end that real `tool.call` rows land in
|
|
`mission_events` from a live mission. That confirmation is the whole point of
|
|
the change and it is the one step not taken. Until it runs, treat the fix as
|
|
plausible rather than proven — the parser is unit-tested against a captured
|
|
real stream, but no mission has driven it.
|
|
|
|
To do it: deploy the image as the mission runtime, launch a mission, then
|
|
|
|
```sql
|
|
SELECT kind, target, count(*) FROM mission_events
|
|
WHERE mission_id = '<id>' GROUP BY 1,2;
|
|
```
|
|
|
|
Expect `tool.call` rows on the container tier for the first time. If they do not
|
|
appear, the next suspects are `topology_exec`'s frame parser and whether the
|
|
gateway forwards the provider's `tool_calls` at all — neither has been exercised
|
|
with a non-empty list.
|
|
|
|
## Then, in order
|
|
|
|
1. **Prove the PreToolUse gate in a real VM.** `vm_tool_gate` is unit-tested and
|
|
shell-tested on the host; it has never run in a guest. Four of its bugs were
|
|
found only by executing the generated script, and two of them (syntax error,
|
|
`IFS`) produce a gate that looks installed while being fully closed or fully
|
|
open. Check `/root/toolgate/denied.jsonl` after a mission that tries
|
|
something denied.
|
|
|
|
2. **Give the direct-session tier a tap.** It is the one tier with no
|
|
observability at all — `session_executor::run_session` is a single
|
|
`container_exec::exec` returning a string. Same settings document, written
|
|
into a container instead of a VM.
|
|
|
|
3. **Deploy the door we already built** (`docs/TOOL-CALL-ARCHITECTURE.md` §3).
|
|
Create `/zeroclaw-data/clawmates-mcp.json`, add a door-shaped provider alias,
|
|
bind mission claws to it. This is config, not code — the provider feature is
|
|
ours and shipped. It would also make `clawmates_skills` genuinely reachable,
|
|
which is the precondition for moving skills from inlined bodies to
|
|
progressive disclosure and making Skill-Use **Trigger** measurable the way
|
|
the paper defines it.
|
|
|
|
4. **Pull upstream's egress policy** — `0db7d999a feat(plugins): add shared
|
|
egress policy foundation (#9137)`. We are 218 commits behind; this is the one
|
|
item identified as worth taking, and it is defence for a problem we have not
|
|
solved.
|
|
|
|
## Open decisions that are yours
|
|
|
|
- **Self-authoring scope.** Agents now apply their own `skill_candidate` items
|
|
with no human click (`CLAWMATES_SKILL_SELF_AUTHORING=0` restores the gate).
|
|
`identity_refinement` and `brain_consolidation` still wait for a human,
|
|
because they change what an agent IS rather than adding a procedure it can
|
|
consult. Say if you want those autonomous too.
|
|
- **Skill-Use Compliance coverage.** Most skills still score `not_applicable` —
|
|
we cannot tell whether they changed anything. `small-focused-commits` and
|
|
`tdd-red-green-refactor` are the next candidates and both need the repository
|
|
diff rather than the turn text.
|
|
|
|
## Deliberately not done
|
|
|
|
- **The mission executor swap** (running turns through `ProviderExecutor` or the
|
|
chat `Runtime`). The blockers are structural, not wiring: `cm-runtime`'s
|
|
`files` tool rejects absolute paths *by construction*, `shell` runs in a
|
|
per-agent sandbox with no mission mount, `ToolContext` carries no path or VM
|
|
handle, and approvals key on `(session_id, message_id)`. The cheap fixes
|
|
deliver what it was wanted for.
|
|
- **`cm-brain` offline tests** — 6 of 9 need live `clawbrainhub.com`. Stubbing
|
|
means reproducing an external registry protocol we have no spec for.
|
|
- **Graph memory / `clawhdf5-agent`** — in the workspace manifest, used by no
|
|
crate. Measure against a baseline before migrating.
|
|
|
|
## Two corrections made this session, worth remembering
|
|
|
|
- **"Missions can't call tools at all" was wrong.** They call `Bash` and `Write`
|
|
with permissions pre-accepted. The gap was observing and gating, not having.
|
|
- **Raw test counts are a bad coverage metric.** They pointed at `cm-safety`,
|
|
whose seven tests already covered its critical paths, and missed a Slack
|
|
replay hole that let one captured request authenticate forever.
|
|
|
|
The recurring shape, now seven times over: **a claim in a comment or a doc,
|
|
believed and never checked.** Every significant finding this session came from
|
|
running the thing rather than reading about it.
|