`select met || ' ' || independent` casts the booleans to `true`/`false`, but the
pattern matched `t`/`f` — psql's *column display* form. So the check reported "no
verdict recorded for the phase" while the row sat in the table saying met=true,
independent=true, glm-4.7.
A check that fails for a reason unrelated to what it checks is worse than no check:
it trains you to ignore the output. The booleans are cast explicitly now so the
shape cannot drift again, and the failure message prints what it actually got.
`verify-mission-delivery.sh microvm` now passes 5/5 against production:
- the agent ran under guest kernel 6.1.128, not the gateway's 6.8.0-124 or the
node's 7.0.0-28 — the one assertion that cannot pass by accident
- the lead delegated to 1 subagent
- the condition was met and judged INDEPENDENTLY by glm-4.7
- the checkout has exactly one writer (uid 65532)
- negative control: a backend no node can run is refused at launch
A REGRESSION I INTRODUCED ONE COMMIT AGO. `capture_finished_coding_phases`
selects on `mp.status = 'completed'`, so the moment an unmet phase correctly began
reporting `failed`, its diff stopped being captured, committed or pushed — the work
was silently discarded. Found by the new harness scenario, whose phase legitimately
missed its condition and then had no artifact at all.
What was produced, and whether the goal was met, are different facts. The artifact
records the first; `mp.status` records the second. Capture now covers terminal
phases (`completed`, `failed`), so a phase that did real work and missed its goal
still delivers a reviewable diff — which is exactly what the next pass needs.
`scripts/verify-mission-delivery.sh microvm` — the regression net this session was
missing. Everything the microVM track proved by hand was guarded by nothing:
- THE KERNEL LINE is the assertion that cannot pass by accident. Every other
check would also pass if the phase had quietly run in a container on the
gateway; only the kernel says WHERE it ran. Compared against the real gateway
and node kernels read at start-up rather than pinned to a version, so
upgrading vmlinux does not manufacture a failure.
- subagent count > 0, from the server's own count of Claude Code's per-subagent
transcripts. Before `Agent` was in the allowlist this was structurally
impossible and nothing said so. A probe that could not run reports "?" and
FAILS the check rather than reading as zero.
- the verdict's judge and whether it was independent.
- negative control, observed passing: a mission whose backend no node can run is
refused at launch and stays draft. Without it the positive scenario would pass
just as well against a scheduler that ignored `backend` entirely — which is
what it did until the first real microvm mission landed on a node with no such
rootfs.
Also fixed in the harness: `api` now sends the JSON body on STDIN (`curl -d @-`)
instead of interpolating it into a single-quoted argument inside a double-quoted
ssh command. A task description containing "the crate's test suite" ended the
quoting and killed the remote shell; two attempts to escape it were themselves
wrong, because the backslashes must survive bash AND sed AND sh. Removing the
interpolation removes the class, and the next author does not need to know that
apostrophes were forbidden.
475 tests pass, clippy clean.
The only rootfs on this track came from clawmates/agent-terminal:dev. Mounted,
it held git and nothing else: no claude, no node, no cargo. A VM booted from it
looks perfect and cannot run a mission, so B4.5 could have been written and
never verified.
images/agent-toolchain — the shared mission toolchain (node 22, git, rust +
cargo-audit, gitleaks/trivy/semgrep, tea/gitea-mcp), lifted from the proven
deploy/clawmates-runtime image minus the zeroclaw daemon: a microVM mission runs
the direct-session model, so there is no daemon to host. A base image rather
than three self-contained Dockerfiles because this layer is ~3 GB and the real
risk is scanner and toolchain versions drifting between per-CLI images — the
evaluator runs the project's own suite to check a claim, so `cargo` present in
one image and absent in another makes the same mission pass or fail by backend
with nothing saying why.
images/agent-claude — plan A6, first of three: the pinned CLI and its env
contract only, so bumping Claude Code does not rebuild the toolchain and cannot
disturb agent-kimi / agent-glm. HOME=/root with an empty .claude for B4.4 to
inject into; no ANTHROPIC_API_KEY, since it silently overrides the subscription
OAuth we already pay for.
Both the builder and the node selftest now ASK the guest for the CLI the image
is named for, instead of trusting the name. `required_cli` maps claude/kimi/glm
to a probe; an unrecognised backend reports unchecked and prints SKIP rather
than passing quietly.
Verified on tank:
- rootfs-claude.ext4 boots; git, node, cargo, a real git commit all work
- `claude --version` → 2.1.220 over vsock, in both the builder and
`--vm-selftest` (11/11, create 1498 ms)
- negative control: the same builder run against agent-terminal with
FC_CLI forced reports `cli rc=127 claude: not found` and exits 1, so the
green result above is a measurement and not a default
- `claude -p hello` fails with "Not logged in · Please run /login" — the CLI
runs headless in the VM, and B4.4 only has to supply the credential
- no leaked firecracker processes or vm dirs afterwards
437 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
The python guest agent only ever worked because Firecracker's CI Ubuntu
image happens to ship python3. NONE of our images do — agent-base has
neither python nor git, agent-terminal has git but no python — so it
could never have run in a real mission rootfs. An agent that dictates
what must be installed in the image has the dependency backwards.
crates/bins/fcagent is a 905K static x86_64-unknown-linux-musl binary
that needs nothing from the rootfs it is dropped into. The wire is
unchanged on purpose — 4-byte BE length + JSON, ops ping/exec/put/get —
so microvm.rs and microvm_client.rs needed no edit at all.
std has no AF_VSOCK and the workspace denies `unsafe`, so it uses the
`vsock` crate. `process_group(0)` gives each command its own group without
unsafe, so a command that spawns background children can be killed
wholesale rather than outliving the run.
A unit test caught a bug that would have broken EVERY exec: sourcing the
image-env file with `. env.sh 2>/dev/null; cmd` returns rc=1 WITHOUT
running cmd, because `.` on a missing file makes a non-interactive POSIX
shell exit immediately. On any rootfs lacking that file every command
would have failed while looking like an ordinary non-zero exit. Guarded
with `if [ -f ]` now.
Other places a failure must not borrow an outcome's representation: a
killed command reports ok:false with no rc (not rc=124, which would read
as a build failure); `get` on a missing path is an error, not an empty
archive; a signalled process reports 128+signal rather than success.
Verified on tank: --vm-selftest still 8/8 with the agent swapped
(create 949ms, wire identical), fc-node-setup 8/8, and — the point of the
change — a rootfs built from clawmates/agent-terminal:dev, which has NO
python3, boots and reports `git version 2.39.5` from inside the VM.
Also fixes a shell bug in fc-build-rootfs.sh: $HOME in a double-quoted
default expanded on this Mac, so it looked for the node's binary under
/Users/quantum on a Linux host.
Co-Authored-By: Claude Opus 5 <[email protected]>
Until now microVMs booted Firecracker's CI Ubuntu image with a python
guest agent bolted on: no git, no toolchain, no CLI. Fine for proving
vsock, useless for running a mission.
Builds FROM a Docker image rather than debootstrapping, because the
per-CLI images (agent-claude / agent-kimi / agent-glm, per A6) are
already Dockerfiles with a tested env contract. Rebuilding that as a VM
image by hand would mean maintaining the same facts twice and finding the
drift in production.
Two things the obvious version gets wrong and this does not:
- `docker export` gives the filesystem with NONE of the image metadata:
no ENV, no ENTRYPOINT, no WORKDIR. A CLI relying on ENV PATH or HOME
would silently behave differently in the VM. The env is extracted
separately and written to /etc/profile.d.
- the ext4 is filled through a mount, not `mkfs -d`, which cannot
handle the device nodes and hard links a container image may contain
and fails late and cryptically when it hits one.
The guest agent is copied from the golden rootfs rather than re-emitted,
so there is ONE copy of the protocol on the node instead of two that can
drift.
It boots what it builds and asks the image for what a mission needs —
git, the profile env, a writable /mission — rather than assuming. An
image that builds and cannot boot is worse than no image, because it
looks finished.
FINDING, and it blocks B4.2: NONE of our images ship python3, so the
python guest agent cannot run in any of them. agent-terminal has git but
no python; agent-base has neither. The guest agent must not dictate the
image's contents — it needs to be a static binary. This script correctly
refuses to build an image whose agent cannot run, so the failure is
visible rather than a VM that boots into nothing.
Co-Authored-By: Claude Opus 5 <[email protected]>
create / inject / exec / collect / destroy / list, riding the node's
existing frame dispatch ({t, id, …} -> {t:"result", id, ok, output}), so
no protocol change was needed. Control is length-prefixed JSON over
vsock; the serial console stays a log, because feeding a guest over stdin
races its startup and arrives half-consumed.
DEVIATION FROM THE PLAN, deliberately: this does NOT implement
cm_sandbox::SandboxDriver. That trait is container-shaped —
attach_pty/resize_pty/argv exec — while missions need
create -> inject -> run -> collect -> destroy. Conforming would mean
building PTY-over-vsock and window-resize semantics that no mission path
calls, purely to satisfy a signature. We give up automatic RemoteDriver
marshalling; orphan reaping is a label/id sweep either way.
Three traps from the B0 spike are handled in code rather than remembered:
- Firecracker does NOT unlink its vsock UDS on exit, so destroy unlinks
it explicitly, and the selftest ASSERTS it is gone. Assuming the VM
tidies up after itself is how the mission checkout accumulated four
uid bugs.
- firecracker is spawned via setsid and killed as a process GROUP, so a
background child cannot outlive the VM holding its workdir open.
- create does not return until the guest agent has answered a ping. A
VM that booted but serves nothing is worse than one that failed, so a
half-created VM is destroyed rather than left registered.
A vm id becomes a path component, so ids are restricted to [A-Za-z0-9_-]
and REJECTED rather than sanitised — a caller that sent `../../etc`
wanted something we should not guess at.
Verified on tank through the real Rust path, as the daemon user, with no
sudo: `clawmates-node --vm-selftest` -> 8/8, create in 986ms, and the
host left with zero firecracker processes and zero VM directories. The
selftest asserts every step, including that a destroyed VM can no longer
be exec'd; a test that only reports the steps it completed cannot
distinguish "passed" from "stopped early".
Co-Authored-By: Claude Opus 5 <[email protected]>
The capability probe reported `kvm: false` on tank and morpheus while the
device sat right there: /dev/kvm is `crw-rw---- root:kvm` and the kvm
group was EMPTY, so the daemon — an ordinary user — could not open it.
The B0 spike missed this entirely because it ran everything under sudo.
This is exactly why the probe opens the device rather than stat-ing it;
a stat-based check would have reported both nodes capable and every
microvm mission would have failed at launch instead of at placement.
fc-node-setup.sh now fixes the group itself, or says precisely what to
run when it cannot.
Co-Authored-By: Claude Opus 5 <[email protected]>
Phase B step 0. Before writing any driver, establish that Firecracker
works on this hardware — the plan called it greenfield, and an
orchestrator built against an unproven runtime is a lot of code betting
on an assumption.
It works, and comfortably: a microVM boots, runs our init, writes a
file and shuts down in ~650-910ms wall clock, with the kernel reaching
our init at 234ms. Host->guest RPC over vsock (AF_VSOCK port 9001, no
network stack) round-trips in 27ms.
The script installs and then PROVES, because installing is not working.
It reports success only after a VM has actually booted and run our code.
Four findings from the spike that the driver must account for:
- Firecracker does NOT unlink its vsock UDS on exit, and leaves it
owned by whoever ran the VM. A driver running as anyone else cannot
clean it up — the same uid trap that cost this codebase four bugs on
the mission checkout. The driver owns the socket path lifecycle.
- tank's FORWARD policy is DROP (Tailscale/Docker), confirming the
article's warning: VM networking rules must be inserted at position
1, not appended, or return traffic dies silently.
- Feeding commands to the guest over the serial console races the
shell's startup and arrives half-consumed (`# ho FC-GUEST-ALIVE`).
The guest runs an init script; stdin is not a control channel.
- `sha256sum -c` compares by filename, so a download saved under any
other name fails for a reason unrelated to integrity. A check that
fails for the wrong reason teaches you to ignore it — compare the
hashes directly.
tank and morpheus are ready. architect requires interactive sudo, so it
is deliberately not provisioned rather than worked around.
Co-Authored-By: Claude Opus 5 <[email protected]>
The last open item in the silent-success class: a coding phase that
changed no files reported `completed` — the same status a phase gets for
delivering tested, reviewed, pushed work. Mission `019fcf62` completed
that way with its agents silently unpinned from the repo, and nothing in
the platform disagreed; it was found by a script diffing the forge.
The verdict is applied at capture rather than at completion, because
capture selects on `status = 'completed'` — the platform does not know
whether a phase produced anything until after it has already finished.
Three conditions must hold before failing a phase, because a false
positive here fails honest work: the phase is a coding phase (research
phases legitimately write nothing to the tree), the diff was actually
computed (an uncomputable diff also reports zero files — blaming the
agent for a platform fault is the same defect wearing different
clothes), and `allow_empty` is not set. Only an explicit `true` opts
out, so a typo leaves the check armed. Registered in phase_config with
its reader named, per the seam-2 rule.
Also closes an ordering hazard this exposed: capture is batched and runs
after a phase completes, so a backlogged mission could close as
'completed' and only then have capture discover an empty phase — leaving
a 'completed' mission holding a 'failed' phase, unfixable because the
mission-close CASE only touches 'running' rows. A repo-bearing mission
now waits for its work to be captured before closing.
Adds a `noop` scenario to the harness: a phase told to change nothing,
which PASSES only when the phase comes back `failed`. Same discipline as
the uid self-test — a check that has never been seen to fire has not
been shown to work.
Co-Authored-By: Claude Opus 5 <[email protected]>
A verification run against the deployed stack found a chain mission whose
phase 0 reported `completed` with zero files, no commit error and no push
error — indistinguishable from a phase that correctly had nothing to do.
Three separate defects had to line up, each of them the same shape: a
failure sharing its representation with a legitimate negative result.
1. `pin_agent_workspaces` embedded the whole config in one `sh -c` argv.
That works until the file grows — config gains a block per provisioned
claw — then fails with `argument list too long`. Now written through
the tar upload API, which has no argv limit, so the failure mode is
gone rather than merely further away.
2. A failed pin was logged "(continuing)". Without the pin, agents write
to their sandboxes and the committer finds nothing in /mission/repo —
the mission cannot deliver, so the launch now fails where someone is
still looking. The restart that applies the pin is fatal for the same
reason.
3. `capture_phase_diff_at` swallowed `git diff` failures with
`unwrap_or_default`, so an unreadable base landed `empty: true,
files_changed: 0` — byte-identical to an honest no-op. The error is now
recorded as `diff_error`, and an empty patch that came from a failed
diff is no longer trusted to mean an unchanged tree.
Adds scripts/verify-mission-delivery.sh, which found #1 and #2 on its
first real run. Its probes are fail-closed: no placeholder values, a
self-test that proves the uid probe can detect the split it looks for,
and FAIL-NORUN for a scenario that never executed. Its own first version
had this bug too — a `die` inside `$(...)` exited the subshell, so a run
that could not authenticate printed "all checks passed" and exited 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
The post-transfer verification added in bb34ef1 failed every deploy: it
compared `.Id` between build host and target, and a BuildKit image on the
build host carries attestation manifests that `docker save | docker load`
does not reproduce. The same build legitimately arrives with a different
Id and a different reported Size — tank had agent-base:dev at 28 MB /
363f23b7, gw-04 at 74 MB / edd46f95, both from the identical build.
`.Created` comes from the config blob, survives the round trip unchanged,
and is what actually answers "is the new build here". Both hosts reported
2026-08-02T16:43:00.599937575-07:00, which is how the false positive was
identified rather than assumed.
The verification itself stays — the truncation it guards against is real.
This corrects what it compares.
Co-Authored-By: Claude Opus 5 <[email protected]>
`docker save | docker load` across two SSH connections spliced through a
workstation truncates when either side stalls — observed as `unexpected
EOF` mid-deploy. Nothing checked afterwards, and `docker load` can exit 0
on a short stream, so a partially-populated image could ship to every
fleet node and look like a success.
Now compressed, pipefail-guarded, and verified by comparing image IDs on
the target after the transfer, with one retry for the transient stall.
A failed transfer fails the deploy rather than passing quietly.
The runtime image no longer travels this path at all: it is registry-
hosted now (100.94.185.103:5000/clawmates-runtime:v083-toolchain), built
from deploy/clawmates-runtime/Dockerfile on tank. Only agent-base /
agent-browser / agent-terminal still need save|load, because they exist
in no registry.
Co-Authored-By: Claude Opus 5 <[email protected]>
Deploys were verifying green and then silently reverting minutes later.
Cause: gw-04 does not deploy from this script's recreate at all.
`clawmates-deploy.timer` runs every 60s, pulls
`$REGISTRY/clawmates/<svc>:latest`, and rolls the stack onto it whenever
the running image differs — so the local `docker tag` + `--force-recreate`
this script did was reverted within the minute. Its own log shows it:
server drift: running=<the new image> target=<the old :latest>
rolling: server frontend
The registry's `:latest` is therefore the only thing that decides what
prod runs — and `docker push …:latest` does NOT reliably move it here.
When the manifest already exists under another tag (the `main-<sha>` we
push immediately before), the push reports a digest but `:latest` keeps
resolving to the old image. Pushing a brand-new tag works, so it is
specific to overwriting an existing one.
Writing the manifest to the tag over the registry HTTP API does move it
(GET the main-<sha> manifest, PUT that body to :latest → 201), after
which the timer converges prod on its own. So:
- repoint :latest via manifest PUT from the build host, failing loudly on
a non-2xx instead of assuming the push landed
- roll gw-04 immediately rather than waiting up to 60s for the timer
- verify against the resolved :latest (what compose and the timer both
deploy from) instead of a main-<sha> tag that is never pulled there
Note for future debugging: image IDs differ per host for the same tag
(buildx OCI index — tank holds the index digest, gw-04 the resolved
platform image), so the trustworthy check is grepping the deployed binary
for a string only the new code contains.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`docker-compose pull server frontend` pulls `:latest`, and the registry
served a STALE manifest for that mutable tag: a deploy pushed
`main-9bc5f6a` correctly, but the gateway's `pull :latest` reported
"image is up to date" and left the previous image running. The verify
step caught it (running 9f2349 = main-0a647c0, expected bbf19f7e), so
the deploy failed loudly rather than silently — but it still could not
ship.
Immutable tags always resolve correctly, so pull `main-<sha>` and retag
it to `:latest` locally on the gateway, then recreate with `--no-deps`
and no compose pull. `:latest` is now just a local alias satisfying the
compose file's image reference; the sha tag is the source of truth.
Also switch the recreate to `--no-deps` (compose v1 has no
`--no-recreate-deps`) so a server/frontend deploy stops recreating
postgres.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
rsync --delete excludes frontend/public/dl/, so the directory does not exist
on the build host and the `cp` of clawmates-node into it aborted the deploy
(set -e) before anything was pushed.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The gateway compose pulls server/frontend from the web-01 registry
(100.94.185.103:5000, tags main-<sha> + :latest), so `docker save | docker
load` + a local retag does NOT stick — the next `docker-compose up` pulls
`:latest` and silently reverts to the last-pushed image (a green edge on the
old image hid this). Rewrite the server/frontend path to: build on the build
host → tag :latest + :main-<sha> → push to the registry → `compose pull +
up --force-recreate` in /opt/clawmates (the real project dir, not the stale
/root/clawmates) → verify the RUNNING image id equals the pushed one (fail
loudly on mismatch instead of trusting HTTP 200). Agent :dev images stay on
the save|load path (not in any registry).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two of the fleet's Gitea Actions runners (morpheus, architect) already
had 8080 permanently bound by unrelated services (nginx on morpheus,
envio-hasura on architect) — every e2e run scheduled there died at
playwright's webServer preflight with "http://127.0.0.1:8080/healthz is
already used". 18080 is unused across morpheus/tank/architect.
Swap 8080 → 18080 in the eight e2e-scoped sites: clawmates.e2e.toml
(listen_addr + slack base_url + oauth redirect_base), dex.yaml (client
redirect URIs must match backend), playwright.config.ts + tests
(p4-slack, p6-oauth), the http.ts dev-fallback origin, and the two
shell scripts (e2e-backend safety check, rehearse-install healthz probe).
Prod compose (/opt/clawmates/docker-compose.yml on gw-04) is untouched;
prod continues to expose the server on 8080 internally on the compose
network (that's per-network, not host-shared).
Apple-silicon Mac nodes (ghost) run an arm64 Docker VM, so the amd64 images run
under Rosetta. This builds native arm64 agent images (run on an arm64 mac, no QEMU)
and loads them onto the arm64 Mac node(s). amd64 (scripts/deploy.sh) still covers
the Linux nodes + Intel Macs.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
CI no longer references k8s; delete the dead k8s surface:
- deploy/helm/ (the chart), ci/check-helm.sh, scripts/netpol-cluster.sh
- cm-sandbox: the feature-gated K8sDriver (src/k8s.rs) + k8s_security test +
the `k8s`/`k8s-tests` features + the optional kube/k8s-openapi/rustls deps
(Cargo.lock drops the kube-rs tree). Nothing outside cm-sandbox referenced it.
Docker (bollard) DockerDriver is the sole sandbox driver. cm-sandbox + cm-runtime
compile, fmt + clippy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- frontend/.../terminal/core.ts: `useResilientTerminal` owns the xterm lifecycle
(init, fit, debounced resize, reconnect, refit-on-visible) with a pluggable
transport. Two connectors: `wsConnector` (agent terminal → container PTY over WS)
and `nodeWebrtcConnector` (node terminal → host PTY, direct WebRTC DataChannel
with WS-relay fallback, input buffered during the race). The agent terminal now
gets the node terminal's robust reconnect for free; both share one xterm setup.
- TerminalApp (agent) + NodeTerminalApp (node) reduced to thin wrappers over the
core — net ~330 lines of duplicated transport/reconnect/xterm code removed.
- scripts/deploy.sh: the deploys were manual, so the locally-built agent images
(agent-base/browser/terminal — not in any registry) were never shipped and 404'd
on provision. The script always (re)builds + loads them onto gw-04 AND every
fleet node, with a skip-if-identical guard so unchanged images aren't re-transferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The Playwright backend harness now aborts if something else (e.g. the
compose stack) is already on :8080, instead of letting reuseExistingServer
silently point every journey at the wrong backend with the wrong seed —
the failure mode that surfaced mid-restyle. Also gitignore the local
data/ blob-store artifacts.
Co-Authored-By: Claude Fable 5 <[email protected]>
- scripts/netpol-cluster.sh: a kind cluster with Calico (default CNI
disabled) — the only way to PROVE the §15 default-deny NetworkPolicy,
since kindnet accepts the object but never enforces it. New live test
on that cluster: outbound connect to 1.1.1.1 dropped, DNS egress
dropped, while API-server exec keeps working (not pod network).
Kernel-level enforcement of the sandbox egress claim, demonstrated
- K8sDriver::connect_with_context: pin a kubeconfig context instead of
ambient. The whole k8s suite now pins its cluster explicitly — the
netpol cluster's creation had silently switched the current context
and stranded the seccomp test on the wrong cluster (fixed and made
impossible to recur)
- CI: netpol-cluster up + calico egress test in the sandbox-k8s job
- Visual-regression lock (plan P6): @visual Playwright spec with
animation-disabled, masked-dynamic-region screenshots of login,
workspace home, chat welcome, computer panel, credits; darwin
baselines committed (5 PNGs); CI excludes @visual until linux
baselines are generated there. Full local suite: 33 journeys
165 Rust tests + 5 live kind tests (2 clusters) + 33 journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
Full-depth rename per the approved plan; the 'claw' product vocabulary
(claws, /claws routes, clawId, Claw Chat) stays — it is now the brand.
- Display brand: Clawmates (manifest, titles, hero, login/rail logo
'clawmates'); default host app.clawmates.work; registry
ghcr.io/clawmates
- Crates tc-* -> cm-* (16 crates + all imports); binaries
clawmates-server/broker/bundler; images clawmates/*; env prefix
CLAWMATES_* (+ CM_TEST_DATABASE_URL / CM_LIVE_LLM); config
clawmates.toml; helm chart deploy/helm/clawmates with clawmates-*
resources; db names clawmates*; sockets /run/clawmates; cookie
cm_session; kind cluster clawmates-test; seccomp node profile
clawmates-agent-profile.json
- All 9 Playwright brand assertions updated in lockstep; historical
spec document left untouched as the only remaining 'TeamClaw'
- Local env migrated: dev pg clawmates-dev-pg/clawmates_dev, shared
test server clawmates-test-pg, kind cluster recreated with image +
profile, compose images rebuilt under clawmates/*
Verified end to end: 161 Rust + 68 frontend tests, 29 Playwright
journeys, 4 live kind tests, helm/install/LOC/placeholder gates, and
the clean-room install rehearsal serving the clawmates login page from
a signed bundle of the rebuilt images.
Co-Authored-By: Claude Fable 5 <[email protected]>
- tc-telemetry: fmt subscriber always; with [telemetry] otlp_endpoint
set, spans batch-export over OTLP/HTTP. Tested against a REAL OTLP
receiver decoding the actual protobuf (official proto types): the
emitted span and service.name arrive on the wire. No endpoint = no
export = no network (air-gap stance). tower-http TraceLayer gives
every API request a span
- The broker finally has its own image (images/broker.Dockerfile,
9.5MB from scratch) — the Helm chart referenced one that never
existed — and the compose deployment now RUNS the broker, sharing a
socket volume with the server (the unix-socket equivalent of the K8s
sidecar). Compose secret flows were silently dead before this
- server.Dockerfile fixes surfaced by the rehearsal: the workspace
build needs tools/ (bundler joined the workspace) and
images/seccomp/ (include_str! profile) in the build context
- scripts/rehearse-install.sh (plan: clean-VM rehearsal): assembles a
REAL signed bundle from the built images (server/frontend/broker/
postgres/socket-proxy), runs the customer path — offline verify,
docker load, compose up — and asserts /healthz plus the served login
page before teardown. Passing locally; wired as a release.yml step,
which also builds/ships the broker + socket-proxy images now
161 Rust tests + 29 journeys; clean-room rehearsal green.
Co-Authored-By: Claude Fable 5 <[email protected]>
- PWA (§16): hand-rolled 60-line service worker (network-first pages with
offline fallback, cache-first hashed statics, /api NEVER touched — SSE
and approvals stay live), app manifest with §2 identity, stdlib-
generated coral claw icons, prod-only registration. E2E asserts
manifest, real PNG icons, an ACTIVATED service worker, and the /api
bypass. (Serwist was tried and dropped: its webpack plugin fights
Next 16's Turbopack builds; sixty lines we own beat a plugin we fight.)
- Route motion (§3): (workspace) template re-mounts per navigation with a
quiet fade-rise, zeroed under prefers-reduced-motion. The a11y sweep
now settles running animations before scanning — axe was reading
mid-fade opacity as contrast failures
- OAuth browser flow vs REAL dex: the e2e harness boots dexidp/dex with
static client + password; the journey drives the actual dex login form
from /api/apps/oauth/start through the callback 303 and asserts the
app reads connected (closing the P4 deferral honestly)
- release.yml: tag-triggered — builds all four images + postgres, saves
tarballs, assembles the SIGNED air-gapped bundle (compose, config,
migrations, seccomp profile, installer, bundler binary), derives the
public key via the new Could not find command "pubkey". subcommand (tested), verifies
the bundle customer-style with the public half only, attaches tarball
+ public key to the GitHub release
153 Rust + 63 frontend tests + 29 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- LlmEvent::Usage across all three providers (Scripted deterministic
word-count accounting; Anthropic message_start/delta usage; OpenAI-compat
stream_options include_usage)
- tc-billing: ceil(tokens/1000) min 1 credit; lots drain oldest-first under
FOR UPDATE; balance clamps at zero while the usage ledger records the
full obligation; promo codes redeem exactly once via CAS (migration 0006)
- Runtime charges every completed run (billing failure never fails a run);
proven: 1 token in + 3 out -> 1 credit deducted
- API: GET /api/team/usage, POST /api/credits/redeem (409 on reuse, audited)
- Credits page: balance, 7-day usage meter with runway estimate, PromoRedeem
- /claws/new is the full §9 wizard: ?step=identity|access|slack deep-linked
progress, accent swatches + name randomizer, access toggles, optional
Slack step, explicit review-and-confirm (creation = live agent), animated
provisioning state -> straight into chat
- E2E: chat decrements the visible balance and fills the usage meter;
WELCOME500 adds exactly 500 once then refuses; wizard round trip
140 Rust + 63 frontend tests + 23 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- app_connections repo; POST /api/apps/connect (keys/basic): the credential
goes to the secret broker over its socket and only the encrypted ref lands
in the row; disconnect endpoint; /api/apps directory merged with live
connection status; audit rows for connect/disconnect
- Broker protocol: InvokeHttp carries a JSON body
- slack.post tool (SendsExternally -> gated): marked broker_executed — the
runtime skips its own grant consumption and the BROKER independently
verifies + consumes the single-use grant, then calls Slack with the bot
token injected; the runtime never sees the credential
- Config: [broker] socket_path + [slack] base_url; e2e harness spawns the
real teamclaw-broker daemon and the server hosts an e2e-only /__slack sink
- SlackApp: Connection tab stores the token via the broker; connected state
- Integration test: blocked while pending -> approved -> sink received
exactly one post with 'Bearer xoxb-test-token' -> grant replay refused
- E2E journey: connect Slack in the panel -> gated post card with preview ->
sink empty while pending -> approve -> exactly one post, queue clear
133 Rust + 63 frontend tests + 21 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
Omar Sobh