5afcf63324651e3e528c4807d02141dda85237a5
43
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
f56d41f5b7 |
feat(backend): local-ornith — a mission backend served by the node's own GPU
Claude Code pointed at the Ollama already installed on every GPU node. Ollama has served a native Anthropic-compatible /v1/messages since v0.14, so this is an env contract rather than a translation layer — the fourth variation on the same idea as agent-glm and agent-kimi. The route is NOT the egress proxy, and that is the design. `egress` speaks CONNECT, takes a destination from the guest, resolves it and decides; every one of those powers is a liability, which is why it refuses non-443 ports and IP literals after a unit test caught them being bypassed. Routing a local model through it would have meant relaxing both. `local_model` is the opposite shape: there is no destination in the protocol. fcagent listens on guest 127.0.0.1:11434 and pumps to vsock 9003; the node splices that onto its own 127.0.0.1:11434 and copies bytes. A compromised guest cannot redirect it because there is nothing to redirect — it is a pipe, not a proxy, and strictly narrower than anything an allow-list could express. The bytes never touch a network, so there is no wire for TLS to protect, and Ollama stays bound to loopback rather than being exposed on the tailnet. The socket is bound only for a backend declared to use a local model, so a `local-ornith` VM reaches the forge through egress and nothing else, while every other backend's guest port simply refuses. Both halves have negative controls. `scripts/fleet-model-setup.sh` exists because of one measurement: stock ornith:9b reported input_tokens=2050 for a 48000-word prompt and answered as though nothing had been dropped. Ollama's default window is ~2K whatever the model card says, and it truncates silently — the exact failure an agent turn would hit and never report. The script pins num_ctx=131072 into a derived tag and then PROVES both the window and tool calling before declaring success. Verified on architect: ~65536 words -> 65604 input tokens, stop_reason=tool_use. Placement needs no new capability key: building the rootfs only on GPU nodes means `nodes::online_for_backend`'s existing `rootfs @> ["local-ornith"]` predicate does the affinity, so morpheus never offers the backend. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar Sobh
|
099a716bfd |
fix(egress): a backend is defined in two maps, and the canary only had one
First canary run failed: phase failed, nothing delivered, and the streamed log said exactly why — "Failed to authenticate. API Error: 403 api.anthropic.com is not on the egress allow-list". Not a 2.1.226 regression. `canary-claude` was added to the server's credential map and not to the node's `provider_hosts`, so the VM booted with a valid subscription token and a door that only opened onto the forge. The fail-closed branch was working correctly: a backend nobody taught that function about reaches no model API, deliberately, so it cannot silently borrow another provider's door. Both maps now name it, each pointing at the other, with a test asserting the canary reaches the same provider as `claude` AND that unknown backends still resolve to nothing. Worth noting what made this a five-second diagnosis instead of an afternoon: the live log streaming built earlier today. The failure was a 403 inside a microVM that no longer exists, and its reason was sitting in the run's checkpoint. |
||
|
Omar Sobh
|
09afa7e7ff |
fix(node): aborting the tail at turn end raced the flush that matters most
Composed runs streamed NOTHING while solo runs streamed fine — same code path, `HubVms::run` is a straight passthrough, and the node logged a tail starting for all five graph nodes with the correct outer run id. The difference was timing. `claude -p ... | tee` makes stdout a PIPE, so the CLI block-buffers and flushes at EXIT. The most valuable output — the agent's summary of what it did — arrives in the instant the turn ends. The node aborted the tail the moment `handle_op` returned, so that flush was a race: a solo turn (minutes long, output already flushed by size) won it and streamed 337 bytes; each node of a composed run (~20s) lost it and streamed zero. The tail now DRAINS. A flag is set when the turn returns, and the loop exits only after a pass that read nothing new — checked AFTER a read, never before one, because exiting on the flag alone would drop exactly the bytes this exists to capture. Bounded by a 20s timeout with the abort kept as a backstop rather than the mechanism, so a VM that stopped answering cannot hold the task open. Worth naming: 5 tails started, 5 logged cleanly, 0 bytes arrived. Every individual step reported success and the feature did nothing — the same shape as the empty Live tab this whole thread began with, one layer down. |
||
|
Omar Sobh
|
28090d1de0 |
fix(node): the log tail gave up before the turn wrote its first byte
First live test of the streaming path: mission passed 6/6, `checkpoint.log` was 0 bytes, and the node logged nothing at all. `stream_vm_log` treated "no progress" as "the turn finished writing". But the guest's `tail` reports EOF after every idle window, and the FIRST idle window is always the one before any output exists — the VM is still booting and the CLI still starting. So the tail returned `at == 0`, the node concluded the turn was done, and it stopped seconds into a run that then went on for minutes. The abort is the terminator, not idleness: the caller already aborts this task when the exec returns, so waiting cannot outlive the turn. No-progress now sleeps and retries instead of returning. Also logs when a tail STARTS. The bug was invisible in exactly the way this session keeps finding: silence on the success path, silence on the give-up path, and an empty Live tab that looked identical to a feature nobody had wired. Method note, since it cost time: I tried to confirm the deployed binary by grepping it for `vm_out` and found zero — then found zero for `pty_out` and `vm_exec` too, in a binary whose PTY streaming demonstrably works. Binary-grep is not a reliable presence test for these literals; `stream_vm_log` and `tail of` being present is what actually showed the code had shipped. |
||
|
Omar Sobh
|
0b89b8316c |
feat(observability): stream a microVM turn's stdout/stderr to the platform live
The Live tab showed nothing while a turn ran, and the agent's own account of it
went to stderr on the node and nowhere a user could reach. This is the path that
carries it.
The blocker was the guest agent. `fcagent` handled one connection at a time,
inline, so during an hour-long turn the VM accepted nothing — which is why every
existing probe (subagents, stop-gate blocks, cap) runs AFTER the turn rather than
during it. It now spawns a thread per connection, wrapped in `catch_unwind`
because this process is pid 1: a panic used to take the accept loop with it, and
an unbootable VM is a far worse outcome than a missing log. A failed spawn logs
and keeps accepting rather than dropping the listener.
PROVED against a live VM before building on it, since "sound reasoning about this
system" and "measurement" have diverged repeatedly today. Patched rootfs, booted
under Firecracker, ran an 8s exec and a concurrent tail:
exec took 8.0s ok=True
+0.0s 'line1\nline2\n' +1.2s 'line4\n' +3.2s 'line6\n' +6.0s 'DONE\n'
VERDICT: CONCURRENT — tail returned data before exec finished
The rest is the pattern the terminal already uses. New `tail` op streams a file
by OFFSET (so a dropped link resumes instead of replaying, and the tail always
terminates — one that never returns pins a thread for the life of the VM). The
node follows the log alongside the turn and pushes `Uplink::VmOut { run_id, at,
data }` over the WebSocket it already holds, mirroring `PtyOut`. The server does
what `PtyOut` deliberately does not: it APPENDS to the run's checkpoint as well
as fanning out, because a terminal has no history worth keeping and a mission log
is the record of what the agent did. `run_events_sse` emits the new bytes as
`step` events, which the live pane already renders — no frontend change.
The turn is `tee`d, not redirected: the file feeds the live stream and stdout
still becomes `VmOutcome::summary`. A redirect would have produced a live view
and an empty summary, which is the same green-and-empty shape as the bug this
fixes. Tested, along with the log living outside the collected tree so it never
lands in a user's delivered diff.
246 lib tests, 20 binaries; node and fcagent build clean.
|
||
|
Omar SobhandClaude Opus 5
|
742724e53c |
feat(fleet): Kimi as a microVM backend — the URL settled by measurement
The base URL took three measurements to find, and the first two were wrong in instructive ways. `api.moonshot.ai/anthropic/v1/messages` EXISTS and speaks the protocol — it answers with Moonshot's own structured error rather than a 404. It also rejects an `sk-kimi-` key, because it belongs to the platform.moonshot.ai account namespace. Two endpoints that both "work" for different accounts is precisely the shape that makes a guessed URL look like a broken key, and it is why this was refused rather than guessed for as long as it was. The Kimi CODE service is the one an `sk-kimi-` key belongs to: `POST https://api.kimi.com/coding/v1/messages` returns a real Anthropic Messages body — `msg_` id, `content` blocks, a `thinking` block with a signature. So `ANTHROPIC_BASE_URL=https://api.kimi.com/coding`, WITHOUT the `/v1`: Claude Code appends `/v1/messages` itself, and `/v1/v1/messages` would 404 in a way that reads as a broken image rather than a bad URL. Two more measured, each otherwise a silent failure at the first turn: `Authorization: Bearer` is accepted (so ANTHROPIC_AUTH_TOKEN is the right injection channel), and a `claude-*` model id is ACCEPTED AND ANSWERED — Kimi maps it onto `kimi-for-coding` exactly as z.ai does, so no ANTHROPIC_MODEL override is needed. Claude Code rather than Moonshot's own `kimi` CLI, deliberately. The mission harness is Claude-Code-shaped throughout: `--agents` JSON roles, the verifier's tool allowlist, the `Stop` hook behind the completion gate, the per-subagent transcripts counted as delegation evidence. `kimi` has none of those flags — its equivalents are TOML files and markdown agent dirs — so using it would mean a second executor with its own untested failure modes. TWO STALE MAPS, caught by the rootfs harness refusing to bless the image: both `fc-build-rootfs.sh` and the node's `required_cli` expected backend `kimi` to contain Moonshot's `kimi` binary. That assumption predates the measurement, and it failed a rootfs that was correct. Both now say `claude` for glm and kimi alike — the binary is the same in all three images; only the endpoint differs. Egress for `kimi` is `api.kimi.com` alone: not moonshot.ai (wrong namespace), not z.ai, not Anthropic. Asserted both ways, like the other two. The image and rootfs are built on tank and the rootfs passes all four checks (boots, git, writable /mission, `claude --version`). 534 tests, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
d3a53e7bf1 |
fix(fleet): a VM reaches its OWN provider and no other, measured not assumed
The GLM backend works — and proving it produced a better boundary than the one I shipped an hour ago. WHAT THE FIRST GLM MISSION SHOWED. It completed, and the delivered file said the model was "claude-opus-5". The node's egress log said the VM had dialled `api.anthropic.com` five times before `api.z.ai`. Either reading alone is consistent with a "GLM backend" that silently runs Anthropic — the exact silent-success shape this project keeps closing — so I did not accept either. THE ABLATION, run on tank rather than reasoned about: deny `anthropic.com` at the proxy and run the same mission again. It **completed**, dialling only `api.z.ai`. So the completions genuinely come from z.ai; Claude Code's calls to anthropic.com are its own telemetry, not its model traffic. And that same agent — served exclusively by z.ai, with Anthropic unreachable — still described itself as "Claude Opus 5 (1M context)". **A model's account of which model it is has no evidential value.** The proxy's log of which host it dialled does. This is the `uname -r` lesson again in a new place: ask the infrastructure, not the agent. So the allow-list is now PER BACKEND rather than a union: a `claude` VM reaches Anthropic and the forge, a `glm` VM reaches z.ai and the forge, and neither can reach the other's endpoint. A union was defensible when it was one host; once the measurement showed a GLM VM never needs Anthropic, keeping it would mean a credential mix-up upstream could still put one provider's secret on another provider's wire. Now it fails at a closed door instead. An unknown backend gets the forge and NO model API — it cannot run anyway, and borrowing somebody else's door is the failure this split prevents. An explicit `CLAWMATES_FC_EGRESS_ALLOW` still wins outright: an operator who set it drew a boundary on purpose. `DEFAULT_ALLOW` is deleted rather than left beside the new function, so there is one answer to "what may a mission reach" and not two. 534 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f7f3dfe495 |
feat(fleet): GLM as a real microVM backend, and per-role models for claws
Three threads, all of which end at the same place: a mission whose verifier does
not share a model with the coder it reviews.
**GLM has a credential contract now.** `microvm_credential_for` returned one env
var name, which quietly assumed every provider reads its secret from the same
place Anthropic does. It returns a `Credential { source, target }` instead —
z.ai's key lives in the server's `ZAI_API_KEY` and Claude Code reads it as
`ANTHROPIC_AUTH_TOKEN`, and collapsing those two names is what forces a guess at
the other end. A wrong guess here sends one provider's credential to another
provider's endpoint.
`images/agent-glm` is the same CLI at the same pinned version as `agent-claude`
with `ANTHROPIC_BASE_URL` baked in. The split is deliberate: the ENDPOINT is a
property of the image, the CREDENTIAL is a property of the turn. That makes the
dangerous mix-up unrepresentable — a GLM VM cannot be handed an Anthropic
subscription token, and a claude VM cannot be pointed at z.ai. Asserted both
ways, because "the GLM VM must not carry CLAUDE_CODE_OAUTH_TOKEN" is the
property that costs a credential if it ever stops holding.
Kimi stays refused. `KIMI_API_KEY` is set and Moonshot serves an
Anthropic-compatible API, but I have not verified its base URL against the
running service, and this function is precisely where guessing a URL is
expensive. It becomes an arm the day someone measures it.
`api.z.ai` joins the node's default egress allow-list. A default that cannot
run the images we ship is a trap rather than a policy — the alternative is an
operator discovering it as a hung agent with no model access.
**Per-role models for claws** (migration 0071). `template_roles` had no model
column, so `mint_team_from_template` bound every role of every mission team to
one literal — a template whose whole point is an independent reviewer minted a
reviewer sharing a model with the coder. A role may now name its own; roles that
say nothing still take the mint's default, so every template written before this
behaves exactly as it did. The literal is now that default rather than a
hardcode.
**A harness scenario for the roster flow.** `verify-mission-delivery.sh roster`
runs the whole Slice 5 loop — planner proposes, human approves, mission runs —
and asserts the roster LANDED on the mission row rather than trusting the API's
answer. That distinction is not theoretical: the first live approval returned an
error while leaving the proposal marked approved.
Built and proven on tank ahead of the deploy: `clawmates/agent-glm:dev` reports
`2.1.223` and `BASE=https://api.z.ai/api/anthropic`, and
`fc-build-rootfs.sh … glm 8G` boots a VM from it that has git, can write
/mission, and answers `claude --version`.
533 tests pass, clippy clean. Migration 0071.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar Sobh
|
d49acaed5e |
fix(missions): exclude build output from COLLECT too, not just inject
The other half of the same bug. The previous commit filtered `mission_fs::pack_dir` (the inject side) and left the guest's `op_get` tarring everything, so the re-run that proved the #54 fix — it survived 480s where it used to die at 210 — still lost its work to `vm_collect ... node timed out`. Two modules written, four subagents used, nothing delivered. `op_get` now takes an `exclude` list, sent by the host from `mission_fs::transport_excludes()` — the same list `mission_delivery` uses for the diff. Policy in one place, applied at both ends of the wire. Matched on directory NAME at any depth, so a workspace's per-crate `target/` dirs are all covered, with a test that plants a nested one and asserts it does not come along. Also proven by that run: the worker no longer kills a live microVM run. It ran 480 seconds straight through the 180s requeue window and the 210s mark where mission 019fd43e died, untouched. And `subagents: 4` — the team addendum did drive real fan-out this time, which is the first evidence the Slice 3 switch does anything. 483 tests pass, clippy clean. Still to prove: a >3-minute mission that actually DELIVERS. The collect fix is tested in isolation but has not yet carried a real mission's work back, and the guest agent needs rebuilding into the rootfs before it can. |
||
|
Omar Sobh
|
4efcde9d4f |
fix(missions): #54 — the worker was killing live microVM runs at 180 seconds
My hypothesis in #54 was WRONG, and it was wrong because I built it on a bad measurement: `grep -c 'microvm phase'` returned 0, so I concluded the completion log never printed and blamed the 15-minute reaper. The line was there all along, at 14:17:45. The real cause is worse. `requeue_stale` has NO TIER FILTER. A microvm run's `updated_at` is written once at insert and never again — it is driven by a `tokio::spawn` that owns it start to finish, and nothing in `microvm_executor` writes `topology_runs`. So at 180s the sweeper declared a perfectly healthy run stale and flipped it to `queued`; `claim_next_queued` (no tier filter either) handed it to the worker; `run_job` tried to parse the microvm graph placeholder, which `TopologyGraph` cannot deserialize; and it failed the run with "missing or invalid graph". Mission 019fd43e: run created 14:11:16, mission failed ~14:14:46. 210 seconds — the 180s window plus a tick. The agent went on working and finished at 14:17:45 with three modules written, by which time the phase was already dead and the VM was orphaned. A firecracker process was still alive 1h37m later. THE UNCOMFORTABLE PART: every microVM mission that appeared to work this session did so only by finishing inside three minutes. The 90-second ones dodged this. The harness scenario dodges it. Nothing about that was visible. `WORKER_DRIVEN_TIERS` (team, company, org, swarm, compare) is now the allowlist for all three sweep paths — claim, requeue, reap. An allowlist rather than a denylist so the next self-driven tier is safe by default instead of exposed until someone remembers the file. `tier='session'` had exactly the same exposure and is covered too. A unit test asserts microvm and session are NOT in it, next to the code that inserts them. Two more fixes from the same wreckage: - `destroy` reported `killed: pgid.is_some()` — true whenever there was a pgid to signal, whether or not anything died. It now sends the signal, polls /proc for the group leader, retries, and reports what it OBSERVED; `signalled` keeps the old meaning so "nothing to kill" is distinguishable from "it would not die". - the run-status update is now guarded with `AND status <> 'cancelled'`. An operator cancelling is a decision; this task reporting an outcome minutes later is an observation, and it must not overwrite one with the other. And the root cause of the collect timeout itself: `mission_fs::pack_dir` shipped `target/` in both directions. `mission_delivery` has excluded build output from the DIFF since day one; the TRANSPORT never knew. The host checkout was 9.4 MB of which 8.9 MB was `target/`, tarred and base64'd over vsock each way. `EXCLUDED_PATHS` is now one list shared by both layers, matched on directory name at any depth so a workspace's per-crate `target/` dirs are all covered. 483 tests pass, clippy clean. |
||
|
Omar Sobh
|
d9f53a3f96 |
fix(fleet): placement requires the backend's rootfs image, not just KVM
The first real microVM mission was placed on morpheus because it reports
{"microvm": true}, while only tank had rootfs-claude.ext4. It failed by name
rather than booting the wrong image — but whether a mission ran came down to
which capable node was listed first, which is a coin flip dressed as scheduling.
`missions.backend` was invisible to the scheduler.
The node now enumerates the images on its disk and reports them as a `rootfs`
ARRAY. `microvm::available_backends` lives beside `rootfs_for`, its inverse,
because the two must agree on what a backend name means; split apart, one drifts
and the scheduler starts promising images the booter cannot find. It only
advertises names `rootfs_for` would accept, and reports an empty array rather than
omitting the key — set_capabilities REPLACES, so a deleted image stops being
advertised instead of leaving a stale claim.
`nodes::online_for_backend` requires microvm AND that the node's list contains the
mission's backend. A node on an older daemon has no `rootfs` key and matches
nothing: unknown is not permission, the same treatment every other capability
gets. `backend_key` maps the three spellings of "the default image" to the one
name the node advertises, and is tested — a mismatch there would reject every node
for an ordinary mission with no backend set.
The launch error now names both halves of the fix, since "no capable node" was
true but unhelpful when the node was capable and merely lacked the image.
Mission gains `backend` on the domain struct; it was a column the executor read
from the phase query while the struct that placement uses could not see it.
464 tests pass, clippy clean.
|
||
|
Omar SobhandClaude Opus 5
|
ebdba34da6 |
feat(fleet): B4.6 — a microVM reaches the API through a vsock CONNECT proxy, with an allow-list
The guest still has no network interface, and now that is the design rather than
a gap. Its only route out is an HTTP CONNECT proxy: agent CLI -> 127.0.0.1:3128
in the guest -> vsock 9002 -> a per-VM Unix socket on the host -> TLS to an
allow-listed host.
Why not TAP + iptables, which is what the Firecracker write-ups do — measured,
not argued:
- `ip tuntap add` is DENIED to the daemon user (needs CAP_NET_ADMIN), so TAP
would need root to pre-provision devices, the same privilege detour the
loop-mounted rootfs already forced.
- tank's FORWARD policy is DROP with Docker and Tailscale chains, so rules
would have to be inserted at position 1; appended ones die silently.
- a leaked TAP is a new class of host litter to reap.
CONNECT needs no privilege at all and is better on the merits: the client hands
us the HOSTNAME, so resolution happens host-side and the guest needs no DNS or
resolv.conf; the allow-list is by name, not address; and nothing in the guest can
reach the network except through one function. The guest end parses nothing and
enforces nothing, so a compromised agent cannot argue with the policy.
Rests on one measured fact: `claude` honours HTTPS_PROXY. With the proxy at a
closed port, `claude -p` fails ConnectionRefused instead of answering.
THE RESULT: a real agent turn now completes inside a VM with no network card, on
subscription auth — `claude -p` replies VM-OK. The selftest asks for it whenever
CLAUDE_CODE_OAUTH_TOKEN is present and SKIPS loudly when it is not, since it
spends a little of the plan.
The audit log earns its keep immediately: during that turn the proxy logged
`egress DENIED http-intake.logs.us5.datadoghq.com` — the CLI's telemetry, which
the mission container permits today without anyone deciding to.
Three bugs found by the checks rather than by review:
- `env_pairs` returned early when a caller sent no env, so the proxy address
was never added and `curl` in a VM with a working tunnel reported "Could not
resolve host". Absent env means "the caller sent none", not "this command
needs no environment".
- the deny check PASSED for the wrong reason — DNS was failing, so nothing was
refused by the allow-list at all. It now requires a 403 from the proxy, so it
cannot go green on a broken tunnel.
- `host_allowed` accepted `evil.test/api.anthropic.com`, which ends with an
allowed suffix. Hostnames are now validated against a character class, which
also refuses IP literals so an address cannot sidestep a list of names.
- `BufReader::into_inner()` discards buffered bytes: wrapping the stream twice
would have dropped the start of the TLS handshake and stalled a tunnel that
looked established. One reader now spans the request, and anything buffered
past the headers is forwarded as payload.
`iproute2` is in agent-toolchain because it is load-bearing: the guest's `lo`
starts DOWN, and while it is down a listener on loopback BINDS and then refuses
every connection with ENETUNREACH. fcagent finds `ip` by absolute path — as pid 1
its PATH comes from the kernel, and execvp's fallback excludes /usr/sbin, where
Debian puts it.
Egress needs both ends up, so `create` reports `egress` and the guest's `ping`
reports its own half. A VM without it is legal but never silent.
Verified on tank: 16/16 with backend=claude (create 1428 ms), 12/12 on the
default rootfs, no leaked processes, VM dirs or proxy sockets. 457 tests pass,
clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
c3297b86cf |
feat(fleet): B4.4 — credentials reach the microVM guest as exec env, and a bad entry refuses the exec
`claude -p` in the VM failed with "Not logged in". The credential now travels on
the exec op: `env` on `vm_exec` → fcagent → the command's environment. An env var
rather than a file because the per-VM rootfs dies with the VM but an env var
never touches the guest disk at all.
**Every problem in an env entry fails the exec.** The tempting alternative —
skip the entry we cannot use and run anyway — produces a `claude -p` with no
credential, and that does not error, it HANGS. A phase stuck at `running` for
ten minutes with nothing in the logs is exactly what a missing token looked like
on the container path. Names are validated ('=' or NUL would define a different
variable than the one asked for via putenv semantics), values must be strings,
and errors name the key and never the value — an error string travels back over
the wire and into logs.
One list of which credentials travel: `forwarded_provider_env` reuses
`forwarded_provider_keys`, and the container path now reads it too. If the two
execution paths diverged, a mission would behave differently depending on where
it landed — including the expensive way, where one path forwards
ANTHROPIC_API_KEY and bills it while the other uses the subscription. A blank
value is omitted rather than forwarded empty, so `claude` reports having no
credential instead of failing authentication with one.
Verified on tank (`--vm-selftest` backend=claude, 13/13, create 1532 ms): an
injected var reaches the guest command over the real vsock wire, and an
unusable entry comes back ok:false with no rc.
FINDING — the CLI leg remains UNPROVEN, and deliberately so. The guest has no
network interface: `create` writes boot-source, drives, machine-config and vsock
and no `network-interfaces` key, and a booted guest has no routes, no
resolv.conf, no DNS and no TCP. So `claude -p` cannot reach the API whatever
credential it holds. Injecting the real token would have proven nothing, because
the failure would have been network and not auth. Filed as B4.6 (task #49) with
the TAP-vs-vsock-proxy trade-off; B4.5 is now blocked on it.
444 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
0c291ed1bb |
feat(fleet): B4.4a — a real agent-claude microVM image, and a check that it has an agent in it
The only rootfs on this track came from clawmates/agent-terminal:dev. Mounted,
it held git and nothing else: no claude, no node, no cargo. A VM booted from it
looks perfect and cannot run a mission, so B4.5 could have been written and
never verified.
images/agent-toolchain — the shared mission toolchain (node 22, git, rust +
cargo-audit, gitleaks/trivy/semgrep, tea/gitea-mcp), lifted from the proven
deploy/clawmates-runtime image minus the zeroclaw daemon: a microVM mission runs
the direct-session model, so there is no daemon to host. A base image rather
than three self-contained Dockerfiles because this layer is ~3 GB and the real
risk is scanner and toolchain versions drifting between per-CLI images — the
evaluator runs the project's own suite to check a claim, so `cargo` present in
one image and absent in another makes the same mission pass or fail by backend
with nothing saying why.
images/agent-claude — plan A6, first of three: the pinned CLI and its env
contract only, so bumping Claude Code does not rebuild the toolchain and cannot
disturb agent-kimi / agent-glm. HOME=/root with an empty .claude for B4.4 to
inject into; no ANTHROPIC_API_KEY, since it silently overrides the subscription
OAuth we already pay for.
Both the builder and the node selftest now ASK the guest for the CLI the image
is named for, instead of trusting the name. `required_cli` maps claude/kimi/glm
to a probe; an unrecognised backend reports unchecked and prints SKIP rather
than passing quietly.
Verified on tank:
- rootfs-claude.ext4 boots; git, node, cargo, a real git commit all work
- `claude --version` → 2.1.220 over vsock, in both the builder and
`--vm-selftest` (11/11, create 1498 ms)
- negative control: the same builder run against agent-terminal with
FC_CLI forced reports `cli rc=127 claude: not found` and exits 1, so the
green result above is a measurement and not a default
- `claude -p hello` fails with "Not logged in · Please run /login" — the CLI
runs headless in the VM, and B4.4 only has to supply the credential
- no leaked firecracker processes or vm dirs afterwards
437 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
6687f8b808 |
feat(fleet): B4.3 — per-mission rootfs selection (missions.backend)
`vm_create` takes a backend name and boots `rootfs-<backend>.ext4`; NULL or "default" boots the golden image. Makes the per-CLI images from B4.1 actually reachable (one image per CLI, per A6). A missing image is an ERROR naming the file and how to build it, never a quiet fall back to the default. That fallback is the tempting version and the wrong one: it would run a claude mission in a kimi VM, or in a rootfs with no CLI at all, and report success for whatever came out. Verified on real hardware, not just in a unit test — the selftest asks for an image that does not exist and FAILS if it boots. `create` now reports the rootfs that actually booted, not the one that was requested, so a mission artifact can show the wrong VM ran. The migration adds no CHECK constraint listing the CLIs. Which images exist is a property of the NODES, not the schema; a constraint would need migrating for every new image while still not guaranteeing the image exists anywhere. The node validates and names what is missing. Backend names are `[A-Za-z0-9_-]` and rejected rather than sanitised, since they become filenames. Verified on tank: default backend 8/8; `CLAWMATES_FC_BACKEND=agent-terminal` 9/9 including the absent-image check, create in 910ms on a rootfs built from a real Docker image. 435 tests green, no leaked processes or VM dirs. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
02ba557c3e |
feat(fleet): B3 — server-side microVM client over NodeHub
cm_api::microvm_client::MicroVm wraps the node's vm_* ops as typed calls
over the existing hub request/response channel: create / inject / exec /
collect / destroy, plus list() for reaping. No new transport.
Fixes a wire-contract mismatch B2 would have shipped. `Uplink::Result`
declares `output: String`, but the node's vm_* handler returned a JSON
object. The frame then failed to deserialize and hit the uplink match's
`Err(_) => {}` arm, so the reply VANISHED and every vm_* call would have
timed out after 20s with nothing anywhere explaining why. The node now
sends a string, matching the contract rather than what looked tidier.
That silent arm is fixed too: an unparseable frame now logs the node, the
parse error and the frame head, and says explicitly that the request it
was answering will time out. It is the arm that would have hidden this.
Two more places where a failure must not borrow a legitimate outcome's
representation:
- vm_exec returning no `rc` is an error, not a zero. A missing exit code
means the guest did not report one; reading it as success is how a
failed command becomes a passing phase.
- vm_collect on a missing path is an error, not an empty archive — an
empty tar looks exactly like a run that produced nothing.
Timeouts: the hub's deadline is the guest's plus 30s, saturating. A
caller passing a huge budget would otherwise wrap to a tiny timeout and
turn a long agent turn into a spurious transport failure. clippy caught
the tautological assertion in the first version of that test, which is
what surfaced the overflow.
Verified: `--vm-selftest` on tank still 8/8 after the output-type change
(create 950ms), 427 tests green.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
2d04c5e257 |
feat(fleet): B2 — vm_* node ops for Firecracker microVMs
create / inject / exec / collect / destroy / list, riding the node's
existing frame dispatch ({t, id, …} -> {t:"result", id, ok, output}), so
no protocol change was needed. Control is length-prefixed JSON over
vsock; the serial console stays a log, because feeding a guest over stdin
races its startup and arrives half-consumed.
DEVIATION FROM THE PLAN, deliberately: this does NOT implement
cm_sandbox::SandboxDriver. That trait is container-shaped —
attach_pty/resize_pty/argv exec — while missions need
create -> inject -> run -> collect -> destroy. Conforming would mean
building PTY-over-vsock and window-resize semantics that no mission path
calls, purely to satisfy a signature. We give up automatic RemoteDriver
marshalling; orphan reaping is a label/id sweep either way.
Three traps from the B0 spike are handled in code rather than remembered:
- Firecracker does NOT unlink its vsock UDS on exit, so destroy unlinks
it explicitly, and the selftest ASSERTS it is gone. Assuming the VM
tidies up after itself is how the mission checkout accumulated four
uid bugs.
- firecracker is spawned via setsid and killed as a process GROUP, so a
background child cannot outlive the VM holding its workdir open.
- create does not return until the guest agent has answered a ping. A
VM that booted but serves nothing is worse than one that failed, so a
half-created VM is destroyed rather than left registered.
A vm id becomes a path component, so ids are restricted to [A-Za-z0-9_-]
and REJECTED rather than sanitised — a caller that sent `../../etc`
wanted something we should not guess at.
Verified on tank through the real Rust path, as the daemon user, with no
sudo: `clawmates-node --vm-selftest` -> 8/8, create in 986ms, and the
host left with zero firecracker processes and zero VM directories. The
selftest asserts every step, including that a destroyed VM can no longer
be exec'd; a test that only reports the steps it completed cannot
distinguish "passed" from "stopped early".
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
0f7fa31f86 |
feat(fleet): B1 — microvm runtime kind and KVM placement predicate
Phase B step 1, on top of the B0 spike that proved microVMs boot here.
KVM is a HARD predicate, not a preference. gw-04 — where every mission
runs today — is itself a VM without nested virtualisation and has no
/dev/kvm, so a microvm mission landing there cannot start at all. The
scheduler therefore has to be able to tell nodes apart, which means the
node has to report what it can host.
Nodes gain a `capabilities` jsonb, populated from a probe on the node
rather than from configuration: /dev/kvm either exists there or it does
not, and nothing on the server can make it appear. The probe OPENS the
device rather than stat-ing it, because it can exist while being
unopenable (wrong group, or a container without the device passed
through) — which is precisely how firecracker will fail.
`microvm` requires BOTH kvm and a firecracker binary. A node with KVM
but no binary looks capable by the obvious test and fails at launch; a
node with the binary but no KVM is gw-04.
Placement fails the launch when no capable node exists, rather than
letting a mission sit in 'running' with nowhere to run. An explicit
target_node_id is treated as a request, not a guarantee — it is honoured
only if that node actually reports the capability.
`capabilities` defaults to '{}' NOT NULL so a node that has never
reported fails every predicate: an unqueried node and an incapable node
must be indistinguishable to the scheduler, because scheduling onto a
node whose abilities are unknown is how you get a mission that cannot
start and does not say why. The report replaces rather than merges, so a
capability the node has LOST disappears instead of leaving a stale true.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar Sobh
|
d8c8793c4a |
ci fixes: cargo fmt, eslint entities, max-lines split
CI on
|
||
|
Omar Sobh
|
bf4af48c80 |
herdr phase 3: INFRA tier Herdr sessions surface
New INFRA category "Herdr sessions" (purple sparkles icon between
Fleet and Local hardware). Shows a card per online fleet node with:
- Node name + hostname + IP
- Per-workspace agent state pills (working / blocked / done /
idle / unknown), colored dots + pane count
- "Open" button → renders that node's full Herdr TUI inline via
xterm.js (same nodeHerdrConnector + WebRTC-with-fallback the
MissionCanvas Live Pane uses)
Backend:
- node daemon: herdr_workspaces + herdr_snapshot ops
(`herdr workspace list`, `herdr api snapshot`)
- fleet_herdr::snapshot helper on top of hub.call_timeout
- GET /api/nodes/{id}/herdr/session route
Fetch flow: /api/nodes filtered to status='online' → for each,
/api/nodes/{id}/herdr/session in parallel. Snapshot errors surface
per-card without failing the whole grid.
The "Open" xterm is separate from the MissionCanvas Live Pane —
this one is scoped to the whole node's Herdr TUI (any workspace),
not a specific mission's pane. Operator toggles between nodes via
the buttons.
Verified: cargo check --workspace + tsc --noEmit both green.
|
||
|
Omar Sobh
|
a5588b0289 |
herdr phase 2: Live Pane tab (xterm.js → node's herdr TUI)
The killer UX feature: click a mission's Live Pane tab and watch the
actual Herdr TUI on the target node in the browser — cursor, colors,
tool output, all live. WebRTC DataChannel direct where the browser
can reach the node peer-to-peer, WS-relayed fallback otherwise
(same auto-negotiation the INFRA node terminal already uses).
Zero new deployment infra — reuses the existing terminal_ticket +
terminal_ws + PTY-over-control-channel machinery. The one primitive
we grew: PtyTarget::Command variant so the node can spawn an
arbitrary program (\`herdr\`) in the PTY instead of the login shell.
Node daemon (clawmates-node):
- PtyTarget grows a Command { argv } variant
- spawn_command_pty resolves bare names against user + system bin
dirs (matches how tool_update finds claude/kimi)
- PtyTarget::from_frame reads the `command` array from the pty_open
frame; precedence Command > Container > Host
cm-api:
- NodeHub::open_pty grows an optional command argv; when set, the
frame carries it and the daemon spawns the program directly.
- routes::nodes::TermCtrl gains a `command: Vec<String>`; the
fallback branch threads it through.
Frontend:
- core.ts::webrtcConnector takes an optional commandOverride
that ships inside the fallback frame
- nodeHerdrConnector(nodeId) — mints the standard ticket + WS URL
but overrides command to ["herdr"]
- MissionCanvas grows a "pane" tab, visible only when
runtime_kind='local_herdr'. LivePane subcomponent uses xterm.js
(already a workspace dep) via useResilientTerminal, shows a
connecting/relayed/direct pill in the corner.
To watch a mission live: pick "On a fleet node (Herdr)" + target
node in the wizard, launch, click Pane tab → node's Herdr TUI
appears. Navigate to the mission workspace in the Herdr sidebar
(mouse or prefix+w) to zoom into the mission's pane.
Focus-a-specific-pane-directly is a later enhancement — Herdr has
no CLI arg for it yet, so operator navigates the sidebar for now.
Verified: cargo check --workspace + tsc --noEmit both green.
|
||
|
Omar Sobh
|
47f986257f |
herdr phase 1b: fleet_herdr dispatch module + node daemon ops
The second-runtime path uses the existing NodeHub control channel —
NOT SSH. Node daemons already accept typed ops over their outbound
websocket; adding three herdr_* ops keeps everything on the auth
model that already works fleet-wide (control-channel token, no new
SSH key management, no server-container-mounted keys).
Node daemon (clawmates-node):
- New herdr_op handler in main.rs dispatching:
* herdr_dispatch — workspace create + pane split + rename + run
* herdr_status — pane get JSON (agent, agent_status, cwd)
* herdr_read — recent-unwrapped scrollback, N lines
- Herdr binary resolved from ~/.local/bin, brew, /usr/local/bin.
Missing binary returns clean error so cm-api can distinguish
"node not set up for Herdr yet" from "Herdr op failed".
cm-api:
- crates/cm-api/src/fleet_herdr.rs — dispatch / status /
read_transcript / wait_for_completion helpers on top of
hub.call_timeout(). wait_for_completion polls until agent_status
hits 'done' or an idle-after-working state, matching the SKILL
file's "either idle or done is completed" semantic.
- routes::missions::herdr_dispatch — POST /api/missions/{id}/
herdr-dispatch { cli, prompt }. Requires runtime_kind = 'local_herdr'
and target_node_id set. Manual trigger so Phase 1b is exercisable
end-to-end before Phase 1c wires the wizard + orchestrator.
Not yet wired: mission_orchestrator::on_launch still ignores
runtime_kind. Phase 1c adds the wizard picker AND the on_launch
branch that auto-dispatches on draft→running for local_herdr
missions. This commit only adds the primitives.
Verified: SQLX_OFFLINE=true cargo check --workspace green.
Phase 0 (Herdr install on fleet nodes) is the blocker to actually
exercising this end-to-end.
|
||
|
Omar Sobh
|
51d501e36e |
clawmates-node: bound write.send with 10s timeout so half-open TCP can't wedge daemon
Under a half-open TCP (server side closed, client OS still buffering writes), the daemon's write.send() inside the tokio::select! branch blocks forever. tokio::select does not preempt a running future, so the whole loop freezes — idle_tick never gets to check last_rx, no 'channel ended' log ever fires, and the daemon silently spins on a dead socket for hours. Observed on architect Jul 5 2026: daemon connected at 12:59:13, heartbeats worked for ~2.5 min, then went silent. TCP session showed ESTABLISHED on the node side, no ESTABLISHED on the gateway side. Restarting the daemon 'fixed' it — but it re-hung within minutes. Fix: wrap both write.send() call sites in a tokio::time::timeout of WRITE_DEADLINE=10s. If a send stalls past that, we log and return Ok() to trigger the main-loop reconnect. Short enough that it fires long before the 40s read-idle would (which was our only escape hatch and never triggered because the loop was frozen). |
||
|
Omar SobhandClaude Opus 4.8
|
1a531e91cd |
Fleet tools: add Rust version card (probe rustc + nightly latest + rustup update)
Adds Rust to the per-node dev-tool cards: daemon probes rustc → reports 'rust'; nightly checker fetches latest stable from GitHub rust-lang/rust; GET endpoint maps rust→Rust (after docker); one-click update runs 'rustup update stable'. Frontend is data-driven (no change). $HOME/.cargo/bin added to probe candidate dirs. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
b82a71d4ea |
Fleet tools: 8s probe cap + post-update refresh retries (updates now reflect)
After 'claude update' replaces the binary, the daemon's re-probe ran the fresh binary which macOS Gatekeeper re-verifies (>2s) — the 2s probe cap missed the new version, so the UI didn't refresh (update worked but looked stale). Bump the cap to 8s; frontend polls the tools endpoint a few times post-update to catch the re-probe. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
ed339c2121 |
Fleet tools: one-click per-node update (Phase 2)
The ↑ badge on each tool card is now a button: confirm → POST
/api/nodes/{id}/tools/{tool}/update → daemon runs the tool's own updater + re-probes.
- daemon: tool_update op (spawned task so the 170s update can't stall the WS loop;
re-probes + re-sends node_tools after). Fixed command allow-list (no arbitrary
shell): claude/glm → `claude update`; kimi → `uv tool upgrade kimi-cli`; ollama →
brew upgrade (mac) / install.sh (linux); else unsupported. 4KB output cap.
- cm-api: call_timeout/request_timeout (long ops); POST .../tools/{tool}/update
(workspace-scoped, allow-list) → {ok,output}.
- frontend: ↑latest becomes an Update button → confirm → spinner → refresh/err.
Note: claude/kimi/glm are user-space (no sudo); ollama on Linux uses install.sh
(needs sudo — works on passwordless nodes, returns an error otherwise; surfaced in UI).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
9263418fcb |
Fleet: per-node dev-tool version cards + nightly latest-check (Phase 1, read-only)
Each node card now shows installed versions of Docker / Claude Code / Kimi / GLM /
Ollama (conditional per node) under the ssh card, with an "update available" badge.
- daemon: probe_tools() finds docker/claude/kimi-cli/ollama across candidate bin dirs,
extracts semver from --version, reports {"t":"node_tools",...} on connect + every 15m.
- migration node_tools + tool_latest; cm-db repo node_tools (upsert/list/latest).
- cm-api: fleet.rs NodeTools uplink → upsert; tool_versions.rs spawn_latest_checker
(24h, npm/pypi/github; docker display-only); GET /api/nodes/{id}/tools (glm mirrors
claude). Spawned in clawmates-server.
- frontend: NodeTools cards on each HostCard with the ↑latest badge.
Phase 2 (one-click update execution) intentionally deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
3554a3aaf2 |
CI: remove k8s stages, fix the Docker-level pipeline green
Survey + fixes so the pipeline passes at the Docker level (no k8s).
- Remove k8s: drop the `sandbox-k8s` job (kind/Calico/--features k8s-tests) and the
"Helm chart lints" gate step. release.yml was already k8s-clean.
- Rust job:
- `cargo fmt --all` — fix pre-existing formatting drift (fmt --check was failing).
- clippy -D warnings: fix 3 lib warnings (cm-brain sort_by_key→Reverse, cm-api
fleet.rs doc list indentation, node_rules map_or→is_none_or).
- Regenerate the .sqlx offline cache (was missing the cm-runtime run_loop test
query → offline compile failed). DB-backed tests use testcontainers at runtime.
- Set SQLX_OFFLINE=true on the rust + e2e jobs so query! macros compile against
the committed cache deterministically (no DB needed at compile time).
- Frontend job:
- Fix the 1 ESLint error (useAgentTelemetry: no setState-synchronously-in-effect;
tag the slice with agentId + derive null on mismatch).
- Fix 2 stale panel-params tests (`terminal` is a valid app id now; assert the
current APP_IDS + use a genuinely-unknown id for the reject case).
Verified locally: fmt clean, clippy --all-targets -D warnings clean (offline),
frontend lint 0 errors, tsc clean, 86/86 frontend tests pass, build OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
12212c72ac |
Daemon: PtyTarget — exec a terminal PTY into an agent container (node-placed terminal foundation)
Generalize the daemon's PTY spawn so a session can target either the node's host shell (today) or `docker exec -it <container> tmux …` (the node-placed agent terminal, which shares the container's node-local ~/drives). A `container` (+ optional `session`) field on pty_open/webrtc_offer selects the container path; absent it, the host shell path is byte-identical to before — so the Infra node terminal is unaffected. Threads PtyTarget through open_pty + rtc handle_offer → build_peer → bridge_pty. Additive; nothing emits `container` yet. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
4de2f31b50 |
Fleet terminal: WebRTC DataChannel direct path (low-latency) + WS fallback
Terminal keystrokes were ~400ms because every byte relayed browser→Cloudflare→
gw-04 (Europe)→tailscale→node, even when the node is on the user's own LAN. Add a
direct browser↔node WebRTC DataChannel so co-located terminals run at LAN speed;
the gateway is reduced to signaling; the WebSocket relay stays as the automatic
fallback (graceful degradation — never worse than before).
Daemon (clawmates-node v0.4.0, new src/rtc.rs):
- Add the `webrtc` crate (reuses the ring crypto provider we already install — no
conflict). Browser is the offerer; we answer, trickle ICE back over the control
channel, and on DataChannel open spawn a host PTY (tmux) bridged DIRECTLY to the
channel. Refactor open_pty → spawn_terminal_pty shared by both transports.
iceServers: STUN + auto host/tailnet candidates (direct, no relay, for LAN/tailnet).
Server (cm-api):
- NodeConn.signal_sinks; Uplink WebRtcAnswer/WebRtcIce/WebRtcFailed routed to the
browser; NodeHub webrtc_offer/ice/close + open_session/open_pty (open_terminal
split so the PTY opens only once the transport is chosen). bridge_terminal relays
signaling over the existing ticket-authed WS and opens the relay PTY on
{type:"fallback"}.
Browser (NodeTerminalApp):
- RTCPeerConnection + reliable/ordered DataChannel; offer/answer/ICE over the WS;
2.5s race → use the DataChannel if it opens, else fall back to the WS relay.
Reconnect wraps both. A direct⚡/relayed indicator shows the live transport.
Deployed; both nodes (morpheus, tank) updated to v0.4.0 and steady online. Direct-
path proof is a browser action (the ⚡ indicator + latency); confirmable from the
daemon's [rtc] logs.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
94828ed887 |
Fleet: robust real-time connectivity + mosh-inspired reconnecting terminal
Nodes flapped online/offline and the terminal died on the first blip. WebSockets are the right transport (outbound, NAT-friendly); the fixes harden around it. Server (cm-api): - Anti-clobber connection epoch: a reconnecting daemon gets a fresh epoch; a stale run_channel's teardown only clears the hub + sets offline if it still owns the slot — so a lingering old channel can't flip a live reconnection offline (the main false-offline cause). - WS keepalive: run_channel now pings every 15s and tears down if no inbound frame (incl. pong) for 35s — dead links detected in seconds, not minutes. - Staleness sweeper backstop: spawn_node_sweeper (8s tick / 20s window) wired in clawmates-server, so a vanished node goes offline within ~28s even if its channel hangs (mark_stale_offline was defined but never called). Daemon (clawmates-node v0.3.0): - Heartbeats off the select thread (dedicated thread owns System + blocking docker/tailscale/disk CLIs) so a slow op never starves heartbeats/pongs. - Each handle_frame runs on its own task; added a 40s inbound idle deadline so a half-open socket triggers a reconnect. Frontend: - useNodes streams /api/nodes/live (SSE push) instead of a 3s poll; isLive() derives online from lastSeen freshness (<15s) so a transient column flip never shows a healthy node down. - Node terminal: clean auto-reconnect loop (re-mint ticket -> reconnect -> tmux re-attaches and redraws the live screen = mosh-style snap-to-state over TCP), replacing the [disconnected] dead-end. Mosh evaluated: harvest principles (session/transport decoupling, snap-to-state, already given by tmux), don't adopt — UDP is incompatible with our browser+CF+NAT topology and it's GPLv3. Removed temporary terminal debug traces + /api/debug route. Verified: node holds steadily online (heartbeat 1-3s, no flap) and goes cleanly offline when the daemon stops. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
95bd022d07 |
Fleet terminal: fix tmux nesting-refusal + frontend resize storm
Root cause (from the server trace: 69 banner bytes then immediate "to_browser
ended" = pty_exit): tmux spawned and exited instantly, writing its "sessions
should be nested with care" warning to stderr (not the PTY). The operator runs
the daemon inside their own tmux, so $TMUX was inherited and the spawned tmux
refused to nest → browser saw nothing.
- daemon (v0.2.2): spawn tmux on a DEDICATED socket (`tmux -L clawmates
new-session -A -s main`) and `env_remove("TMUX")`, so it can never collide with
or be refused by the operator's tmux.
- NodeTerminalApp: debounce the ResizeObserver (150ms). The pull-out animates
open, firing the observer on every pixel — previously ~80 resize frames per
open, each fit()+SIGWINCH. Now one resize after layout settles.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
27f5d05f96 |
Fleet terminal: daemon self-diagnosis + immediate banner + debug route
The server trace showed pty_open is sent but the daemon (morpheus, v0.2.0) emits
no pty_out — so the PTY spawn was dying silently. Instrument it:
- daemon open_pty: log open/tmux/first-read/EOF/error/total to stdout, and send
an IMMEDIATE banner pty_out ("[clawmates] host shell on <host> — starting…") so
the browser confirms the relay even before the shell draws. If open_pty fails,
send the error as pty_out (was a silent pty_exit). Bump to v0.2.1.
- cm-api: temp GET /api/debug/node-pty/{id}?dbg=… opens a node terminal and reads
~2s of output with no browser/auth, to test the relay in isolation.
- NodeTerminalApp + ticket/ws routes already log each hop ([node-term]/[fleet-term]).
Diagnostic logic: banner shows + shell doesn't → relay ok, shell is the problem;
nothing shows → relay broken; daemon "EOF after N bytes" → shell exited.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
8e71c8403b |
clawmates-node: bump to v0.2.0 (selftest + tmux re-hosted)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
c2a0309ad7 |
clawmates-node --selftest + server terminal tracing (diagnose blank terminal)
- Daemon: factor the terminal command into terminal_command(); add `--selftest` which opens the host terminal PTY locally and prints ~2.5s of raw output, so you can confirm tmux/zsh actually draws on a given node without the browser. (Verified locally: tmux spawns zsh + draws its status bar.) - cm-api: temporary [fleet-term] eprintln tracing in open_terminal, the pty_out router, and the browser bridge (byte counts + sink presence) to locate where output stops between daemon→server→browser. To be removed once diagnosed. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
6e81eaa52b |
clawmates-node: host terminal uses tmux (resumable, redraw-on-attach)
The node terminal opened a bare bash PTY — bash prints its prompt once, so a freshly-attached xterm showed nothing. Switch to `tmux new-session -A -s clawmates` on the HOST (mirrors the agent terminal, but on the node itself, not in a container): tmux redraws the whole screen on attach (no blank), and the session is resumable across reopens. Starts in $HOME. Falls back to a login shell if tmux is unavailable. ensure_tmux() best-effort installs tmux via the host package manager when the daemon runs as root (systemd); otherwise logs a hint to `apt install tmux`. Rebuilt + re-hosted both binaries. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
cf6c331b02 |
Fleet: node hostname/IP on register + node terminal in the infra computer
Hostname/IP: - Daemon reports the machine's hostname (sysinfo) + primary outbound IPv4 on each heartbeat. migrations/0021 adds nodes.hostname/local_ip; cm-db heartbeat stores them; node JSON exposes them. Cards now title on the real hostname (falling back to name) + show the IP, instead of the "New node" placeholder. `name` stays user-overridable (rename). Terminal moved into the pull-out computer (no more per-card modal): - New infra computer app NodeTerminalApp (computer/apps/infra) — xterm bridged to a node's host shell over the node control channel, filling the app window (mirrors the agent Terminal's layout + ResizeObserver). Added "terminal" to the INFRA_CATALOG grid; a ?node= panel param targets a specific node (picker when unset). Clicking Terminal on a node card now opens the infra computer to that node's shell instead of a separate full-screen window. Deleted NodeTerminal.tsx. Rebuilt + re-hosted both daemon binaries (hostname change). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
cfa16751c5 |
clawmates-node: fix rustls CryptoProvider panic + non-intrusive Tailscale
Two bugs surfaced running the daemon on a real node:
- Linking cm-sandbox (bollard) brought a second rustls provider into the graph,
so rustls couldn't auto-pick one and panicked at the WSS TLS handshake. Install
the ring provider explicitly at startup (rustls dep + install_default()).
- The daemon auto-ran `tailscale set --ssh`, which tries to reroute the user's
live SSH session and aborts ("will result in your session disconnecting"). Now
Tailscale is only touched when an auth key is explicitly passed (opt-in), with
--accept-risk=lose-ssh to avoid the interactive abort.
Rebuilt + re-hosted both binaries (linux-amd64, darwin-arm64).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
fb59378aa2 |
Fleet P2b: run agent sandboxes on connected nodes (RemoteDriver + placement)
Agents can now provision their sandbox on a connected fleet node instead of the gateway host. Local stays the strict default, so existing agents are byte-for- byte unaffected until explicitly placed elsewhere. Security parity: the daemon links the REAL cm-sandbox DockerDriver and runs the typed container ops (sb_provision/sb_exec/sb_destroy/sb_health/sb_list) through it — identical hardening (cap-drop ALL, seccomp, no-net, read-only, non-root) to local sandboxes. cm-sandbox spec types are now Serialize/Deserialize so the spec crosses the channel. - cm-api: RemoteDriver (impl SandboxDriver over the node channel) + HubDriverProvider (impl cm_runtime::NodeDriverProvider, hands out a driver only for connected nodes via a sync online set) + NodeHub.call/is_connected. AppState.with_node_hub so the hub is shared with the placement provider. - cm-runtime SandboxManager: driver_for(node_id) routes by the recorded agent_containers.node_id (local default = existing driver, identical path); placement_node() reads the workspace setting and falls back to local if the node is offline; exec/release route accordingly. NodeDriverProvider trait. - DB: 0020_workspace_placement + repo (for_agent/get/set/clear). - main.rs: build the NodeHub first; inject HubDriverProvider into the agent manager + share the hub with AppState. - API+UI: GET/PUT /api/fleet/placement + a "Run agents on: Local / <node>" selector in the Fleet overview. Note: a node must be able to pull the agent image (the daemon docker-pulls it); interactive PTY for agent containers on remote nodes is not wired (Terminal app stays local) — the in-dashboard node shell already covers host access. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
33aa9c0693 |
Fleet P2b: node sandbox-readiness check (hardened workload on a node)
Proves a connected node can host hardened agent workloads end-to-end, without
touching the agent run loop (zero blast radius on existing agents).
- Daemon: typed `sb_check` op — pulls a tiny image and runs it fully locked down
(cap-drop ALL, no-new-privileges, no network, read-only rootfs, non-root,
memory/pids caps), then tears it down. Fixed command; nothing caller-supplied
runs (preserves the exec-hardening invariant).
- cm-api: NodeHub.sandbox_check + POST /api/nodes/{id}/sandbox-check.
- UI: a shield "sandbox check" button on each online node card streams the
result (✓ SANDBOX READY + container id/uname).
This validates the full provision→run→destroy mechanism on nodes. The remaining
P2 work — wiring real agent deploys to auto-place onto nodes — is its own
subsystem (a RemoteDriver reusing the local DockerDriver for security parity,
agent-image distribution to nodes, and node-routing in SandboxManager) and is
best done as a focused pass; it is intentionally NOT bundled here to keep the
core agent path untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
f5f96508eb |
Fleet P2a: in-dashboard remote terminal (PTY over the WSS channel)
You can now open a real shell on any connected node from the dashboard — the
daemon spawns a host PTY and streams it over the existing outbound control
channel (no inbound port, no Tailscale brokering needed).
Daemon:
- portable-pty host shell sessions: pty_open/pty_in/pty_resize/pty_close ops; a
reader thread streams base64 pty_out frames. Outbound frames now funnel through
one mpsc channel so PTY output and heartbeats interleave.
cm-api NodeHub:
- per-connection pty_sinks + sid multiplexing; open_terminal/terminal_input/
terminal_resize/terminal_close; in-memory single-use terminal tickets (the
browser WS can't carry a bearer, and the session is instance-local anyway).
- routes/nodes.rs: POST /api/nodes/{id}/terminal/ticket + GET .../terminal/ws
(bridges browser xterm <-> node PTY: binary = keystrokes, text = resize).
Frontend:
- NodeTerminal xterm modal (reuses the agent Terminal's xterm setup); a Terminal
button on each online node card opens a shell.
This proves the bidirectional streaming-over-channel mechanism the RemoteDriver
will reuse. Remaining P2: RemoteDriver + placement (run agent workloads on nodes).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
7332d69f8a |
Fleet P1: BYO Tailscale + network metrics, Tailscale SSH, exec hardening
Security hardening: - The gateway no longer sends arbitrary shell to nodes. The WSS exec op is replaced by a typed `verify` op the daemon runs itself (fixed host+docker check); future container ops are typed too. cm-api NodeHub.verify() + the daemon's handle_command only dispatches vetted ops. BYO Tailscale: - migrations/0019_workspace_tailscale.sql + cm-db fleet_tailscale repo (store the user's Tailscale API key + tailnet, server-side only). - cm-api routes/tailscale.rs: POST/GET/DELETE /api/fleet/tailscale + GET /api/fleet/tailscale/devices (proxies api.tailscale.com device list). - Daemon: --tailscale-authkey → `tailscale up --authkey … --ssh` (enables Tailscale SSH for keyless user access); else `tailscale set --ssh=true`. Reports its tailscale IP (already). UI: - Fleet overview gains a Tailscale section: connect (key+tailnet) + live tailnet device status (online/last-seen/IP/os). Node cards show a copyable Tailscale SSH target (ssh <ip>). Remaining: P2 — RemoteDriver + placement (run agents on nodes) and the in-UI remote terminal (PTY proxied over the WSS channel). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
2bdd0a23e8 |
Fleet P0: node registry + daemon + health + connect-host wizard
Users can connect their own local-hardware nodes into a fleet. Each node runs a
new Rust daemon that dials home over an outbound WebSocket, reports host health,
and runs commands we send.
Backend:
- migrations/0018_fleet_nodes.sql: nodes + node_health tables + agent_containers
(node_id, workspace_id) index. cm-domain NodeId.
- cm-db repo/nodes.rs: create/auth/list+health/get/heartbeat/set_status/delete
(unchecked sqlx, no .sqlx regen).
- cm-api fleet.rs NodeHub: live daemon channels (node_id→sender) + the WS channel
runner (heartbeat→DB upsert, exec request/response framing). routes/nodes.rs:
POST /pair, GET /nodes, SSE /nodes/live, POST /{id}/exec-test, DELETE /{id},
WS /nodes/agent (token-auth). Wired into AppState + router.
Daemon (new crate crates/bins/clawmates-node):
- sysinfo host metrics (cpu/mem/pressure/swap/disk/load/containers), outbound WSS
dial + reconnect, heartbeat loop, exec command handling, tailscale-ip probe.
install.sh convenience installer.
Frontend:
- Fleet sidebar item + FleetOverview + LocalHardware node-health cards (live via
/api/nodes, 3s poll) + ConnectHostWizard (install → verify connection →
exec-test). InfraStage dispatches fleet/local; default selection = fleet.
Deferred: P1 (BYO Tailscale + network metrics), P2 (RemoteDriver + placement so
agents actually run on connected nodes).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|