The python guest agent only ever worked because Firecracker's CI Ubuntu
image happens to ship python3. NONE of our images do — agent-base has
neither python nor git, agent-terminal has git but no python — so it
could never have run in a real mission rootfs. An agent that dictates
what must be installed in the image has the dependency backwards.
crates/bins/fcagent is a 905K static x86_64-unknown-linux-musl binary
that needs nothing from the rootfs it is dropped into. The wire is
unchanged on purpose — 4-byte BE length + JSON, ops ping/exec/put/get —
so microvm.rs and microvm_client.rs needed no edit at all.
std has no AF_VSOCK and the workspace denies `unsafe`, so it uses the
`vsock` crate. `process_group(0)` gives each command its own group without
unsafe, so a command that spawns background children can be killed
wholesale rather than outliving the run.
A unit test caught a bug that would have broken EVERY exec: sourcing the
image-env file with `. env.sh 2>/dev/null; cmd` returns rc=1 WITHOUT
running cmd, because `.` on a missing file makes a non-interactive POSIX
shell exit immediately. On any rootfs lacking that file every command
would have failed while looking like an ordinary non-zero exit. Guarded
with `if [ -f ]` now.
Other places a failure must not borrow an outcome's representation: a
killed command reports ok:false with no rc (not rc=124, which would read
as a build failure); `get` on a missing path is an error, not an empty
archive; a signalled process reports 128+signal rather than success.
Verified on tank: --vm-selftest still 8/8 with the agent swapped
(create 949ms, wire identical), fc-node-setup 8/8, and — the point of the
change — a rootfs built from clawmates/agent-terminal:dev, which has NO
python3, boots and reports `git version 2.39.5` from inside the VM.
Also fixes a shell bug in fc-build-rootfs.sh: $HOME in a double-quoted
default expanded on this Mac, so it looked for the node's binary under
/Users/quantum on a Linux host.
Co-Authored-By: Claude Opus 5 <[email protected]>
create / inject / exec / collect / destroy / list, riding the node's
existing frame dispatch ({t, id, …} -> {t:"result", id, ok, output}), so
no protocol change was needed. Control is length-prefixed JSON over
vsock; the serial console stays a log, because feeding a guest over stdin
races its startup and arrives half-consumed.
DEVIATION FROM THE PLAN, deliberately: this does NOT implement
cm_sandbox::SandboxDriver. That trait is container-shaped —
attach_pty/resize_pty/argv exec — while missions need
create -> inject -> run -> collect -> destroy. Conforming would mean
building PTY-over-vsock and window-resize semantics that no mission path
calls, purely to satisfy a signature. We give up automatic RemoteDriver
marshalling; orphan reaping is a label/id sweep either way.
Three traps from the B0 spike are handled in code rather than remembered:
- Firecracker does NOT unlink its vsock UDS on exit, so destroy unlinks
it explicitly, and the selftest ASSERTS it is gone. Assuming the VM
tidies up after itself is how the mission checkout accumulated four
uid bugs.
- firecracker is spawned via setsid and killed as a process GROUP, so a
background child cannot outlive the VM holding its workdir open.
- create does not return until the guest agent has answered a ping. A
VM that booted but serves nothing is worse than one that failed, so a
half-created VM is destroyed rather than left registered.
A vm id becomes a path component, so ids are restricted to [A-Za-z0-9_-]
and REJECTED rather than sanitised — a caller that sent `../../etc`
wanted something we should not guess at.
Verified on tank through the real Rust path, as the daemon user, with no
sudo: `clawmates-node --vm-selftest` -> 8/8, create in 986ms, and the
host left with zero firecracker processes and zero VM directories. The
selftest asserts every step, including that a destroyed VM can no longer
be exec'd; a test that only reports the steps it completed cannot
distinguish "passed" from "stopped early".
Co-Authored-By: Claude Opus 5 <[email protected]>
The capability probe reported `kvm: false` on tank and morpheus while the
device sat right there: /dev/kvm is `crw-rw---- root:kvm` and the kvm
group was EMPTY, so the daemon — an ordinary user — could not open it.
The B0 spike missed this entirely because it ran everything under sudo.
This is exactly why the probe opens the device rather than stat-ing it;
a stat-based check would have reported both nodes capable and every
microvm mission would have failed at launch instead of at placement.
fc-node-setup.sh now fixes the group itself, or says precisely what to
run when it cannot.
Co-Authored-By: Claude Opus 5 <[email protected]>
Phase B step 0. Before writing any driver, establish that Firecracker
works on this hardware — the plan called it greenfield, and an
orchestrator built against an unproven runtime is a lot of code betting
on an assumption.
It works, and comfortably: a microVM boots, runs our init, writes a
file and shuts down in ~650-910ms wall clock, with the kernel reaching
our init at 234ms. Host->guest RPC over vsock (AF_VSOCK port 9001, no
network stack) round-trips in 27ms.
The script installs and then PROVES, because installing is not working.
It reports success only after a VM has actually booted and run our code.
Four findings from the spike that the driver must account for:
- Firecracker does NOT unlink its vsock UDS on exit, and leaves it
owned by whoever ran the VM. A driver running as anyone else cannot
clean it up — the same uid trap that cost this codebase four bugs on
the mission checkout. The driver owns the socket path lifecycle.
- tank's FORWARD policy is DROP (Tailscale/Docker), confirming the
article's warning: VM networking rules must be inserted at position
1, not appended, or return traffic dies silently.
- Feeding commands to the guest over the serial console races the
shell's startup and arrives half-consumed (`# ho FC-GUEST-ALIVE`).
The guest runs an init script; stdin is not a control channel.
- `sha256sum -c` compares by filename, so a download saved under any
other name fails for a reason unrelated to integrity. A check that
fails for the wrong reason teaches you to ignore it — compare the
hashes directly.
tank and morpheus are ready. architect requires interactive sudo, so it
is deliberately not provisioned rather than worked around.
Co-Authored-By: Claude Opus 5 <[email protected]>
