Author SHA1 Message Date
Omar SobhandClaude Opus 5 7e07c389c6 feat(missions): wire copy-in/copy-out behind CLAWMATES_MISSION_FS=copy
With the flag set, ensure_container omits the /mission bind, the checkout
is pushed into the container at phase launch, and the agent's work is
pulled back before capture.

The simplification that makes this small: sync_out unpacks over the SAME
host path the checkout came from. The host directory stays a server-owned
staging area with exactly one writer, and capture_phase_diff_at needs no
change at all — it still finds a normal checkout exactly where it always
has. Delivery, gating, commit and push are untouched.

Two failures are deliberately loud rather than silent:

- copy-IN failure fails the phase launch. Continuing would start a phase
  against an empty directory, and the agent would cheerfully report having
  done work on a repo that was not there.
- copy-OUT failure SKIPS capture. Capturing anyway would diff a stale host
  tree and record "no changes" for work that exists — success reported for
  nothing, which is the exact failure mode this codebase keeps paying for.

Opt-in: the bind path is what production has run since the beginning, and
the test asserts a near-miss value leaves it there rather than silently
switching every mission.

414 tests, clippy clean.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-04 15:41:47 -07:00
Omar SobhandClaude Opus 5 389b41f8e6 feat(missions): copy-in/copy-out primitive for the mission checkout
The first half of removing the shared bind mount. Not wired yet — this
adds the mechanism and its tests.

One cause, four fixes so far: .git/objects permission denied
(core.sharedRepository), the capture base being overwritten each phase,
COMMIT_EDITMSG root-owned, and reset --hard deleting a prior phase's work
(.git/clawmates-in-use). core.sharedRepository was never a general
solution — it covers objects and refs, and every OTHER file git touches
is a fresh opportunity. Copy-in/copy-out removes the cause instead: the
agent owns its filesystem with no second writer.

Measured before building, because the plan named copy cost as the open
risk: a real 65 MB checkout of this repo copies in 0.23s and out 0.18s on
gw-04. Not a risk at this size; re-measure an order of magnitude larger.
No compression — the payload crosses a local socket, so gzip would spend
CPU to save nothing.

Two safety properties, both tested:

- The archive comes back from a container the agent controls as ROOT, so
  it is untrusted input. A `../ESCAPED` entry must not write outside the
  destination. The test writes the tar header bytes by hand because the
  tar crate refuses to BUILD such an entry through its safe API — which
  is reassuring, but means the hostile case has to be constructed the way
  an attacker would.
- Symlinks are packed as links, never dereferenced. Following them on
  copy-IN would smuggle host files into the container; the test plants a
  host secret behind a symlink and asserts its contents never appear in
  the archive.

Ownership is deliberately not preserved on unpack: the archive's uids are
the container's root, and re-applying them on the host would recreate the
exact uid split this exists to remove.

413 tests, clippy clean.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-04 15:15:22 -07:00
Omar Sobh ac6bf72943 Merge: per-mission runtime data (stop sharing the door token)
ci / gates (push) Failing after 15s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
2026-08-04 12:30:31 -07:00
Omar SobhandClaude Opus 5 0d9498ec6e fix(missions): copy an allow-list, not the whole 1.7GB seed dir
Checking before deploying caught a mistake in the previous commit. The
seed dir on gw-04 is 1.7 GB and the first version copied all of it per
mission — tens of seconds each, and ~17 GB across ten concurrent
missions.

1.5 GB of that is .rustup: a Rust toolchain that installed itself into
the data dir back when HOME=/zeroclaw-data and the image had no
toolchain. The image now ships Rust at /usr/local/cargo, which is what
the container's PATH actually resolves — verified live. The data-dir copy
is dead weight and is not even reachable.

SEEDED_PATHS now copies only what carries per-mission identity or
secrets: .zeroclaw (config.toml with the door token, sessions.db,
devices.db), clawmates-mcp.json, .claude + .claude.json, .kimi-code,
glm-home, agents. Roughly 46 MB instead of 1.7 GB — about 37x smaller.

Caches and toolchains are deliberately excluded: .rustup, .npm, .cargo,
.cache, .local. They hold no secrets and a mission reads the image's.

Absent paths are tolerated: a fresh deployment has no .kimi-code until
Kimi is first used, and that must not fail container creation.

The test asserts both directions — the token-bearing paths ARE copied
and the caches are NOT — because either mistake is silent: copying
everything just makes missions slow, and copying nothing quietly
restores the credential sharing.

409 tests, clippy clean.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-04 12:26:04 -07:00
Omar SobhandClaude Opus 5 15e7608e4a fix(missions): give each mission its own runtime data
Every per-mission container bind-mounted the SAME host seed dir as
/zeroclaw-data — shared with each other AND with the singleton runtime.
That directory holds config.toml, which carries the §15 door bearer
token, plus sessions.db and devices.db.

So one mission could read another mission's credential, and anything it
wrote there was inherited by every later mission. teardown_container
only removes /var/lib/clawmates-missions/{id}, so the shared directory
was never cleaned — the contamination was permanent.

The code already knew. The comment on DEFAULT_SEED_DIR names the sqlite
race and calls copy-on-write per mission the long-term fix. This is that
fix: seed_runtime_data copies the seed into
<missions_root>/<mission>/runtime-data at container create, and the
mount points there. Cleanup is free — teardown already removes that tree.

The copy runs in a throwaway container because cm-api cannot see the seed
dir: it hands that host path to Docker but never mounts it itself. The
runtime image is reused so nothing extra is pulled, and `cp -a /seed/.`
copies dotfiles — `/seed/*` would silently skip .zeroclaw/ and produce a
runtime with no config at all.

A copy failure is fatal to container creation on purpose. Falling back to
the shared mount would silently restore the credential sharing this
removes, and silent fallback to a weaker posture is the failure mode this
codebase keeps paying for.

The test asserts path shape rather than behaviour: an edit that points
the mount back at the seed dir restores credential sharing with no other
visible symptom, so the path IS the invariant.

408 tests, clippy clean.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-04 12:14:38 -07:00
Omar SobhandClaude Opus 5 5d98fcf44a feat(missions): forward ZAI/KIMI keys so one binary serves three backends
ci / gates (push) Failing after 7s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
All three providers run through the SAME `claude` binary, verified live:

  Anthropic  CLAUDE_CODE_OAUTH_TOKEN                          -> ANTHROPIC-OK
  GLM        ANTHROPIC_BASE_URL=https://api.z.ai/api/anthropic -> GLM-OK
  Kimi       ANTHROPIC_BASE_URL=https://api.kimi.com/coding/   -> KIMI-OK

That is a stronger multi-provider story than a provider-per-implementation:
skills, subagents, MCP, hooks and tool policy are identical across all
three because it is literally the same harness.

The `kimi` CLI (0.31.1, shipped in the image) 401s on this key and is not
needed -- the claude binary reaches Kimi's Anthropic-compatible endpoint
directly. Worth knowing before someone debugs the CLI.

forwarded_provider_keys now ships ZAI_API_KEY and KIMI_API_KEY into
mission containers in BOTH auth modes: they are unrelated to the Anthropic
credential, so the api_key/subscription split does not apply to them. A
mission that selects a backend without its key present would otherwise
fail at the first turn.

Keys persisted in /opt/clawmates/.env and passed through compose.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-04 10:51:34 -07:00
Omar SobhandClaude Opus 5 4ff4e6f7ee fix(missions): a root-owned COMMIT_EDITMSG must not block delivery
ci / gates (push) Failing after 18s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Mission 019fcd0c produced correct work — a reviewed, tested function plus
a REVIEW.md quoting a real cargo test summary — and delivered none of it:

  git commit → exit 128: could not open '.git/COMMIT_EDITMSG': Permission denied

The agent ran `git commit` itself inside the mission container (as root),
leaving that file owned by root at 0644. core.sharedRepository covers
objects and refs — .git/index lands at 0666, which is why commits work at
all — but not COMMIT_EDITMSG, which git writes with the default umask.

Unlinking works where overwriting does not: removing a file needs write
permission on the DIRECTORY, and .git/ is owned by the server. Silent on
failure by design, so the commit reports the real error rather than this
speculative cleanup.

Third distinct instance of the same uid-split class (objects, then the
capture base, now this). The pattern holds: the checkout is one directory
written by two users, and each new file git touches is a new opportunity.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-04 07:35:37 -07:00
Omar Sobh deb60be98d Merge: direct session executor for missions (flag-gated)
ci / gates (push) Failing after 6s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
2026-08-03 22:56:30 -07:00
Omar SobhandClaude Opus 5 758b2dbd96 feat(missions): run a phase as one direct session, behind a flag
CLAWMATES_MISSION_EXECUTOR=session makes launch_phase run the whole phase
as a single `claude -p` against /mission/repo instead of driving turns
through ZeroClaw. Opt-in, because silently changing how every mission
executes is exactly the sort of change that should require someone to
have typed it.

It still writes ONE topology_runs row. The entire downstream lifecycle --
close_finished_phases, evaluation, capture, commit, gate, publish -- keys
off those rows, and inventing a second completion path would mean two ways
for a phase to finish with one of them untested. The session is simply a
run with tier='session' and an empty graph.

Spawned rather than awaited: launch_phase runs inside the sweep loop, and
blocking it for the length of a coding session would stall every other
mission.

The session's own summary is logged as diagnostics only. Whether the phase
actually did anything is still decided downstream by capture and delivery
against the repository -- a 0-exit session that pushed nothing was measured
at ~5%, so the agent's account can never be the verdict.

406 tests, clippy clean.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-03 22:56:22 -07:00
Omar SobhandClaude Opus 5 37fac288d2 feat(missions): bring the direct session executor onto a live branch
Rescues session_executor from the stranded spike branch. Multi-provider
missions are not needed for now, so the direct path is worth nailing down:
run a mission as one `claude -p` session against its checkout instead of
routing turns through ZeroClaw.

Measured today against a real checkout in the runtime container, using the
executor's exact argv:

  direct `claude -p`   7s, file written
  via ZeroClaw         minutes per turn, and THREE config failures before
                       it worked at all (no credential in the mission
                       container; Write/Edit denied; no tools granted --
                       the last of which COMPLETED a mission having
                       written nothing)

Each of those failures came from the same root: with claude_cli, ZeroClaw
is a WebSocket-to-subprocess adapter whose own controls (risk profiles,
tool gating, memory) do not reach the subprocess. The adapter adds
failure modes without adding governance.

What ZeroClaw still earns for the rest of the platform is unchanged and
not in question here: interactive chat, the brain, A2A and door identity,
terminal, agent routines, and non-Claude providers.

Not yet wired into phase_runner — this commit only makes the executor
reachable and keeps it building. SessionOutcome::delivered() still
requires a clean exit AND an observed branch, because a 0-exit session
that pushed nothing was measured at ~5%.

405 tests, clippy clean.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-03 21:26:48 -07:00
Omar SobhandClaude Opus 5 5232175c88 fix(missions): forward the subscription token into mission containers
ci / gates (push) Failing after 9s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Switching agents to claude_cli left missions hanging: the per-mission
container had claude_cli configured but no credential, so `claude -p`
waited forever. A phase sat at `running` for ten minutes with nothing in
the logs — no error, because there is nothing to error on.

The original subscription design assumed a persisted `claude /login`
under a bind-mounted $HOME. That holds for the shared runtime and NOT for
a mission container, which gets its own data dir and therefore no login.
So subscription mode now forwards CLAUDE_CODE_OAUTH_TOKEN.

The two Anthropic credentials remain mutually exclusive, and there is now
a test asserting it in both directions: Claude Code ranks ANTHROPIC_API_KEY
above the OAuth token, so shipping both bills the API while the deployment
believes it is on the subscription — visible only on the invoice.

Deployment: CLAWMATES_RUNTIME_AUTH=subscription and CLAUDE_CODE_OAUTH_TOKEN
added to compose + .env on gw-04.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-03 17:34:13 -07:00
Omar SobhandClaude Opus 5 ac47dcbe94 feat(runtime): run agents on the subscription via the real claude binary
ci / gates (push) Failing after 5s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
provider_alias_for now resolves Claude models to `claude_cli.default`,
which spawns the actual `claude` binary, instead of `anthropic.default`,
which posts to the raw API with Claude Code identity headers. Agent work
is ~99% of tokens, so this moves essentially all of it onto the Max
subscription and onto the supported client.

The judge deliberately stays on the API key. If both rode one credential,
a single subscription limit would blind the verifier at exactly the
moment there is most to verify; this way a throttle degrades missions but
verification keeps working.

Runtime config (applied on gw-04, reloaded via loopback — remote admin
reload is disabled by design):
  - [providers.models.claude_cli.default] with mcp_config pointing at the
    §15 door, so a subscription agent can ACT and not merely reason
  - disallowed_tools denies Claude Code's own Bash/Write/Edit/WebFetch so
    the gated door is the ONLY actuator and nothing bypasses the audit log
  - env CLAUDE_CODE_OAUTH_TOKEN = "$CLAUDE_CODE_OAUTH_TOKEN" — the $NAME
    form reads the daemon env, keeping the token out of config.toml
  - anthropic.judge swapped to the API key (0 oat01 left in config)

Verified before changing anything: the real binary returned SUBSCRIPTION-OK
through the token, then ENV-OK once the daemon carried it in env.

Note for future readers: /api/config/prop reflects what is CONFIGURED, not
what the binary supports — `openai` 404s there too. An earlier note that
the image "has no claude_cli in its schema" was true of the old :sync
image and is not true of the rebuilt one.

400 tests, clippy clean.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-03 17:14:42 -07:00
Omar SobhandClaude Opus 5 ad89ef94cd feat(library): attribute a run to a mission, and prove what it contributed
ci / gates (push) Failing after 7s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
`corpus_items.mission_id` has existed since the table landed and nothing
could populate it. `POST /api/library/runs` now accepts `missionId`, which
is the seam the wizard needs: a mission-driven run is the same run, tagged.

`corpus::contributed()` answers the question a continuous mission has to
be able to answer — did THIS run add anything new. Because `record` never
reassigns mission_id on conflict, the mission that first found a source
keeps the credit, so a rerun cannot inflate its own count by re-recording
what an earlier run already held. The test asserts exactly that: two
missions see the same paper, the finder reports 1 and the rerun reports 0.

This is the check the 0030-0044 generation of continuous research did not
have. It could run weekly forever and every run looked like success.

The test also earned its FK: the first version attributed to a bare UUID
and the database refused it. Attribution to a mission that does not exist
is not attribution, so the test now seeds real mission rows.

400 tests, clippy clean.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-03 11:43:41 -07:00
Omar SobhandClaude Opus 5 9de2cf34e4 feat(auto-merge): merge additive branches, refuse everything else
ci / gates (push) Failing after 10s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Closes the branch pile-up: a catalogue branch that only adds notes now
merges into main by itself, so the work is actually in the vault rather
than waiting in a branch nobody opened.

Additive-only is measured from the diff, not assumed from the mission
type. Three conditions, all required: the type declares additive_only,
the run verified, and `git diff --name-status base...branch` contains
only A entries. A research harvest that somehow rewrote a hand-written
note is refused by the same check that lets its new notes through —
which is the case the test pins down, asserting README.md on main is
byte-identical afterwards.

Renames and deletes count as non-additive. A rename is a delete plus an
add and the delete half can destroy hand-written work.

Unknown merge_policy values fail closed to Never. A typo must not grant
auto-merge.

The diff is taken against FETCH_HEAD, freshly fetched, using `...` so an
unrelated commit landing on main meanwhile is not misread as ours. A
conflicted merge aborts and leaves the branch for a human rather than
wedging the checkout for the next run.

merge_reason is always populated and surfaced in the API: a branch that
quietly did not merge is indistinguishable from one never delivered.

399 tests, clippy clean.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-03 11:33:41 -07:00
Omar SobhandClaude Opus 5 3124fd3c8f feat(library): weekly harvest on a systemd timer
ci / gates (push) Failing after 7s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Monday 07:00, Persistent=true so a week missed to downtime fires on next
boot rather than leaving a silently empty library. 30-minute timeout so a
wedged run cannot hold the slot until the following week.

The script is deliberately thin — it calls the API and reports — so it
never needs changing when the harvest does. Auth is a long-lived operator
session in /etc/clawmates/library.token (root-only, 600); rotate by
replacing the file.

Exit status follows `healthy`, not paper count. A mature library shelves
nothing most weeks and that is success; a run that errored is a failure
even if it shelved something.

The first manual fire caught a real bug in this script, in the opposite
direction to this week's usual: the harvest genuinely shelved 15 papers
and pushed them, and the reporter crashed on an escaped quote inside an
f-string, so systemd marked the unit FAILED. A false failure destroys
trust in the signal exactly as a false success does. The reporter now
avoids backslashes entirely (it is embedded in a single-quoted shell
string) and was proved against the real response shape before being
trusted.

Verified end to end on gw-04:
  run 1: 25 candidates, 10 already held, 15 shelved, pushed
  run 2: 25 candidates, 25 already held,  0 shelved, no branch, healthy

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-03 10:59:13 -07:00
Omar Sobh cf076bd8ea Merge: paper library — corpus, arXiv harvest, vault catalogue, API
ci / gates (push) Failing after 7s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
2026-08-03 10:53:20 -07:00
Omar SobhandClaude Opus 5 107f0dbced feat(library): expose the library over the API
POST /api/library/runs harvests now; GET /api/library/items lists what
the library holds. Thin wrappers — the work stays in crate::library — so
a run can be started by a person, a schedule or the UI rather than only
from an integration test.

The response reports `healthy` explicitly rather than leaving a caller to
infer it from an empty `shelved` list. A quiet week and a broken run both
shelve zero papers, and collapsing those two is the exact ambiguity that
cost most of this week.

Failure reasons go to the log, not the response body: they can carry the
remote URL and raw git stderr.

AppState gains an optional blob store (the shelf), wired from the server
binary where storage is already constructed. Optional because AppState::new
is used by tests that never touch blobs; a route that needs it fails
loudly rather than the constructor demanding it everywhere.

393 tests, clippy clean.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-03 10:21:00 -07:00
Omar SobhandClaude Opus 5 09c6496725 feat(library): clone the vault, harvest our topics, push the catalogue
Completes the loop: the notes now land in the real vault. Topics come
from what the project is actually working on — papers/dynamic-agentic-
topologies.md (topology search, ADAS/Darwin-Godel/SwarmAgentic) plus the
two problems this week ran into, verifying what an agent did and giving
a long-running agent memory of what it covered.

Never pushes to main. The vault is a live Obsidian vault a human edits
and syncs; pushing to main races that sync and can lose hand-written
work. Every run lands on its own branch for a human to merge, the same
rule the mission delivery path was validated 20/20 under.

PDFs are NOT committed. A few hundred papers is gigabytes and would make
the vault painful to clone and slow to open, so they stay on the blob
store shelf and the note carries the key.

My own test caught me repeating this week's branch-collision bug: I named
branches from the HEAD of a UUIDv7, which is a 48-bit timestamp, so two
runs in the same millisecond produce the identical name — exactly what
hit mission 019fc42b. Fixed by taking the tail. The test now loops 100
ids instead of sampling two (a one-shot check passes by luck whenever the
millisecond ticks between calls) and additionally asserts the head-based
scheme DOES collide, so it cannot rot into a no-op.

Live against the real vault:
  10 candidates, 1 already held, 9 shelved, 0 failed
  branch clawmates/library-019fc82292e8, pushed
  9 notes verified on the forge, 9 PDFs verified %PDF on the shelf
  (the "1 already held" is cross-topic dedupe inside a single run)

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-03 08:03:14 -07:00
Omar SobhandClaude Opus 5 30eaa50c50 feat(harvest): one run — find, skip what we hold, shelve the rest
Turns the parts into a job. Order is the point: the checkmark list is
consulted BEFORE anything downloads. Checking afterwards would still
dedupe the catalogue while re-downloading every paper we already have,
every week, forever.

Two properties the tests pin down, both learned the hard way this week:

- A quiet week is not a failure. `shelved == 0` with no errors is a
  healthy run against a mature library; `shelved == 0` with errors is
  broken. Harvest::healthy() and ::added_anything() keep those apart
  rather than collapsing them into one ambiguous "did nothing".
- A failed download leaves the paper UNSEEN. Checking it off before the
  PDF is safely shelved would mean one transient network error retires
  that paper permanently. The checkmark is written last, after the bytes
  and the note are both on disk.

The skip test gives every candidate a pdf_url pointing at a closed port,
so if the skip ever regresses the test fails loudly instead of quietly
re-fetching.

Live end-to-end against arXiv, run twice:
  RUN1  3 candidates, 0 already held, 3 shelved, 0 failed
  RUN2  3 candidates, 3 already held, 0 shelved, 0 failed

Library<'_> groups the five values that always describe one library;
passing them loose is how a run shelves into one place and catalogues
into another (also silences clippy::too_many_arguments honestly rather
than by allow).

391 tests, clippy clean.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-03 07:53:56 -07:00
Omar SobhandClaude Opus 5 e4a395b72e feat(papers): find papers on arXiv, shelve the PDF, catalogue the note
Corrects a misread of the design. I had built this as "read the vault to
find papers"; the vault is the CARD CATALOGUE, not the source. Papers are
found on arXiv, the PDF is pulled down and shelved in our own library,
and a note recording it goes in the vault.

Three parts, and which is which matters:
  arXiv       — where papers are found
  blob store  — the shelf; the PDF lives there (cm-files, local + S3)
  the vault   — the catalogue; one note per paper, pointing at the shelf

The checkmark list (corpus, 0064) is what makes this continuous rather
than a job that redoes itself every week — the failure that killed the
previous attempt (0030-0044, dropped in 0053).

The load-bearing detail: every catalogue note carries
`source_id: arxiv:NNNN.NNNNN` in frontmatter, which is exactly the key
corpus::parse_note reads. So the checkmark list is rebuildable FROM the
vault. If the database were lost, re-indexing restores what we have —
the catalogue is authoritative, the index is derived. A test asserts that
round trip rather than trusting the two halves to agree.

Version suffixes are stripped (2401.12345v3 -> 2401.12345) or a weekly
job re-downloads a paper every time authors post a revision. Fetches are
rejected unless the bytes start with %PDF: arXiv serves an HTML holding
page while a PDF renders, and shelving that leaves a file that looks
present and is unreadable.

Verified against live arXiv, not fixtures:
  arxiv:2607.29678 TokTier: Exact Stateful Tokenization for Agentic LLM…
  arxiv:2607.29677 ExtractBench: A Benchmark for Schema-Guided Enterpri…
  arxiv:2607.29658 Reusing Past Repairs Through Hierarchical Trajectory…
  pdf: 1,361,770 bytes, %PDF verified

388 tests, clippy clean.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-03 07:38:07 -07:00
Omar SobhandClaude Opus 5 6e5ccc25a6 feat(corpus): record what a continuous mission has already covered
Slice 2 of the adopt-or-build plan. A recurring mission's hard problem is
not running the agent — that is 23 seconds — it is knowing what it did
last time. This repository already tried continuous research once:
migrations 0030-0044 built research_topics/loops, 0053 dropped them all,
and the reason they could not survive is that research_topics carried a
status lifecycle but no seen-set. It could run forever and never know
what it had covered.

Two kinds of row, because the real vault forced it. The plan assumed
notes carry arxiv:/doi:/url: frontmatter. Measured against the actual
valhalla-vault: 416 notes, 145 with frontmatter, and ZERO with any of
those keys — the dominant keys are repo-sync metadata (node, org, gitea)
and course fields (presenter, session). An ingester keyed only on
external identity would have indexed nothing, which is the same shape of
failure as everything else found this week. So `note` rows record
coverage (keyed by path) and `source` rows record consumption (keyed by
natural id); a continuous mission needs both.

Two decisions the data forced:

- `source:` is deliberately NOT an identity key. The vault uses it for
  local paths of course material (/Users/quantum/Downloads/...), which is
  provenance, not citable identity. Accepting it would fill the seen-set
  with 25 rows keyed on a laptop path.
- The hash covers the body, not the whole file. Repo-sync notes rewrite
  updated:/size_kb: on every sync without the prose changing; hashing the
  file would report 103 phantom edits per run and make "unchanged"
  meaningless.

Authoritative in Postgres rather than ZeroClaw memory, per the Slice 1
spike: memory is agent-scoped and mission agents are ephemeral
claw_<uuid> aliases (~100 already present). A seen-set that disappears
with the agent that wrote it is not a seen-set. The spike did find that
POST /api/memory upserts by key, so mirroring content there later would
inherit idempotence for free if keyed by source_id.

Verified against the live 416-note vault, not a fixture:
  PASS1 { scanned: 416, inserted: 416, updated: 0, unchanged: 0 }
  PASS2 { scanned: 416, inserted: 0,   updated: 0, unchanged: 416 }

382 tests, clippy clean.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-03 07:00:21 -07:00
Omar SobhandClaude Opus 5 2380c2cb0b fix(deploy): identify agent images by build stamp, not image ID
ci / gates (push) Failing after 6s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
The post-transfer verification added in bb34ef1 failed every deploy: it
compared `.Id` between build host and target, and a BuildKit image on the
build host carries attestation manifests that `docker save | docker load`
does not reproduce. The same build legitimately arrives with a different
Id and a different reported Size — tank had agent-base:dev at 28 MB /
363f23b7, gw-04 at 74 MB / edd46f95, both from the identical build.

`.Created` comes from the config blob, survives the round trip unchanged,
and is what actually answers "is the new build here". Both hosts reported
2026-08-02T16:43:00.599937575-07:00, which is how the false positive was
identified rather than assumed.

The verification itself stays — the truncation it guards against is real.
This corrects what it compares.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 19:02:08 -07:00
Omar SobhandClaude Opus 5 ec85f6c8da fix(missions): close the three seams behind this run of failures
ci / gates (push) Failing after 6s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Seam 1 — delivery inferred checkout state from the tree, so whether work
survived depended on what the agent happened to do. 019fc444 committed
and left a clean tree; 019fc476 had its base advanced to match HEAD;
019fc450 survived only because a phase FAILED to commit and left the tree
dirty. Same code, opposite outcomes, decided by the agent.

mark_phase_started records the fact at phase launch, before the agent
acts, so every one of those states answers identically. The tree checks
remain as a second line of defence for pre-existing checkouts.

Seam 2 — phase config was accepted, stored and read by nobody. That was
`task`: every phase of every mission got identical instructions. The new
phase_config registry names the reader for each live key and lists the
eight that are declared-but-unimplemented, reporting both at mission
creation so an author sees what will not happen. Its CI test found one I
had missed: security_hardening.toml sets phase-level mcp_bundles asking
for gitea_forge + security_scan, but bundles come from the TEAM template
and the phase gets neither.

Seam 4 — push_url_for collapsed a failed query, an unbound repo and a
missing clone_url into one None, so a database fault was recorded as
"nothing to push to" and metadata read `pushed: null, push_error: null` —
the same ambiguity commit_error already fixed. Each case now carries its
reason into the artifact, and a local git failure during publish is
recorded rather than dropped by .ok().

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 18:58:45 -07:00
Omar SobhandClaude Opus 5 bb34ef1b7e fix(deploy): verify agent images landed instead of trusting the pipe
ci / gates (push) Failing after 6s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
`docker save | docker load` across two SSH connections spliced through a
workstation truncates when either side stalls — observed as `unexpected
EOF` mid-deploy. Nothing checked afterwards, and `docker load` can exit 0
on a short stream, so a partially-populated image could ship to every
fleet node and look like a success.

Now compressed, pipefail-guarded, and verified by comparing image IDs on
the target after the transfer, with one retry for the transient stall.
A failed transfer fails the deploy rather than passing quietly.

The runtime image no longer travels this path at all: it is registry-
hosted now (100.94.185.103:5000/clawmates-runtime:v083-toolchain), built
from deploy/clawmates-runtime/Dockerfile on tank. Only agent-base /
agent-browser / agent-terminal still need save|load, because they exist
in no registry.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 17:39:34 -07:00
Omar SobhandClaude Opus 5 f7e336ff5f fix(missions): make an unrunnable test suite legible, and check the runtime at boot
ci / gates (push) Failing after 6s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Two changes against the same defect: the platform could not tell a missing
capability from a legitimate negative result.

verify_tests returned Option<bool>, collapsing four outcomes into None:
no suite found, docker unreachable, exec failed, and no exit status. When
clawmates-runtime shipped without cargo, every on_green_tests phase
returned None and landed on -wip — identical to the reading for "this
repo has no tests", which is the conclusion I drew and reported. The gate
was correct throughout; it simply could not say why it was unproven.

TestOutcome now names the four cases. Gating is unchanged (only Passed
clears, unproven is never a pass), and tests_verified keeps its tri-state
meaning for existing readers. tests_status and tests_detail are new, so an
artifact distinguishes no_suite from could_not_run, and a CouldNotRun is
logged as the infrastructure fault it is rather than passing quietly.

runtime_preflight probes the runtime container at boot for every tool the
platform invokes inside it and names what each absence disables. This is
the check that was missing: the Dockerfile gained a toolchain, the image
was never built, gw-04 ran the old one for days, and the only symptoms
were an ungated suite and a security scan that scanned nothing. A report,
not a gate — a missing scanner should stop us believing a scan, not stop
the server. Its test guards the probes themselves, since a typo would
produce a permanent false "missing" and train operators to ignore it.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 17:22:47 -07:00
Omar SobhandClaude Opus 5 9bdc3cd89b fix(missions): stop titling commits "phase phase work"
ci / gates (push) Failing after 5s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Mission 019fc4e0 pushed "clawmates: phase phase work" — the iteration
marker was interpolated into a slot whose default already said "phase".
A rerun read correctly ("pass 2 phase work"), so only the common case was
wrong. Cosmetic, but it lands in the operator's git history under their
own name now that delivery commits as them.

Subject is now "clawmates: phase work" and "clawmates: phase work
(pass 2)". The test covers both, since the bug lived only in the branch
the previous shape did not exercise.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 16:56:29 -07:00
Omar SobhandClaude Opus 5 ddab8e35f5 feat(missions): commit as the operator, overridable per deployment
ci / gates (push) Failing after 6s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Delivery commits now carry "Omar Sobh <[email protected]>" by default, so
pushed branches associate with the operator's forge account the way their
own commits do. CLAWMATES_COMMIT_NAME / CLAWMATES_COMMIT_EMAIL override
it — a shared instance wants a bot identity, not a person's.

This is attribution, not the fix. What made 019fc450's phase fail was the
*absence* of any identity: the server container has none of its own, so
git commit exits 128 regardless of which name would have been used. That
was fixed in 25d9805; this only changes the value. The push credential is
GITEA_TOKEN throughout and is untouched by any of it.

Since the author line now names a person, the commit body says plainly
that agents authored the work — otherwise autonomous commits would be
indistinguishable from hand-written ones in git log. Also fixes 13 stray
spaces that a string continuation had baked into every message body.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 16:32:02 -07:00
Omar SobhandClaude Opus 5 1a979f500f fix(missions): judge local work against the remote tip, not the capture base
ci / gates (push) Failing after 6s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Mission 019fc476 lost phase 0's work again, and this time the cause was
the interaction between two fixes I had just shipped.

has_local_work compared HEAD against .git/clawmates-base to decide
whether a checkout held mission work. advance_base_commit moves that
marker to each phase's committed head. So the moment a phase committed
successfully, base == HEAD, has_local_work reported "pristine", and the
next phase's launch reset the work away. Phase 1 wrote CHAIN_MISSING.md.

The preceding mission survived only because its phase 0 FAILED to commit
and left a dirty tree. Fixing that failure is what exposed this one.

One marker was carrying two meanings: "where should the next diff start"
(rolling, per phase) and "is this checkout untouched" (fixed for the
mission). Only the first belongs to clawmates-base. The second is now
`origin/<branch>`, which does not move for the life of the mission, so a
HEAD that differs from it means a phase committed — one commit ago or
five. An unresolvable remote ref preserves, since wrongly skipping a
refresh costs staleness while wrongly resetting destroys a phase.

The existing test passed throughout because it never advanced the base.
It now does, which makes it a reproduction rather than a restatement, and
it needs a real bare origin to resolve origin/main the way a clone does.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 14:59:56 -07:00
Omar SobhandClaude Opus 5 25d9805806 fix(missions): commit under the pipeline's own git identity
ci / gates (push) Failing after 6s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Mission 019fc450 lost its first phase to:

  git commit → exit 128: Author identity unknown

The server container has no git identity — `git config --global
user.email` exits 1 — so any commit fails unless one is supplied.

This is the third consecutive failure whose trigger was agent behaviour
rather than our code. Earlier missions committed only because an agent
had happened to run `git config user.email` in the checkout, leaving a
local identity the server inherited. Alongside the object-permission
split and the reset, the pattern is the same: delivery depended on
incidental side effects of what an agent chose to do, so identical
missions succeeded or failed for reasons invisible in our code.

Supplied via GIT_AUTHOR_*/GIT_COMMITTER_* env on every git call, which
overrides config without a leaked string per invocation and names the
committer as the pipeline. Agents' own commits keep the identity they set.

The test asserts the identity *overrides* an existing local config rather
than trying to unset the developer's global — an override necessarily
also applies when config is absent, and it does not race parallel tests.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 14:19:27 -07:00
Omar SobhandClaude Opus 5 08b2adae23 fix(missions): stop resetting a checkout that holds mission work
ci / gates (push) Failing after 6s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Mission 019fc444 ran two coding phases. Phase 0 created ALPHA.md and
delivery committed it; phase 1 then started and ALPHA.md was gone from
the working tree, so the second phase never saw the first's output.

`ensure_checkout` is called at every phase launch, not once per mission,
and its reuse path runs `git reset --hard origin/<branch>`. That is right
for a checkout picked up cold and destructive for one mid-mission.

Delivery is what made this reachable. Before the mission branch existed,
agent output stayed untracked and a hard reset left it alone. Committing
it makes it tracked, and tracked files absent from origin/<branch> are
exactly what a hard reset removes — so the slice written to stop work
being destroyed is what put it in reach of the thing destroying it. The
flagship shape is the casualty: in research_and_code, the coding phase
never sees the research brief.

`has_local_work` now gates the refresh. It checks both a dirty tree and a
HEAD that has moved off the recorded base, because the two failure shapes
differ: an agent that committed leaves a CLEAN tree at a new HEAD, which
a dirty-tree check alone would miss — and that is precisely the shape
being destroyed. With no recorded base it preserves, since wrongly
skipping a refresh costs staleness while wrongly resetting costs a phase.

This also makes the base-advance fix in 8bad869 live. It was inert in
production: fetch_and_reset calls record_base_commit, overwriting the
advanced base at every phase launch, so both artifacts of 019fc444
recorded origin/main. Their correct per-phase attribution came from the
reset having deleted the earlier work, not from the fix. The two only
compose now that the reset is skipped.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 14:05:56 -07:00
Omar SobhandClaude Opus 5 5b53705c97 fix(missions): let the server and the agent share one git checkout
ci / gates (push) Failing after 5s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Mission 019fc437 lost both phases' work to:

  git add → exit 128: insufficient permission for adding an object
            to repository database .git/objects

cm-api runs as uid 65532; the mission runtime container runs as root;
they share one bind-mounted checkout. Git's .git/objects/xx/ fan-out
directories inherit the ownership of whoever creates them, so an agent
that writes objects first locks the server out of those directories.

The failure is intermittent, which is why the previous run looked clean.
Mission 019fc42b's agents committed their own work, so the blobs already
existed and the server's `git add` never had to write one. Same template,
different agent behaviour, opposite outcome.

`core.sharedRepository` is git's own mechanism for this: objects and refs
are created group- and world-writable, and both parties read the setting
from the shared .git/config. It grants the agent nothing — it is already
root over the whole checkout — and unblocks the server, which was the
party being refused. Applied on clone and on checkout reuse.

Two supporting changes. The artifact now records `commit_error`: this
failure surfaced as `branch: null, push_error: null`, indistinguishable
from a phase that never had work to commit, with the reason only in host
stderr. And the test seeder now calls the production setup function
instead of reimplementing it — building the checkout by hand is what let
a clone-path defect stay invisible to fourteen tests.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 13:53:06 -07:00
Omar SobhandClaude Opus 5 8bad869248 fix(missions): give each phase its own task and its own capture base
ci / gates (push) Failing after 5s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
The first push run against a scratch repo (mission 019fc42b) delivered
two branches correctly but exposed two bugs behind them.

Per-phase instructions were inert. `phase_task_text` took only
(kind, title, description), so `mission_phases.config.task` was accepted
by the API, stored, and read by nothing. Every phase of a mission
received byte-identical text differing only by the kind directive —
so both coding phases did the whole mission instead of their slice,
producing the same two files. The task now reaches the agent as a
trailing THIS PHASE'S TASK block, scoped against the shared brief.

The capture base never advanced. `.git/clawmates-base` is written once
at clone time, so phase two diffed against the original clone point and
reported the union of both phases' files as its own. It now moves to
each phase's committed head after the patch is on disk; the pushed
branch stays cumulative because it is built from HEAD.

Both regression tests were confirmed to fail with their fix disabled.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 13:39:40 -07:00
Omar SobhandClaude Opus 5 e2871c4361 feat(missions): publish the mission branch, gated by commit_policy
ci / gates (push) Failing after 13s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Completes delivery. A phase's work is now captured, committed, gated and
pushed — in that order, so every failure costs strictly less than the one
before it.

Publishing is last for a reason. By the time it runs the patch is on disk, the
artifact is registered and the work is on a local branch, so a rejected ref, a
rotated token or an unreachable forge costs a push and nothing else. A test
pushes at a path that does not exist and asserts the commit is still there
afterwards.

The gate decides the branch name, never whether the work survives:

- green, or policy `always`  → `clawmates/mission-<m8>-<p8>`
- red / unrunnable / no suite → `…-wip`
- `on_reviewer_approval`      → `…-review`

Both land on the forge. A human can inspect, fix and re-push a branch; nobody
can recover work discarded for failing a test. Deleting a red branch
reproduces the old behaviour on purpose rather than by accident.

`verify_tests` runs the project's own suite through the runtime container and
returns `Option<bool>` — `None` for "could not establish", which the gate
treats as unproven. An unreadable exit status is not a pass. That is the same
fail-closed stance as the phase evaluator, and it is here because this tranche
has now found four separate things reporting success while doing nothing.

Never force-push. A rejected update is reported and left alone: the remote ref
belongs to whoever set it, and overwriting it to make delivery look tidy is
how a mission eats someone else's commit.

The push URL is built fresh from the repo row and the ambient token, not read
from `.git/config` — which no longer carries credentials, since agents run as
root in a container that mounts the checkout.

Tests push to a real `git init --bare` remote and assert the ref and its
content actually arrived. A mock would have accepted anything.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 13:15:38 -07:00
Omar SobhandClaude Opus 5 3ea288dbb5 fix(missions): every phase of a mission shared one branch
ci / gates (push) Failing after 7s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
`branch_name` took `[..8]` of both the mission and the phase id. Both are
UUIDv7, which leads with a 48-bit timestamp, so ids minted in the same
millisecond — which is exactly what happens when a mission inserts its phases
in one transaction — share their leading hex. Production produced:

    clawmates/mission-019fc40e-019fc40e

for both the research and the coding phase. Each phase's commit moved the ref
the previous one had just set, so a two-phase mission ended with one branch
and the earlier phase's work reachable only by sha.

The segments now come from opposite ends: the mission keeps its time-ordered
prefix so branches group and sort usefully, and the phase contributes its
random tail so siblings cannot collide.

The existing test missed this because it compared iteration 0 against
iteration 1 of the *same* phase, where the `-i2` suffix guaranteed a
difference. The new test asserts the precondition explicitly — two v7 ids
minted together do share leading hex — and then that their branches differ
anyway.

Also adds the `commit_policy` gate, which three workflow recipes have declared
since they were written with nothing reading it. Two properties it must have:
a failed gate redirects work to `<branch>-wip` rather than discarding it, and
an unrunnable or undiscoverable test suite counts as unproven, never as green.
`discover_test_command` returns None for a `package.json` with no test script,
because `npm test` exits non-zero for a missing script and would read as a red
suite rather than an absent one. Not yet wired to publishing.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 13:08:08 -07:00
Omar SobhandClaude Opus 5 ca1fd46e08 feat(missions): commit captured work to a branch of its own
ci / gates (push) Failing after 5s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Second half of delivery, minus the push. After the patch is on disk and the
artifact registered, the phase's work is committed onto
`clawmates/mission-<mission8>-<phase8>`, with `-i<N>` for re-runs so a second
pass cannot collide with the first.

Three rules hold throughout:

- Never the default branch. The name is derived from the mission and phase, so
  a mission can only ever add a ref nobody else owns.
- Never force. A rejected update gets reported, not overwritten.
- The same exclusions as capture. What was too noisy for a patch is too noisy
  for someone's history — build output, vendored trees, and the workaround
  files agents write when infrastructure fights them. A test drops a 50 KB
  binary in `target/` and a `.gitconfig_temp` beside the real change and
  asserts neither is committed.

Ordering is deliberate: commit runs *after* capture, and a commit failure is
logged without failing the capture. The patch is the guarantee; the branch is
the convenience on top.

The branch is created even when there is nothing to stage, because agents
often commit their own work — `rust_sdlc` has a committer role — and that
commit is unreachable once the checkout is reaped unless a ref points at it.

One test changed meaning rather than breaking: it asserted capture left the
working tree untouched, which was correct while capture stood alone. Capture
now commits, so it asserts the new invariant — work on a namespaced branch, a
clean tree, and the created file present in the commit.

Push is still deliberately absent. Everything here is local, so a bug costs a
retry rather than reaching a remote.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 12:37:31 -07:00
Omar SobhandClaude Opus 5 a0e6b16abc fix(missions): stop agents having to work around git ownership
ci / gates (push) Failing after 9s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
The captured diff from mission 019fc3ba contained the deliverable and, beside
it, a file the agent had invented:

    +++ b/.gitconfig_temp
    +[safe]
    +	directory = /mission/repo

The server clones as uid 65532 and the mission container runs as root, so
every `git` an agent runs is refused with "detected dubious ownership". Agents
do not surface that as a failure — they improvise around it, and the
improvisation lands in the repository. Left alone it would have been committed
and pushed to the user's repo alongside the real work.

The judge got `GIT_CONFIG_*` for this in dd8dad2; the mission containers never
did. They do now — git's environment form of `-c`, inherited by subprocesses,
so it covers the agent's own git, the `git_operations` tool, and anything that
shells out. Scoped to the checkout, never `--global`.

`.gitconfig_temp` is also added to the capture exclusions. The cause is fixed,
but a stray workaround from some future agent should not reach a user's
repository, and the exclusion costs nothing.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 12:11:36 -07:00
Omar SobhandClaude Opus 5 3a383aede6 fix(missions): give the uncapturable marker a real file
The marker registered an artifact at a path with nothing behind it, so any
reader following it would get a bare 404. `_outputs` survives teardown even
when the checkout does not, so the file can and should be written — and it
says plainly what happened rather than leaving an operator to infer it from
an empty response.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 12:03:37 -07:00
Omar SobhandClaude Opus 5 e089360ac8 fix(missions): unblock the capture batch, and restore fetch auth
ci / gates (push) Failing after 10s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Two defects, both found by running a second real coding mission (019fc3ba)
after the first round of fixes. The agent created the file correctly this
time — `file_write` did its job — and capture still produced nothing.

**Head-of-line blocking.** `capture_phase_diff` returns `Ok(None)` when the
checkout is gone, and the caller treated that as success without recording
anything. The phase therefore stayed eligible forever, and because the batch
is bounded at five, five reaped phases from earlier test missions occupied
every slot permanently. A freshly finished coding phase, with its checkout
still on disk, was never reached — and nothing was logged, because nothing had
failed.

Fixed on both axes: an unreachable checkout now writes a `code_diff` marker
recording `captured: false` and why, so the row stops being selected; and the
batch orders newest-first, so live work is captured before archaeology. The
marker also distinguishes "this phase changed nothing" from "we lost the
checkout before looking", which an operator reading the mission needs to be
able to tell apart.

**Fetch lost its credentials.** `scrub_remote_credentials` (P1.1) strips the
token from `.git/config` so agents running as root cannot read it — but
`fetch_and_reset` fetched from the stored remote, which is now anonymous:

    git fetch origin <branch> → exit 128:
    fatal: could not read Username for 'https://git.redclaw.dev'

I accounted for push building a fresh authenticated URL and overlooked that
fetch needs one too. `fetch_and_reset` now takes the authenticated URL the
caller already computes, as does the `--unshallow` deepen. Stderr stays
redacted.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 12:00:54 -07:00
Omar SobhandClaude Opus 5 409ca65ee7 fix(missions): capture from the clone point, and let agents create files
ci / gates (push) Failing after 7s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Two defects found by running a real coding mission (019fc372) rather than a
test. Both made a coding phase look like it produced nothing.

**Capture measured the wrong baseline.** It diffed the working tree against
HEAD, which is correct only while work stays uncommitted. `rust_sdlc` has a
*committer* role, so committing is the intended path — meaning a mission that
did its job properly leaves a clean tree and captured nothing. That is exactly
what happened: the agent created `DELIVERY_PROBE.md`, committed it as
`aa3be95`, and the artifact recorded `empty: true` beside a commit that
plainly contained the work.

`mission_workspace` now records the clone point in `.git/clawmates-base` (in
`.git/`, so it travels with the checkout, stays invisible to the repository,
and cannot be reached by an agent through its pinned workspace), refreshed
whenever `fetch_and_reset` moves HEAD. Capture diffs from there, covering
committed, staged and unstaged changes in one pass. Checkouts predating the
marker fall back to HEAD and say so via `base_recorded: false`.

**Agents could not create files.** `coding_readwrite` granted `file_edit` but
not `file_write`. `file_edit` replaces an exact existing string and rejects an
empty `old_string`, so creating a new file was impossible. The mission
transcript is unambiguous: "the tool rejected empty old_string... the shell is
restricted", after which the agent worked around it through `shell`. The
comment above that profile has claimed it grants file_write since the day it
was written; the list never contained it.

Also broadens capture from coding/benchmark/security_scan to every phase kind
of a repo-bearing mission: `phase_task_text` tells research phases to "save
findings under /mission/repo/research/", so filtering by kind would have
discarded every research brief such a mission produced.

Regression tests cover committed-only and committed-plus-uncommitted work
against a real git repo.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 10:20:00 -07:00
Omar SobhandClaude Opus 5 322c1be89c feat(missions): capture runs automatically, and once more before teardown
ci / gates (push) Failing after 5s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Wires diff capture into the two sweeps that matter.

`phase_runner::sweep_once` gains `capture_finished_coding_phases`, guarded by
`NOT EXISTS (code_diff for this phase)`. Deliberately a separate step rather
than a hook on `close_finished_phases` or `evaluate_finished_phases`: a phase
reaches `completed` through one or the other depending on whether it declared
a `done_when`, so hanging capture off either would silently skip half the
missions. The guard also makes it retryable — a capture that errors is simply
re-selected next tick.

`mission_runtime::sweep_once` captures anything still outstanding immediately
before `teardown_container`, which deletes the checkout. This covers what the
phase sweep structurally cannot: a mission that ended `failed` mid-coding
still has real work on disk, and reaping it unexamined destroys the only
evidence of what the agents actually did.

Applies to coding, benchmark and security_scan phases — all three operate on
a repo.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-02 10:03:37 -07:00
Omar SobhandClaude Opus 5 716ee9a304 feat(missions): capture a coding phase's diff to durable storage
First half of mission delivery: the work is captured before anything is
published. A coding mission has until now produced nothing durable — the
checkout is deleted thirty minutes after completion and `register_artifact`
had no callers at all, so the only surviving output was an LLM narrative of
what the agents said they did.

`capture_phase_diff` writes `diff.patch`, `diffstat.txt` and `delivery.json`
under `<missions_root>/_outputs/<mission>/<phase>/` and registers a
`code_diff` artifact. That directory is a *sibling* of the per-mission
directories the sweeper removes, and outside every bind mount handed to a
container — so teardown cannot take the record with it and agents cannot edit
their own evidence.

Three details that decide whether this works at all:

- `git add --intent-to-add` before diffing. Untracked files are invisible to
  `git diff`, and a phase that only *creates* files is the likeliest shape for
  generated code — silently capturing an empty patch would be the worst
  possible failure. The index is reset afterwards so capture leaves the tree
  exactly as the agents left it, which the test asserts.
- Build output is excluded by pathspec (`target`, `node_modules`, `.venv`, …).
  A phase that ran `cargo build` leaves a directory larger than the repo.
- An empty diff is still an artifact, flagged `empty: true`. "This coding
  phase wrote no code" is currently invisible to an operator and is worth
  saying out loud.

`RegisterArtifact` gains `metadata`, which the column has had since 0047 and
nothing ever wrote; the diffstat and base sha go there. No migration needed —
`kind` is unconstrained TEXT and the column already exists.

Tests run against a real `git init` repo rather than a mock: every bug in this
area so far came from git behaving differently than assumed, and a fake git
would have agreed with the assumption. `capture_phase_diff_at` takes explicit
paths so parallel tests cannot race through the process-global
CLAWMATES_MISSIONS_ROOT — the first version of these tests did exactly that
and two of four failed non-deterministically.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-01 22:46:15 -07:00
Omar SobhandClaude Opus 5 ea3d145aac fix(missions): stop leaving an access token in every mission checkout
`with_ambient_auth` embeds GITEA_TOKEN in the clone URL, and git persists that
URL verbatim as the `origin` remote. The checkout is bind-mounted into a
container the agents run in as root, so the token sat in a file every mission
agent could read — and it reaches every repository that token reaches, not
just the one being worked on.

The remote is now rewritten to the bare URL immediately after clone. Delivery
does not depend on the stored URL: it will build a fresh authenticated URL at
push time, which also means a rotated token starts working at once rather than
after the next clone. Best-effort and non-fatal — a checkout that keeps its
token still works, and failing a mission over it would trade a real capability
for a situation already logged.

`strip_credentials` only treats an `@` in the *authority* as a separator, so a
path containing `@` (scoped npm-style names) is left alone.

Also, two changes delivery needs:

- `--depth 1` becomes `--filter=blob:none --single-branch`. A shallow clone
  usually cannot push a new branch ("shallow update not allowed"), which is
  exactly what mission delivery must do. A partial clone keeps full history —
  so a base commit stays meaningful and a diff has something to be relative
  to — while fetching blobs on demand.
- `fetch_and_reset` deepens a pre-existing shallow checkout once, up front,
  rather than letting the push fail later with work on the line.

Fetch stderr is now redacted too; it can echo the remote URL.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-01 22:03:07 -07:00
Omar SobhandClaude Opus 5 dd8dad2ad4 fix(evaluator): git ownership exception now reaches tools that call git
ci / gates (push) Failing after 6s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
The argv rewrite added in 491449f fixed `git status` and nothing else.
gitleaks, trivy and semgrep run git themselves, so they still hit:

    fatal: detected dubious ownership in repository at '...'

Mission 019fc073 showed both halves at once: git reported a clean tree while
gitleaks "scanned 0 commits", and the judge correctly refused to call the
condition met rather than accepting a scan that had examined nothing. That is
the fail-closed behaviour working — and a scan reporting clean after scanning
zero commits is precisely the false signal this tranche keeps finding.

Replaces the argv rewrite with `GIT_CONFIG_COUNT`/`_KEY_0`/`_VALUE_0`, git's
documented environment form of `-c`. Being environment, it is inherited by
subprocesses, so one setting covers git and every tool that shells out to it.
Still scoped to the single checkout — never `--global` or `*`, which would
disable the protection container-wide.

`container_exec` grows `exec_with_env`; `exec` keeps its signature.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-01 20:16:05 -07:00
Omar SobhandClaude Opus 5 d90a42b759 fix: three gaps the P0 validation runs exposed
ci / gates (push) Failing after 7s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Validating P0 against production found one bug in each of the three pieces,
none of which any test would have caught.

**The scanners were installed but not allow-listed.** Mission 019fc058's
condition asked for a gitleaks result; `gitleaks detect` came back
`ran=false`, and the judge said it could not verify. P0.3 put the binaries in
the image and never added them to `evaluator_tools::ALLOWED_PROGRAMS`, so the
judge could not invoke the tools installed for it. Adds gitleaks, trivy,
semgrep and `which`.

**Every `continue` after a fire claim leaked the claim.** Introduced by the
scheduler fix itself: the orphan-agent and empty-action paths skipped
`complete_fire`, so the row stayed `claimed` — which reads as a crash
mid-fire, meaning the routine is re-claimed forever and the table grows one
stuck row per occurrence. Observed in production: five `claimed` rows, no
dispatch, no `routine_runs`. Both paths now settle with a reason, and log it.

**The agent writes its own identity files into the user's repository.**
`workspace.path` is pinned to the repo root, so the runtime drops AGENTS.md,
HEARTBEAT.md, IDENTITY.md, MEMORY.md, SOUL.md, TOOLS.md and USER.md into the
checkout — SOUL.md opens "Who You Are / You're not a chatbot." Two
consequences: every mission's tree is permanently dirty, so a `done_when`
about a clean tree can never pass; and P1's `git add -A` would have committed
the agent's SOUL.md into someone's repository and pushed it. The P1 deny-list
covered build artifacts and would not have caught this.

Fixed by writing the names to `.git/info/exclude` after clone — local to the
checkout, never itself a change, and it suppresses only *untracked* files, so
a repo that genuinely tracks its own AGENTS.md still reports modifications to
it. Idempotent, and preserves any pre-existing exclude.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-01 19:54:34 -07:00
Omar SobhandClaude Opus 5 491449f3ce fix(evaluator): git refused the checkout it was asked to verify
ci / gates (push) Failing after 5s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
Found by the P0.1 verification run, which is the point of it. Mission
019fc02e's judge executed `git status` for real — and got exit 128:

    fatal: detected dubious ownership in repository at
    '/var/lib/clawmates-missions/019fc02e-.../repo'

The server clones as uid 65532; the runtime container the judge execs into
runs as root; git's ownership check refuses the repository. So the judge's
most direct verification tool was failing on every mission. It recovered here
by inferring a clean tree from `ls -la` and `find`, and reasoned correctly —
but that is inference from a directory listing standing in for the command
that answers the question directly.

`git` invocations now carry `-c safe.directory=<workdir>`, scoped to that one
checkout. Not `--global`: the protection exists for multi-user machines where
another user could plant a hostile `.git/config`, and disabling it container-
wide to fix one path would trade a real guarantee for convenience. Applied
per-invocation rather than baked into the image so it travels with the workdir
and cannot drift out of sync with it.

Tests cover the rewrite, that non-git commands are untouched, and that a
rewritten `git push` still fails the allow-list — the injected `-c` flags must
not become a way past validation.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-01 19:01:30 -07:00
36 changed files with 7125 additions and 79 deletions
Generated
+14
View File
@@ -946,6 +946,7 @@ dependencies = [
"cm-config", "cm-config",
"cm-db", "cm-db",
"cm-domain", "cm-domain",
"cm-files",
"cm-llm", "cm-llm",
"cm-orchestrator", "cm-orchestrator",
"cm-runtime", "cm-runtime",
@@ -969,6 +970,8 @@ dependencies = [
"serde_yaml", "serde_yaml",
"sha2", "sha2",
"sqlx", "sqlx",
"tar",
"tempfile",
"thiserror 2.0.18", "thiserror 2.0.18",
"time", "time",
"tokio", "tokio",
@@ -5027,6 +5030,17 @@ dependencies = [
"windows", "windows",
] ]
[[package]]
name = "tar"
version = "0.4.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f6221d9a6003c78398e3b239969f352578258df48c8eb051caadae0015bc840"
dependencies = [
"filetime",
"libc",
"xattr",
]
[[package]] [[package]]
name = "tempfile" name = "tempfile"
version = "3.27.0" version = "3.27.0"
+3
View File
@@ -36,6 +36,9 @@ publish = false
# Shared dependency versions; crates opt in via { workspace = true }. # Shared dependency versions; crates opt in via { workspace = true }.
serde = { version = "1", features = ["derive"] } serde = { version = "1", features = ["derive"] }
serde_json = "1" serde_json = "1"
# Streaming tar for mission copy-in/copy-out (no compression: the payload is
# a git checkout on a local socket, so CPU spent zipping buys nothing).
tar = "0.4"
thiserror = "2" thiserror = "2"
uuid = { version = "1", features = ["v7", "serde"] } uuid = { version = "1", features = ["v7", "serde"] }
proptest = "1" proptest = "1"
+7 -1
View File
@@ -266,7 +266,7 @@ async fn run() -> Result<(), String> {
terminals, terminals,
providers: provider_registry, providers: provider_registry,
}, },
blob, blob.clone(),
); );
// Durable §15 path: expires overdue approvals and resumes decided runs // Durable §15 path: expires overdue approvals and resumes decided runs
// even if the deciding request's process died mid-flight. // even if the deciding request's process died mid-flight.
@@ -388,6 +388,7 @@ async fn run() -> Result<(), String> {
.with_broker(PathBuf::from(&config.broker.socket_path)) .with_broker(PathBuf::from(&config.broker.socket_path))
.with_oauth(config.oauth.clone()) .with_oauth(config.oauth.clone())
.with_billing(config.billing.clone()) .with_billing(config.billing.clone())
.with_blobs(blob.clone())
.with_file_root( .with_file_root(
(config.storage.backend == cm_config::StorageBackend::Local) (config.storage.backend == cm_config::StorageBackend::Local)
.then(|| PathBuf::from(&config.storage.data_dir)), .then(|| PathBuf::from(&config.storage.data_dir)),
@@ -402,6 +403,11 @@ async fn run() -> Result<(), String> {
.await .await
.map_err(|e| format!("bind {} failed: {e}", config.listen_addr))?; .map_err(|e| format!("bind {} failed: {e}", config.listen_addr))?;
println!("clawmates-server listening on {}", config.listen_addr); println!("clawmates-server listening on {}", config.listen_addr);
// Say plainly whether the mission runtime carries the tools we invoke in
// it. The image on the host silently fell behind its Dockerfile once, and
// every consequence — an ungated test suite, a scan that scanned nothing —
// looked like a normal result rather than a broken deployment.
cm_api::runtime_preflight::report_at_boot();
// Graceful shutdown: on SIGTERM/Ctrl-C, stop accepting, finish in-flight // Graceful shutdown: on SIGTERM/Ctrl-C, stop accepting, finish in-flight
// requests, then DRAIN the sandbox managers so no container is left running. // requests, then DRAIN the sandbox managers so no container is left running.
let shutdown = async move { let shutdown = async move {
+3
View File
@@ -32,6 +32,8 @@ cm-brain = { path = "../cm-brain" }
cm-config = { path = "../cm-config" } cm-config = { path = "../cm-config" }
cm-db = { path = "../cm-db" } cm-db = { path = "../cm-db" }
cm-domain = { path = "../cm-domain" } cm-domain = { path = "../cm-domain" }
cm-files = { path = "../cm-files" }
tar = { workspace = true }
cm-llm = { path = "../cm-llm" } cm-llm = { path = "../cm-llm" }
cm-orchestrator = { path = "../cm-orchestrator", features = ["provider"] } cm-orchestrator = { path = "../cm-orchestrator", features = ["provider"] }
cm-runtime = { path = "../cm-runtime" } cm-runtime = { path = "../cm-runtime" }
@@ -51,6 +53,7 @@ uuid = { workspace = true }
[dev-dependencies] [dev-dependencies]
axum = { version = "0.8", features = ["ws"] } axum = { version = "0.8", features = ["ws"] }
tempfile = "3"
jsonwebtoken = "9" jsonwebtoken = "9"
eventsource-stream = "0.2" eventsource-stream = "0.2"
reqwest = { version = "0.12", default-features = false, features = [ reqwest = { version = "0.12", default-features = false, features = [
+222
View File
@@ -0,0 +1,222 @@
//! Merging a delivered branch into the base, when that is provably safe.
//!
//! Every mission type delivers to a branch and never to `main`. For most that
//! is where it should stop — a human reads the code and merges. But some
//! missions only ever *add* files in a folder they own: a paper catalogue, a
//! benchmark record. Those branches carry no judgement call, and leaving them
//! to pile up unmerged means the work is done but not actually in the vault.
//!
//! # Additive-only is a property, not a preference
//!
//! The gate is not "is this mission type trusted". It is measured from the
//! diff: if the branch modifies or deletes anything that already existed, it
//! does not qualify, whatever its template says. A research harvest that
//! somehow rewrote a hand-written note would be refused by the same check
//! that lets its new notes through.
//!
//! Three conditions, all required:
//!
//! 1. the mission type declares [`MergePolicy::AdditiveOnly`]
//! 2. verification passed — a run that did not prove its work does not merge
//! 3. the diff against the base contains only additions
//!
//! Anything else lands as a branch for a human, which is the existing
//! behaviour and the safe default.
use std::path::Path;
/// What a mission type is allowed to do with its own branch.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MergePolicy {
/// Always leave the branch for a human. Correct for anything that touches
/// code: `refactor`, `research_and_code`, security patches.
Never,
/// Merge automatically when the diff is provably additive and the run
/// verified. Correct for catalogues and recorded measurements.
AdditiveOnly,
}
impl MergePolicy {
/// Parse a template's `merge_policy`. Unknown values fall back to `Never`
/// and say so: a typo must not silently grant auto-merge.
pub fn parse(raw: Option<&str>) -> MergePolicy {
match raw.map(str::trim) {
Some("additive_only") => MergePolicy::AdditiveOnly,
Some("never") | None => MergePolicy::Never,
Some(other) => {
eprintln!(
"auto_merge: unknown merge_policy {other:?} — refusing to auto-merge"
);
MergePolicy::Never
}
}
}
}
/// Why a branch was or was not merged. The reason is always recorded: a
/// branch that silently did not merge is indistinguishable from one that was
/// never delivered.
#[derive(Debug, Clone)]
pub struct MergeOutcome {
pub merged: bool,
pub reason: String,
}
impl MergeOutcome {
fn refused(reason: impl Into<String>) -> MergeOutcome {
MergeOutcome {
merged: false,
reason: reason.into(),
}
}
}
/// Classify a `git diff --name-status` body.
///
/// Returns the offending entries, empty when every change is an addition.
/// Split out so the rule is testable without a repository.
pub fn non_additive_changes(name_status: &str) -> Vec<String> {
name_status
.lines()
.filter(|l| !l.trim().is_empty())
.filter(|l| {
// Status is the first field: A/M/D/R###/C###.
!matches!(l.chars().next(), Some('A'))
})
.map(|l| l.trim().to_string())
.collect()
}
async fn git(repo: &Path, args: &[&str]) -> Result<String, String> {
let out = tokio::process::Command::new("git")
.arg("-C")
.arg(repo)
.args(["-c", &format!("safe.directory={}", repo.display())])
.args(args)
.env("GIT_AUTHOR_NAME", crate::mission_delivery::commit_identity().0)
.env("GIT_AUTHOR_EMAIL", crate::mission_delivery::commit_identity().1)
.env(
"GIT_COMMITTER_NAME",
crate::mission_delivery::commit_identity().0,
)
.env(
"GIT_COMMITTER_EMAIL",
crate::mission_delivery::commit_identity().1,
)
.output()
.await
.map_err(|e| format!("spawn git: {e}"))?;
if !out.status.success() {
return Err(format!(
"git {}{}: {}",
args.first().copied().unwrap_or("?"),
out.status,
crate::mission_workspace::redact_token(&String::from_utf8_lossy(&out.stderr))
.chars()
.take(300)
.collect::<String>()
));
}
Ok(String::from_utf8_lossy(&out.stdout).into_owned())
}
/// Merge `branch` into `base` and push, if all three conditions hold.
///
/// Never returns `Err` for a refusal — a refusal is a normal outcome with a
/// reason. `Err` is reserved for the merge itself going wrong after we decided
/// to attempt it.
pub async fn try_merge(
repo: &Path,
push_url: &str,
branch: &str,
base: &str,
policy: MergePolicy,
verified: bool,
) -> Result<MergeOutcome, String> {
if policy != MergePolicy::AdditiveOnly {
return Ok(MergeOutcome::refused(
"merge_policy is not additive_only; left for a human",
));
}
if !verified {
return Ok(MergeOutcome::refused(
"run did not verify; refusing to merge unproven work",
));
}
// Compare against the base as the REMOTE has it, not a local ref that may
// be stale. `...` gives changes on the branch since it diverged, so an
// unrelated commit landing on main meanwhile is not misread as ours.
git(repo, &["fetch", push_url, base]).await?;
let diff = git(
repo,
&["diff", "--name-status", &format!("FETCH_HEAD...{branch}")],
)
.await?;
let offending = non_additive_changes(&diff);
if !offending.is_empty() {
return Ok(MergeOutcome::refused(format!(
"diff is not additive ({} non-add change(s), first: {}); left for a human",
offending.len(),
offending.first().map(String::as_str).unwrap_or("?")
)));
}
if diff.trim().is_empty() {
return Ok(MergeOutcome::refused("branch adds nothing"));
}
// Merge onto the freshly fetched base rather than a local branch.
git(repo, &["checkout", "-B", base, "FETCH_HEAD"]).await?;
if let Err(e) = git(
repo,
&["merge", "--no-ff", "-m", &format!("auto-merge {branch}"), branch],
)
.await
{
// Leave the repo clean so the next run is not fighting a wedged merge.
let _ = git(repo, &["merge", "--abort"]).await;
return Ok(MergeOutcome::refused(format!(
"merge conflicted ({e}); left for a human"
)));
}
git(repo, &["push", push_url, &format!("HEAD:refs/heads/{base}")]).await?;
Ok(MergeOutcome {
merged: true,
reason: format!("additive-only and verified; merged into {base}"),
})
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn only_pure_additions_qualify() {
assert!(non_additive_changes("A\t60 Papers/a.md\nA\t60 Papers/b.md\n").is_empty());
// A modification disqualifies the whole branch.
let m = non_additive_changes("A\t60 Papers/a.md\nM\tREADME.md\n");
assert_eq!(m.len(), 1);
assert!(m[0].contains("README.md"));
// So do deletes and renames — a rename is a delete plus an add, and
// the delete half can destroy hand-written work.
assert_eq!(non_additive_changes("D\tnotes/old.md\n").len(), 1);
assert_eq!(non_additive_changes("R100\ta.md\tb.md\n").len(), 1);
}
#[test]
fn an_unknown_policy_never_grants_auto_merge() {
assert_eq!(MergePolicy::parse(None), MergePolicy::Never);
assert_eq!(MergePolicy::parse(Some("never")), MergePolicy::Never);
assert_eq!(
MergePolicy::parse(Some("additive_only")),
MergePolicy::AdditiveOnly
);
// A typo must fail closed, not open.
assert_eq!(MergePolicy::parse(Some("aditive_only")), MergePolicy::Never);
assert_eq!(MergePolicy::parse(Some("always")), MergePolicy::Never);
}
}
+19 -1
View File
@@ -90,7 +90,19 @@ pub async fn exec(
argv: &[String], argv: &[String],
timeout: Duration, timeout: Duration,
) -> Result<ExecOutput, String> { ) -> Result<ExecOutput, String> {
let fut = exec_inner(docker, container, workdir, argv); exec_with_env(docker, container, workdir, argv, &[], timeout).await
}
/// As [`exec`], with extra environment for the command.
pub async fn exec_with_env(
docker: &Docker,
container: &str,
workdir: Option<&str>,
argv: &[String],
env: &[String],
timeout: Duration,
) -> Result<ExecOutput, String> {
let fut = exec_inner(docker, container, workdir, argv, env);
match tokio::time::timeout(timeout, fut).await { match tokio::time::timeout(timeout, fut).await {
Err(_) => Err(format!( Err(_) => Err(format!(
"timed out after {}s (the command may still be running in {container})", "timed out after {}s (the command may still be running in {container})",
@@ -105,6 +117,7 @@ async fn exec_inner(
container: &str, container: &str,
workdir: Option<&str>, workdir: Option<&str>,
argv: &[String], argv: &[String],
env: &[String],
) -> Result<ExecOutput, String> { ) -> Result<ExecOutput, String> {
let created = docker let created = docker
.create_exec( .create_exec(
@@ -112,6 +125,11 @@ async fn exec_inner(
CreateExecOptions { CreateExecOptions {
cmd: Some(argv.to_vec()), cmd: Some(argv.to_vec()),
working_dir: workdir.map(str::to_string), working_dir: workdir.map(str::to_string),
env: if env.is_empty() {
None
} else {
Some(env.to_vec())
},
attach_stdout: Some(true), attach_stdout: Some(true),
attach_stderr: Some(true), attach_stderr: Some(true),
..Default::default() ..Default::default()
+495
View File
@@ -0,0 +1,495 @@
//! What a continuous mission has already covered.
//!
//! A recurring mission's hard problem is not running the agent — that is 23
//! seconds — it is knowing what it already did last time. A research mission
//! with no memory of prior runs resurfaces the same papers forever and reports
//! success every time.
//!
//! This module keeps that record. It is deliberately small: an index derived
//! from the corpus, never the corpus itself. The vault is the source of truth,
//! the index is rebuildable, and a hand-edited note is never "wrong".
//!
//! # Two kinds, because the real vault forced it
//!
//! The plan assumed notes would carry `arxiv:` / `doi:` / `url:` frontmatter.
//! Measured against the actual vault: **416 notes, 145 with frontmatter, and
//! zero with any of those keys.** The dominant keys are repo-sync metadata
//! (`node`, `org`, `gitea`) and course-note fields (`presenter`, `session`).
//! An ingester keyed only on external identity would have indexed nothing —
//! the same shape of failure as everything else this week.
//!
//! So `note` rows record coverage (what the vault already contains, keyed by
//! path) and `source` rows record consumption (external things a mission
//! read, keyed by natural id). They answer different questions and a
//! continuous mission needs both: "have I already written about this topic?"
//! and "have I already read this paper?".
use sha2::{Digest, Sha256};
use uuid::Uuid;
/// A note parsed out of the vault, ready to be indexed.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ParsedNote {
/// Vault-relative path, used as identity for `kind = 'note'`.
pub path: String,
pub title: Option<String>,
pub content_hash: String,
/// An external identity the note declares for itself, if any. Nothing in
/// the vault does this today; missions writing new notes are expected to.
pub declared_source_id: Option<String>,
}
impl ParsedNote {
/// `note:<path>` — the `source_id` this note occupies in the index.
pub fn source_id(&self) -> String {
format!("note:{}", self.path)
}
}
/// Hash content for change detection. Not a dedupe key — identity is
/// `source_id`; this only distinguishes "unchanged" from "edited".
pub fn content_hash(body: &str) -> String {
let mut h = Sha256::new();
h.update(body.as_bytes());
format!("{:x}", h.finalize())
}
/// Split YAML frontmatter from the body.
///
/// Returns `(frontmatter, body)`. A note without frontmatter — 271 of the 416
/// in the real vault — yields `("", whole file)` rather than being skipped.
/// Skipping them would drop two thirds of the corpus on the floor.
fn split_frontmatter(text: &str) -> (&str, &str) {
let Some(rest) = text.strip_prefix("---") else {
return ("", text);
};
let rest = rest.strip_prefix('\n').unwrap_or(rest);
match rest.find("\n---") {
Some(end) => {
let body = &rest[end + 4..];
(&rest[..end], body.strip_prefix('\n').unwrap_or(body))
}
// An opening fence with no close is malformed; treat the whole file as
// body rather than swallowing it as frontmatter.
None => ("", text),
}
}
/// Read one scalar key out of a frontmatter block.
///
/// Deliberately not a YAML parser. The vault's frontmatter is flat
/// `key: value` with occasional quotes and one list (`tags`), and pulling in a
/// YAML dependency to read three keys would be more surface than it is worth.
fn frontmatter_value<'a>(fm: &'a str, key: &str) -> Option<&'a str> {
for line in fm.lines() {
let line = line.trim();
let Some((k, v)) = line.split_once(':') else {
continue;
};
if !k.trim().eq_ignore_ascii_case(key) {
continue;
}
let v = v.trim().trim_matches('"').trim_matches('\'').trim();
if !v.is_empty() {
return Some(v);
}
}
None
}
/// Which frontmatter keys may declare an external identity, in priority order.
///
/// None of these appear in the vault today. They are the contract for notes
/// that missions write from here on, and the reason a `source:` key is NOT in
/// the list: the vault already uses `source:` for local filesystem paths of
/// course material (`/Users/quantum/Downloads/...`), which is provenance, not
/// a citable external identity. Treating it as one would fill the seen-set
/// with 25 rows keyed on a laptop path.
const IDENTITY_KEYS: &[&str] = &["source_id", "arxiv", "doi", "url", "permalink"];
/// Parse a note. `path` must be vault-relative.
pub fn parse_note(path: &str, text: &str) -> ParsedNote {
let (fm, body) = split_frontmatter(text);
let declared_source_id = IDENTITY_KEYS.iter().find_map(|k| {
frontmatter_value(fm, k).map(|v| {
// `source_id` is already qualified; the others name their scheme.
if *k == "source_id" || v.contains(':') {
v.to_string()
} else {
format!("{k}:{v}")
}
})
});
// Title: the first markdown H1, else the filename stem. Frontmatter has no
// consistent title key in this vault.
let title = body
.lines()
.find_map(|l| l.strip_prefix("# ").map(str::trim))
.filter(|t| !t.is_empty())
.map(str::to_string)
.or_else(|| {
std::path::Path::new(path)
.file_stem()
.map(|s| s.to_string_lossy().into_owned())
});
ParsedNote {
path: path.to_string(),
title,
// Hash the body, not the whole file: re-syncing a repo note rewrites
// `updated:`/`size_kb:` in frontmatter without the prose changing, and
// that should not read as an edit.
content_hash: content_hash(body),
declared_source_id,
}
}
/// What a re-index actually did. `unchanged` is the number that matters: on a
/// vault nobody edited it should equal the note count.
#[derive(Debug, Default, Clone, PartialEq, Eq)]
pub struct IndexStats {
pub scanned: usize,
pub inserted: usize,
pub updated: usize,
pub unchanged: usize,
}
/// Walk a checkout and index every markdown note.
///
/// Skips `.git` and Obsidian's own `.obsidian` config directory — indexing an
/// editor's workspace state as knowledge would be noise.
pub fn collect_notes(root: &std::path::Path) -> Vec<ParsedNote> {
fn walk(dir: &std::path::Path, root: &std::path::Path, out: &mut Vec<ParsedNote>) {
let Ok(entries) = std::fs::read_dir(dir) else {
return;
};
for entry in entries.flatten() {
let path = entry.path();
let name = entry.file_name();
let name = name.to_string_lossy();
if name.starts_with('.') {
continue;
}
if path.is_dir() {
walk(&path, root, out);
} else if path.extension().and_then(|e| e.to_str()) == Some("md") {
let Ok(text) = std::fs::read_to_string(&path) else {
continue;
};
let rel = path
.strip_prefix(root)
.unwrap_or(&path)
.to_string_lossy()
.into_owned();
out.push(parse_note(&rel, &text));
}
}
}
let mut out = Vec::new();
walk(root, root, &mut out);
out.sort_by(|a, b| a.path.cmp(&b.path));
out
}
/// Upsert one item. Returns whether the row was new.
#[allow(clippy::too_many_arguments)]
pub async fn record(
pool: &sqlx::PgPool,
workspace_id: Uuid,
corpus_id: &str,
kind: &str,
source_id: &str,
title: Option<&str>,
path: Option<&str>,
url: Option<&str>,
content_hash: &str,
mission_id: Option<Uuid>,
) -> Result<bool, String> {
// `last_seen_at` always moves; `first_seen_at` and `mission_id` never do.
// The first mission to find a source keeps the credit, which is what makes
// "did THIS run contribute anything new" answerable.
let row: (bool,) = sqlx::query_as(
"INSERT INTO corpus_items
(id, workspace_id, corpus_id, kind, source_id, title, path, url,
content_hash, mission_id)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)
ON CONFLICT (workspace_id, corpus_id, source_id) DO UPDATE
SET last_seen_at = now(),
title = COALESCE(EXCLUDED.title, corpus_items.title),
path = COALESCE(EXCLUDED.path, corpus_items.path),
url = COALESCE(EXCLUDED.url, corpus_items.url),
content_hash = EXCLUDED.content_hash
RETURNING (xmax = 0) AS inserted",
)
.bind(Uuid::now_v7())
.bind(workspace_id)
.bind(corpus_id)
.bind(kind)
.bind(source_id)
.bind(title)
.bind(path)
.bind(url)
.bind(content_hash)
.bind(mission_id)
.fetch_one(pool)
.await
.map_err(|e| format!("record corpus item {source_id}: {e}"))?;
Ok(row.0)
}
/// Has this corpus already seen this `source_id`?
pub async fn seen(
pool: &sqlx::PgPool,
workspace_id: Uuid,
corpus_id: &str,
source_id: &str,
) -> Result<bool, String> {
// `SELECT 1` is INT4; binding it as i64 fails to decode.
let row: Option<(i32,)> = sqlx::query_as(
"SELECT 1 FROM corpus_items
WHERE workspace_id = $1 AND corpus_id = $2 AND source_id = $3",
)
.bind(workspace_id)
.bind(corpus_id)
.bind(source_id)
.fetch_optional(pool)
.await
.map_err(|e| format!("seen({source_id}): {e}"))?;
Ok(row.is_some())
}
/// Of these candidate ids, which has this corpus NOT seen?
///
/// The shape a research agent actually needs: it has ten search hits and wants
/// to know which are worth fetching. One round trip, not ten.
pub async fn unseen(
pool: &sqlx::PgPool,
workspace_id: Uuid,
corpus_id: &str,
candidates: &[String],
) -> Result<Vec<String>, String> {
if candidates.is_empty() {
return Ok(Vec::new());
}
let rows: Vec<(String,)> = sqlx::query_as(
"SELECT source_id FROM corpus_items
WHERE workspace_id = $1 AND corpus_id = $2 AND source_id = ANY($3)",
)
.bind(workspace_id)
.bind(corpus_id)
.bind(candidates)
.fetch_all(pool)
.await
.map_err(|e| format!("unseen: {e}"))?;
let known: std::collections::HashSet<String> = rows.into_iter().map(|r| r.0).collect();
Ok(candidates
.iter()
.filter(|c| !known.contains(*c))
.cloned()
.collect())
}
/// How many NEW sources a mission contributed.
///
/// The verification predicate for a continuous research mission. `record`
/// never reassigns `mission_id` on conflict, so the first mission to find a
/// source keeps the credit and a rerun cannot inflate its own count by
/// re-recording what an earlier run already had.
///
/// A mission whose answer is zero produced nothing, whatever its transcript
/// says — which is the check the 0030-0044 generation of this feature lacked.
pub async fn contributed(
pool: &sqlx::PgPool,
workspace_id: Uuid,
corpus_id: &str,
mission_id: Uuid,
) -> Result<i64, String> {
let row: (i64,) = sqlx::query_as(
"SELECT count(*) FROM corpus_items
WHERE workspace_id = $1 AND corpus_id = $2 AND mission_id = $3
AND kind = 'source'",
)
.bind(workspace_id)
.bind(corpus_id)
.bind(mission_id)
.fetch_one(pool)
.await
.map_err(|e| format!("contributed({mission_id}): {e}"))?;
Ok(row.0)
}
/// Index every note in a checkout. Idempotent by construction.
pub async fn index_vault(
pool: &sqlx::PgPool,
workspace_id: Uuid,
corpus_id: &str,
root: &std::path::Path,
) -> Result<IndexStats, String> {
let notes = collect_notes(root);
let mut stats = IndexStats {
scanned: notes.len(),
..Default::default()
};
for note in &notes {
let existing: Option<(String,)> = sqlx::query_as(
"SELECT content_hash FROM corpus_items
WHERE workspace_id = $1 AND corpus_id = $2 AND source_id = $3",
)
.bind(workspace_id)
.bind(corpus_id)
.bind(note.source_id())
.fetch_optional(pool)
.await
.map_err(|e| format!("lookup {}: {e}", note.path))?;
match existing {
Some((hash,)) if hash == note.content_hash => {
stats.unchanged += 1;
continue;
}
Some(_) => stats.updated += 1,
None => stats.inserted += 1,
}
record(
pool,
workspace_id,
corpus_id,
"note",
&note.source_id(),
note.title.as_deref(),
Some(&note.path),
None,
&note.content_hash,
None,
)
.await?;
// A note that declares an external identity also registers as a
// consumed source, so a later mission does not re-read what an
// earlier one already wrote up.
if let Some(sid) = &note.declared_source_id {
record(
pool,
workspace_id,
corpus_id,
"source",
sid,
note.title.as_deref(),
Some(&note.path),
None,
&note.content_hash,
None,
)
.await?;
}
}
Ok(stats)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn frontmatter_is_split_from_body() {
let (fm, body) = split_frontmatter("---\ntype: lecture\n---\n# Title\n\ntext\n");
assert_eq!(fm, "type: lecture");
assert!(body.starts_with("# Title"));
}
/// 271 of the vault's 416 notes have no frontmatter. Dropping them would
/// discard two thirds of the corpus.
#[test]
fn a_note_without_frontmatter_is_still_a_note() {
let (fm, body) = split_frontmatter("# Plain\n\nno frontmatter here\n");
assert_eq!(fm, "");
assert!(body.starts_with("# Plain"));
let n = parse_note("Daily/x.md", "# Plain\n\nbody\n");
assert_eq!(n.title.as_deref(), Some("Plain"));
assert_eq!(n.declared_source_id, None);
}
/// An unterminated fence must not swallow the file.
#[test]
fn malformed_frontmatter_is_treated_as_body() {
let (fm, body) = split_frontmatter("---\nbroken: yes\nno closing fence\n");
assert_eq!(fm, "");
assert!(body.contains("no closing fence"));
}
/// The vault's real `source:` values are local filesystem paths of course
/// material. Treating those as citable identity would fill the seen-set
/// with 25 rows keyed on a laptop path.
#[test]
fn a_local_source_path_is_not_an_external_identity() {
let note = parse_note(
"50 APESS 2026/Lectures/talk.md",
"---\nsource: \"/Users/quantum/Downloads/Material_APESS_2026/x.pdf\"\n\
date: 2026-07-27\ntype: lecture\n---\n# Agentic Design\n",
);
assert_eq!(
note.declared_source_id, None,
"a Downloads path is provenance, not a citable source id"
);
assert_eq!(note.title.as_deref(), Some("Agentic Design"));
assert_eq!(note.source_id(), "note:50 APESS 2026/Lectures/talk.md");
}
#[test]
fn declared_identities_are_scheme_qualified() {
let a = parse_note("p.md", "---\narxiv: 2401.12345\n---\n# T\n");
assert_eq!(a.declared_source_id.as_deref(), Some("arxiv:2401.12345"));
let d = parse_note("p.md", "---\ndoi: 10.1000/xyz\n---\n# T\n");
assert_eq!(d.declared_source_id.as_deref(), Some("doi:10.1000/xyz"));
// Already-qualified values are not double-prefixed.
let s = parse_note("p.md", "---\nsource_id: arxiv:2401.99999\n---\n# T\n");
assert_eq!(s.declared_source_id.as_deref(), Some("arxiv:2401.99999"));
// A URL carries its own scheme and must not become `url:https:...`.
let u = parse_note("p.md", "---\nurl: https://example.com/p\n---\n# T\n");
assert_eq!(
u.declared_source_id.as_deref(),
Some("https://example.com/p")
);
}
/// Repo-sync notes rewrite `updated:`/`size_kb:` on every sync without the
/// prose changing. Hashing the whole file would report 103 phantom edits
/// per run and make "unchanged" meaningless.
#[test]
fn frontmatter_churn_does_not_count_as_an_edit() {
let a = parse_note("Repos/x.md", "---\nupdated: 2026-08-01\nsize_kb: 12\n---\n# X\n\nbody\n");
let b = parse_note("Repos/x.md", "---\nupdated: 2026-08-03\nsize_kb: 14\n---\n# X\n\nbody\n");
assert_eq!(a.content_hash, b.content_hash);
let c = parse_note("Repos/x.md", "---\nupdated: 2026-08-03\n---\n# X\n\nDIFFERENT\n");
assert_ne!(a.content_hash, c.content_hash, "real edits must be visible");
}
#[test]
fn note_identity_is_its_path() {
let n = parse_note("30 Resources/a b.md", "# A\n");
assert_eq!(n.source_id(), "note:30 Resources/a b.md");
}
#[test]
fn collect_skips_dotfiles_and_non_markdown() {
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path();
std::fs::create_dir_all(root.join(".obsidian")).unwrap();
std::fs::create_dir_all(root.join("Daily")).unwrap();
std::fs::write(root.join(".obsidian/workspace.md"), "# editor state\n").unwrap();
std::fs::write(root.join("Daily/note.md"), "# Real\n").unwrap();
std::fs::write(root.join("image.png"), "notmd").unwrap();
let notes = collect_notes(root);
assert_eq!(notes.len(), 1, "only the real note: {notes:?}");
assert_eq!(notes[0].path, "Daily/note.md");
}
}
+83 -2
View File
@@ -64,7 +64,16 @@ const ALLOWED_PROGRAMS: &[&str] = &[
"cargo", "npm", "pnpm", "yarn", "node", "python", "python3", "pytest", "make", "just", "go", "cargo", "npm", "pnpm", "yarn", "node", "python", "python3", "pytest", "make", "just", "go",
"pnpx", "npx", "bun", "dotnet", "mvn", "gradle", "ruff", "mypy", "eslint", "tsc", "jest", "pnpx", "npx", "bun", "dotnet", "mvn", "gradle", "ruff", "mypy", "eslint", "tsc", "jest",
"vitest", "phpunit", "rspec", "bundle", "poetry", "uv", "tox", "vitest", "phpunit", "rspec", "bundle", "poetry", "uv", "tox",
// Version control, narrowed by subcommand. // Security scanners. These ship in the runtime image specifically so a
// `done_when` can be written about them ("gitleaks reports no secrets"),
// and a judge that cannot invoke them has to fall back to asking the
// agents — which is the failure this module exists to prevent. Installing
// them without allow-listing them left exactly that gap.
"gitleaks", "trivy", "semgrep",
// Locate a tool before running it. Cheap, read-only, and it saves the
// judge from concluding a tool is missing when the real answer is that it
// guessed the wrong name.
"which", // Version control, narrowed by subcommand.
"git", "git",
]; ];
@@ -249,11 +258,12 @@ impl Sandbox {
Err(e) => return CheckOutcome::could_not_run(argv, format!("COULD NOT RUN: {e}")), Err(e) => return CheckOutcome::could_not_run(argv, format!("COULD NOT RUN: {e}")),
}; };
let workdir = self.workdir.display().to_string(); let workdir = self.workdir.display().to_string();
let out = crate::container_exec::exec( let out = crate::container_exec::exec_with_env(
&docker, &docker,
&self.container, &self.container,
Some(&workdir), Some(&workdir),
argv, argv,
&git_ownership_env(&workdir),
COMMAND_TIMEOUT, COMMAND_TIMEOUT,
) )
.await; .await;
@@ -289,6 +299,36 @@ impl Sandbox {
} }
} }
/// Let git read a checkout it does not own — including from inside another
/// tool.
///
/// The server clones the mission repo as uid 65532; the runtime container the
/// judge execs into runs as root. Git's ownership check then refuses the
/// repository:
///
/// ```text
/// fatal: detected dubious ownership in repository at '/var/lib/clawmates-missions/<id>/repo'
/// ```
///
/// The first fix rewrote `git` argv to carry `-c safe.directory=…`, which
/// worked for `git status` and did nothing for `gitleaks`, which runs git
/// itself. Observed on mission 019fc073: git reported a clean tree while
/// gitleaks "scanned 0 commits" and the judge — correctly — refused to call
/// the condition met.
///
/// `GIT_CONFIG_COUNT`/`_KEY_n`/`_VALUE_n` is git's documented environment form
/// of `-c`, and it is inherited, so one setting covers git, gitleaks, trivy,
/// semgrep and anything else that shells out. Scoped to this checkout; never
/// `--global`, which would disable the protection container-wide for every
/// path.
fn git_ownership_env(workdir: &str) -> Vec<String> {
vec![
"GIT_CONFIG_COUNT=1".to_string(),
"GIT_CONFIG_KEY_0=safe.directory".to_string(),
format!("GIT_CONFIG_VALUE_0={workdir}"),
]
}
/// One verification command and what became of it. /// One verification command and what became of it.
/// ///
/// This exists because the first version recorded *attempted* commands. The /// This exists because the first version recorded *attempted* commands. The
@@ -366,6 +406,27 @@ mod tests {
} }
} }
/// The scanners exist in the runtime image so conditions can be written
/// about them. Shipping the binaries without allow-listing them left the
/// judge unable to run the very tools installed for it — observed on
/// mission 019fc058, where `gitleaks detect` came back `ran=false` and the
/// judge had to say it could not verify.
#[test]
fn security_scanners_are_runnable() {
for cmd in [
vec!["gitleaks", "detect", "--no-git"],
vec!["trivy", "fs", "."],
vec!["semgrep", "--config=auto"],
vec!["cargo", "audit"],
vec!["which", "gitleaks"],
] {
assert!(
check_argv(&argv(&cmd)).is_ok(),
"{cmd:?} must be runnable — it is installed in the runtime image"
);
}
}
#[test] #[test]
fn refuses_programs_off_the_list() { fn refuses_programs_off_the_list() {
assert_eq!( assert_eq!(
@@ -440,6 +501,26 @@ mod tests {
assert!(check_argv(&argv(&["bash", "-c", "ls"])).is_err()); assert!(check_argv(&argv(&["bash", "-c", "ls"])).is_err());
} }
/// The exception must reach tools that invoke git internally, not just
/// `git` itself — the first version rewrote argv and left gitleaks
/// scanning 0 commits.
#[test]
fn git_ownership_is_set_by_environment_so_subprocesses_inherit_it() {
let env = git_ownership_env("/missions/abc/repo");
assert_eq!(
env,
vec![
"GIT_CONFIG_COUNT=1".to_string(),
"GIT_CONFIG_KEY_0=safe.directory".to_string(),
"GIT_CONFIG_VALUE_0=/missions/abc/repo".to_string(),
]
);
// Scoped to the one checkout. `--global`, or a bare `*`, would switch
// the protection off for every path in the container.
assert!(!env.iter().any(|e| e.contains('*')));
assert!(!env.iter().any(|e| e.contains("--global")));
}
// ── What a check may claim about itself ──────────────────────────── // ── What a check may claim about itself ────────────────────────────
/// The property the whole struct exists for. A refused command and an /// The property the whole struct exists for. A refused command and an
+227
View File
@@ -0,0 +1,227 @@
//! One run of the library: find, skip what we have, shelve the rest.
//!
//! This is the piece that makes the others a *job* rather than parts on a
//! bench. Order matters and it is deliberate:
//!
//! 1. **search** arXiv for candidates
//! 2. **skip** everything already on the checkmark list — before any download
//! 3. **fetch** the PDF for what is left, and verify it really is a PDF
//! 4. **shelve** it in the blob store
//! 5. **catalogue** it: write the vault note
//! 6. **check it off** so next week skips it
//!
//! Step 2 comes before step 3 on purpose. Checking after downloading would
//! still dedupe the catalogue, but it would re-download every paper we already
//! have, every week, forever — and the whole point of the checkmark list is to
//! not do the work twice.
//!
//! # Nothing new is a success, not a failure
//!
//! A weekly run that finds no new papers has worked correctly. A run that
//! *crashed* has not. [`Harvest`] keeps those apart, because collapsing them
//! is precisely the "reported success while doing nothing" shape that this
//! codebase has been bitten by repeatedly. `shelved == 0` with `failed.empty()`
//! is a quiet week; `shelved == 0` with failures is a broken run.
use std::path::Path;
use std::sync::Arc;
use uuid::Uuid;
use crate::corpus;
use crate::papers::{self, Paper};
/// What one run did. Every number here is observed, not claimed.
#[derive(Debug, Default, Clone)]
pub struct Harvest {
/// Papers the search returned.
pub candidates: usize,
/// Of those, how many were already on the checkmark list.
pub already_had: usize,
/// Successfully downloaded, shelved and catalogued.
pub shelved: Vec<String>,
/// `(source_id, why)` for each paper that could not be shelved.
pub failed: Vec<(String, String)>,
/// Vault-relative paths of the notes written.
pub notes_written: Vec<String>,
}
impl Harvest {
/// Did this run add anything? The verification predicate for a continuous
/// research mission: a run that contributes no new source has produced
/// nothing, whatever its transcript says.
pub fn added_anything(&self) -> bool {
!self.shelved.is_empty()
}
/// A run is healthy if nothing errored — including a run that found
/// nothing new, which is the normal state of a mature library.
pub fn healthy(&self) -> bool {
self.failed.is_empty()
}
pub fn summary(&self) -> String {
format!(
"{} candidates, {} already held, {} shelved, {} failed",
self.candidates,
self.already_had,
self.shelved.len(),
self.failed.len()
)
}
}
/// Where a library lives: its records, its shelf, and its catalogue.
///
/// Grouped rather than passed as loose arguments because these five always
/// travel together and always describe one library — splitting them at a call
/// site is how a run ends up shelving into one place and cataloguing into
/// another.
pub struct Library<'a> {
pub pool: &'a sqlx::PgPool,
/// The shelf: where PDFs are stored.
pub blobs: &'a Arc<dyn cm_files::BlobStore>,
pub workspace_id: Uuid,
/// Which checkmark list, e.g. `"valhalla-vault"`.
pub corpus_id: &'a str,
/// Checkout the catalogue notes are written into.
pub vault_root: &'a Path,
}
/// Shelve a specific set of papers. Split from [`run`] so the skip/shelve
/// logic is testable without reaching arXiv.
pub async fn shelve(
lib: &Library<'_>,
candidates: &[Paper],
mission_id: Option<Uuid>,
) -> Result<Harvest, String> {
let Library { pool, blobs, workspace_id, corpus_id, vault_root } = *lib;
let mut out = Harvest {
candidates: candidates.len(),
..Default::default()
};
// One round trip for the whole batch rather than one query per paper.
let ids: Vec<String> = candidates.iter().map(Paper::source_id).collect();
let fresh: std::collections::HashSet<String> =
corpus::unseen(pool, workspace_id, corpus_id, &ids)
.await?
.into_iter()
.collect();
out.already_had = candidates.len() - fresh.len();
for paper in candidates {
let sid = paper.source_id();
if !fresh.contains(&sid) {
continue;
}
// Fetch first. If the PDF cannot be had, nothing is recorded — the
// paper stays unseen so a later run retries it, rather than being
// checked off with an empty shelf slot behind it.
let bytes = match papers::fetch_pdf(paper).await {
Ok(b) => b,
Err(e) => {
out.failed.push((sid, e));
continue;
}
};
let key = paper.blob_key();
if let Err(e) = blobs.put(&key, &bytes).await {
out.failed.push((sid, format!("shelve {key}: {e}")));
continue;
}
// Catalogue note next to the shelf. Written into the vault checkout;
// committing and pushing it is the caller's job, through the delivery
// path that already exists.
let note = papers::catalogue_note(paper, &key);
let note_path = vault_root.join(paper.note_path());
if let Some(parent) = note_path.parent() {
if let Err(e) = std::fs::create_dir_all(parent) {
out.failed.push((sid, format!("create {}: {e}", parent.display())));
continue;
}
}
if let Err(e) = std::fs::write(&note_path, &note) {
out.failed
.push((sid, format!("write {}: {e}", note_path.display())));
continue;
}
// Check it off LAST. If anything above failed we did not get the
// paper, and marking it seen would mean never trying again.
corpus::record(
pool,
workspace_id,
corpus_id,
"source",
&sid,
Some(&paper.title),
Some(&paper.note_path()),
Some(&format!("https://arxiv.org/abs/{}", paper.arxiv_id)),
&corpus::content_hash(&note),
mission_id,
)
.await?;
out.notes_written.push(paper.note_path());
out.shelved.push(sid);
}
Ok(out)
}
/// A full run: search arXiv, then shelve whatever is new.
pub async fn run(
lib: &Library<'_>,
query: &str,
limit: usize,
mission_id: Option<Uuid>,
) -> Result<Harvest, String> {
let candidates = papers::search(query, limit).await?;
let harvest = shelve(lib, &candidates, mission_id).await?;
let corpus_id = lib.corpus_id;
eprintln!("harvest[{corpus_id}] query={query:?}{}", harvest.summary());
for (sid, why) in &harvest.failed {
eprintln!("harvest[{corpus_id}] FAILED {sid}: {why}");
}
Ok(harvest)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_quiet_week_is_healthy_but_adds_nothing() {
let quiet = Harvest {
candidates: 5,
already_had: 5,
..Default::default()
};
assert!(quiet.healthy(), "finding nothing new is not an error");
assert!(
!quiet.added_anything(),
"but it must not count as having produced something"
);
let broken = Harvest {
candidates: 5,
already_had: 0,
failed: vec![("arxiv:1".into(), "timeout".into())],
..Default::default()
};
assert!(!broken.healthy());
assert!(!broken.added_anything());
let good = Harvest {
candidates: 5,
already_had: 4,
shelved: vec!["arxiv:2".into()],
..Default::default()
};
assert!(good.healthy() && good.added_anything());
}
}
+22
View File
@@ -16,6 +16,16 @@ mod mcp_door;
mod mcp_skills; mod mcp_skills;
pub mod mission_orchestrator; pub mod mission_orchestrator;
pub mod mission_refiner; pub mod mission_refiner;
pub mod auto_merge;
pub mod corpus;
pub mod harvest;
pub mod library;
pub mod mission_delivery;
pub mod mission_fs;
pub mod papers;
pub mod phase_config;
pub mod session_executor;
pub mod runtime_preflight;
pub mod mission_runtime; pub mod mission_runtime;
pub mod mission_workspace; pub mod mission_workspace;
pub mod node_rules; pub mod node_rules;
@@ -61,6 +71,9 @@ pub struct AppState {
pub file_root: Option<std::path::PathBuf>, pub file_root: Option<std::path::PathBuf>,
/// Live control channels to connected fleet-node daemons. /// Live control channels to connected fleet-node daemons.
pub node_hub: std::sync::Arc<fleet::NodeHub>, pub node_hub: std::sync::Arc<fleet::NodeHub>,
/// The shelf. Present once the server wires storage; `None` in the
/// bare-`new` path used by tests that never touch blobs.
pub blobs: Option<std::sync::Arc<dyn cm_files::BlobStore>>,
} }
impl AppState { impl AppState {
@@ -75,6 +88,7 @@ impl AppState {
billing: cm_config::BillingConfig::default(), billing: cm_config::BillingConfig::default(),
file_root: None, file_root: None,
node_hub: std::sync::Arc::new(fleet::NodeHub::new()), node_hub: std::sync::Arc::new(fleet::NodeHub::new()),
blobs: None,
} }
} }
@@ -83,6 +97,12 @@ impl AppState {
self self
} }
/// The shelf — where the paper library stores PDFs.
pub fn with_blobs(mut self, blobs: std::sync::Arc<dyn cm_files::BlobStore>) -> AppState {
self.blobs = Some(blobs);
self
}
pub fn with_oauth(mut self, oauth: cm_config::OAuthConfig) -> AppState { pub fn with_oauth(mut self, oauth: cm_config::OAuthConfig) -> AppState {
self.oauth = oauth; self.oauth = oauth;
self self
@@ -308,6 +328,8 @@ pub fn router(state: AppState) -> Router {
.route("/api/sessions", post(routes::sessions::create)) .route("/api/sessions", post(routes::sessions::create))
.route("/api/sessions/history", get(routes::sessions::history)) .route("/api/sessions/history", get(routes::sessions::history))
.route("/api/gateway", post(routes::gateway::gateway)) .route("/api/gateway", post(routes::gateway::gateway))
.route("/api/library/runs", post(routes::library::run))
.route("/api/library/items", get(routes::library::list))
.route("/api/routines", get(routes::routines::list)) .route("/api/routines", get(routes::routines::list))
.route("/api/routines", post(routes::routines::create)) .route("/api/routines", post(routes::routines::create))
.route("/api/routines/runs", get(routes::routines::runs)) .route("/api/routines/runs", get(routes::routines::runs))
+295
View File
@@ -0,0 +1,295 @@
//! A library run end to end: clone the vault, harvest, push the catalogue.
//!
//! [`harvest`](crate::harvest) writes catalogue notes into a directory. This
//! puts that directory somewhere real: a checkout of the vault repo, with the
//! new notes committed and pushed.
//!
//! # Never `main`
//!
//! The vault is a live Obsidian vault that a human edits and syncs. Pushing
//! straight to `main` races that sync and can lose hand-written work. Every
//! run lands on its own branch, exactly like the mission delivery path that
//! was validated 20/20 earlier — a human merges when they have looked at it.
//!
//! # The PDFs do not go here
//!
//! Only notes are committed. PDFs are shelved in the blob store, because a
//! few hundred papers is gigabytes and a vault that size is painful to clone
//! and slow to open. The note carries the blob key, so the catalogue always
//! knows where its shelf is.
use std::path::{Path, PathBuf};
use std::sync::Arc;
use uuid::Uuid;
use crate::harvest::{self, Harvest, Library};
use crate::mission_workspace;
/// What a full run produced, including whether it reached the forge.
#[derive(Debug, Clone)]
pub struct LibraryRun {
pub harvest: Harvest,
pub branch: String,
/// `true` only when the push was observed to succeed. A run that shelved
/// papers but could not push still has the PDFs and the checkmarks; the
/// notes are simply not on the forge yet.
pub pushed: bool,
/// Whether the branch was auto-merged into `main`.
pub merged: bool,
/// Always populated — a branch that quietly did not merge is
/// indistinguishable from one that was never delivered.
pub merge_reason: String,
pub error: Option<String>,
}
fn git_identity() -> [(&'static str, String); 4] {
let (name, email) = crate::mission_delivery::commit_identity();
[
("GIT_AUTHOR_NAME", name.clone()),
("GIT_AUTHOR_EMAIL", email.clone()),
("GIT_COMMITTER_NAME", name),
("GIT_COMMITTER_EMAIL", email),
]
}
async fn git(repo: &Path, args: &[&str]) -> Result<String, String> {
let mut cmd = tokio::process::Command::new("git");
cmd.arg("-C").arg(repo);
cmd.args(["-c", &format!("safe.directory={}", repo.display())]);
cmd.args(args);
for (k, v) in git_identity() {
cmd.env(k, v);
}
let out = cmd.output().await.map_err(|e| format!("spawn git: {e}"))?;
if !out.status.success() {
return Err(format!(
"git {}{}: {}",
args.first().copied().unwrap_or("?"),
out.status,
mission_workspace::redact_token(&String::from_utf8_lossy(&out.stderr))
.chars()
.take(300)
.collect::<String>()
));
}
Ok(String::from_utf8_lossy(&out.stdout).into_owned())
}
/// Clone the vault fresh into `work_root`, returning the checkout path.
///
/// Fresh each run rather than reused: a library run is short, the vault is
/// small (measured 6.9 MB / 416 notes), and a stale checkout is how the
/// mission path lost work three times this week.
pub async fn clone_vault(clone_url: &str, work_root: &Path) -> Result<PathBuf, String> {
let path = work_root.join("vault");
if path.exists() {
tokio::fs::remove_dir_all(&path)
.await
.map_err(|e| format!("clear {}: {e}", path.display()))?;
}
tokio::fs::create_dir_all(work_root)
.await
.map_err(|e| format!("mkdir {}: {e}", work_root.display()))?;
let auth = mission_workspace::with_ambient_auth(clone_url);
let out = tokio::process::Command::new("git")
.args(["clone", "--quiet", "--depth", "1", &auth])
.arg(&path)
.output()
.await
.map_err(|e| format!("spawn git clone: {e}"))?;
if !out.status.success() {
return Err(format!(
"clone vault → {}: {}",
out.status,
mission_workspace::redact_token(&String::from_utf8_lossy(&out.stderr))
.chars()
.take(300)
.collect::<String>()
));
}
// The token must not stay in .git/config: the checkout may be handed to a
// container later, and a credential in a file an agent can read is a
// credential an agent has.
mission_workspace::scrub_remote_credentials(&path, &auth);
Ok(path)
}
/// One complete library run.
#[allow(clippy::too_many_arguments)]
pub async fn run_to_vault(
pool: &sqlx::PgPool,
blobs: &Arc<dyn cm_files::BlobStore>,
workspace_id: Uuid,
corpus_id: &str,
clone_url: &str,
work_root: &Path,
queries: &[String],
per_query: usize,
mission_id: Option<Uuid>,
) -> Result<LibraryRun, String> {
let vault = clone_vault(clone_url, work_root).await?;
let lib = Library {
pool,
blobs,
workspace_id,
corpus_id,
vault_root: &vault,
};
// Accumulate across queries. Topics overlap — "agentic topology" and
// "multi-agent orchestration" return some of the same papers — and the
// checkmark list dedupes across them within a single run as well as
// between runs, because each shelve records before the next query starts.
let mut total = Harvest::default();
for q in queries {
let h = harvest::run(&lib, q, per_query, mission_id).await?;
total.candidates += h.candidates;
total.already_had += h.already_had;
total.shelved.extend(h.shelved);
total.failed.extend(h.failed);
total.notes_written.extend(h.notes_written);
}
// The TAIL of the uuid, not the head. UUIDv7 leads with a 48-bit
// timestamp, so two ids minted in the same millisecond share their first
// 12 hex characters exactly — the branch-name collision that hit mission
// 019fc42b earlier. The tail is the random part.
let branch = format!("clawmates/library-{}", branch_suffix(Uuid::now_v7()));
if total.notes_written.is_empty() {
// A quiet run is a success with nothing to push. Creating an empty
// branch every week would be noise.
return Ok(LibraryRun {
harvest: total,
branch,
pushed: false,
merged: false,
merge_reason: "nothing new to push".into(),
error: None,
});
}
git(&vault, &["checkout", "-B", &branch]).await?;
git(&vault, &["add", "--", "60 Papers"]).await?;
let message = format!(
"library: {} new paper(s)\n\n{}\n\nShelved in the blob store; this commit is the catalogue.",
total.shelved.len(),
total
.shelved
.iter()
.map(|s| format!("- {s}"))
.collect::<Vec<_>>()
.join("\n")
);
git(&vault, &["commit", "--no-verify", "-m", &message]).await?;
let auth = mission_workspace::with_ambient_auth(clone_url);
let refspec = format!("HEAD:refs/heads/{branch}");
match git(&vault, &["push", &auth, &refspec]).await {
Ok(_) => {
// A catalogue branch only ever adds notes under `60 Papers/`, so
// it qualifies for auto-merge — but the check is measured from the
// diff, not assumed from the mission type. Verified here means the
// run shelved something and errored on nothing.
let verified = total.healthy() && !total.shelved.is_empty();
let merge = crate::auto_merge::try_merge(
&vault,
&auth,
&branch,
"main",
crate::auto_merge::MergePolicy::AdditiveOnly,
verified,
)
.await
.unwrap_or_else(|e| crate::auto_merge::MergeOutcome {
merged: false,
reason: format!("merge attempt failed: {e}"),
});
eprintln!("library: branch {branch}{}", merge.reason);
Ok(LibraryRun {
harvest: total,
branch,
pushed: true,
merged: merge.merged,
merge_reason: merge.reason,
error: None,
})
}
Err(e) => Ok(LibraryRun {
harvest: total,
branch,
pushed: false,
merged: false,
merge_reason: "not pushed, so not merged".into(),
error: Some(e),
}),
}
}
/// Distinct-per-run branch suffix. See the note at the call site: taking the
/// head of a UUIDv7 yields the timestamp, which collides.
fn branch_suffix(id: Uuid) -> String {
let s = id.simple().to_string();
s[s.len() - 12..].to_string()
}
/// The topics this library currently tracks.
///
/// Drawn from what the project is actually working on: `papers/dynamic-
/// agentic-topologies.md` (topology search and evolution, citing ADAS,
/// Darwin-Gödel and SwarmAgentic), plus the problems this week's work ran
/// into — verifying what an agent actually did, and giving a long-running
/// agent memory of what it has already covered.
pub fn default_topics() -> Vec<String> {
[
"all:\"agentic topology\" OR all:\"multi-agent topology\"",
"all:\"multi-agent orchestration\" AND all:LLM",
"all:\"agent memory\" AND all:\"long-term\"",
"all:\"LLM agent\" AND all:verification",
"all:\"prompt injection\" AND all:agent",
]
.iter()
.map(|s| s.to_string())
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn topics_are_non_empty_and_arxiv_shaped() {
let topics = default_topics();
assert!(topics.len() >= 3);
for t in &topics {
assert!(t.contains("all:"), "arXiv field prefix missing in {t:?}");
assert!(!t.trim().is_empty());
}
}
/// Two runs in the same millisecond must not collide.
///
/// This caught a real repeat of the mission-path bug (019fc42b): UUIDv7
/// leads with a 48-bit timestamp, so the FIRST 12 hex characters of two
/// ids minted together are identical. Taking the tail fixes it. Looping
/// rather than sampling twice, because a one-shot check passes by luck
/// whenever the millisecond happens to tick between the two calls.
#[test]
fn every_run_gets_a_distinct_branch() {
let ids: Vec<String> = (0..100).map(|_| branch_suffix(Uuid::now_v7())).collect();
let unique: std::collections::HashSet<&String> = ids.iter().collect();
assert_eq!(unique.len(), ids.len(), "branch suffixes collided: {ids:?}");
// And the head-based scheme really does collide, so this test has teeth.
let heads: Vec<String> = (0..100)
.map(|_| Uuid::now_v7().simple().to_string()[..12].to_string())
.collect();
let head_unique: std::collections::HashSet<&String> = heads.iter().collect();
assert!(
head_unique.len() < heads.len(),
"the head of a UUIDv7 was expected to collide but did not"
);
}
}
File diff suppressed because it is too large Load Diff
+273
View File
@@ -0,0 +1,273 @@
//! Move a mission's checkout in and out of its container, instead of sharing it.
//!
//! Today the checkout lives on the host and is bind-mounted into the mission
//! container. That single directory is written by **two users** — cm-api as
//! uid 65532 and the agent as root — and every bug that pattern can produce,
//! it has produced:
//!
//! | Symptom | Fix that was needed |
//! |---|---|
//! | `.git/objects` permission denied | `core.sharedRepository=0777` |
//! | capture base overwritten each phase | advance the base after commit |
//! | `.git/COMMIT_EDITMSG` root-owned | unlink before commit |
//! | `reset --hard` deleting a prior phase | `.git/clawmates-in-use` marker |
//!
//! Four fixes, one cause. `core.sharedRepository` was never a general
//! solution — it covers objects and refs, and every *other* file git touches
//! is a fresh opportunity.
//!
//! Copy-in/copy-out removes the cause: the agent owns its filesystem
//! completely, as root, with no other writer. Nothing on the host is shared,
//! so nothing on the host can collide.
//!
//! # Cost
//!
//! Measured on gw-04 against a real 65 MB checkout of this repository:
//! **0.23s in, 0.18s out**. That was the one open risk in the plan — a
//! monorepo copied per phase — and it is not a risk at this size. Measure
//! again before assuming it holds for a repository an order of magnitude
//! larger.
//!
//! No compression: the payload crosses a local Docker socket, so gzip would
//! spend CPU to save nothing.
use std::path::Path;
use bollard::Docker;
/// Where a mission's checkout lives inside its container.
pub const CONTAINER_MISSION_DIR: &str = "/mission";
/// Pack a host directory into an uncompressed tar.
///
/// `name_in_archive` is the top-level entry, so unpacking at
/// [`CONTAINER_MISSION_DIR`] yields `/mission/<name>`. Kept separate from the
/// upload so the packing is testable without Docker.
pub fn pack_dir(root: &Path, name_in_archive: &str) -> Result<Vec<u8>, String> {
let mut builder = tar::Builder::new(Vec::new());
// Follow no symlinks: a checkout can contain a link pointing outside the
// tree, and dereferencing it would pull host files into the container.
builder.follow_symlinks(false);
builder
.append_dir_all(name_in_archive, root)
.map_err(|e| format!("pack {}: {e}", root.display()))?;
builder
.into_inner()
.map_err(|e| format!("finish archive for {}: {e}", root.display()))
}
/// Unpack a tar into a host directory.
///
/// `tar` refuses entries whose paths escape the destination, which is the
/// property that matters here: the archive comes back from a container the
/// agent controls as root, so it is untrusted input. A `../../etc` entry must
/// not be able to write outside the collection directory.
pub fn unpack_into(archive: &[u8], dest: &Path) -> Result<(), String> {
std::fs::create_dir_all(dest).map_err(|e| format!("mkdir {}: {e}", dest.display()))?;
let mut ar = tar::Archive::new(archive);
ar.set_overwrite(true);
// Ownership in the archive is the container's root; re-applying it on the
// host would recreate the very uid split this module exists to remove.
ar.set_preserve_permissions(false);
ar.unpack(dest)
.map_err(|e| format!("unpack into {}: {e}", dest.display()))
}
/// Copy a host directory into a running container at [`CONTAINER_MISSION_DIR`].
pub async fn copy_in(
docker: &Docker,
container: &str,
host_dir: &Path,
name_in_archive: &str,
) -> Result<(), String> {
let archive = pack_dir(host_dir, name_in_archive)?;
let opts = bollard::query_parameters::UploadToContainerOptionsBuilder::default()
.path(CONTAINER_MISSION_DIR)
.build();
docker
.upload_to_container(container, Some(opts), bollard::body_full(archive.into()))
.await
.map_err(|e| format!("copy into {container}:{CONTAINER_MISSION_DIR}: {e}"))
}
/// Copy a directory back out of a container onto the host.
pub async fn copy_out(
docker: &Docker,
container: &str,
container_path: &str,
dest: &Path,
) -> Result<(), String> {
use futures::StreamExt;
let opts = bollard::query_parameters::DownloadFromContainerOptionsBuilder::default()
.path(container_path)
.build();
let mut stream = docker.download_from_container(container, Some(opts));
let mut archive = Vec::new();
while let Some(chunk) = stream.next().await {
let bytes = chunk.map_err(|e| format!("copy out of {container}:{container_path}: {e}"))?;
archive.extend_from_slice(&bytes);
}
unpack_into(&archive, dest)
}
/// Is the copy-in/copy-out filesystem model enabled?
///
/// Opt-in. The bind-mount path is what production has run since the beginning,
/// and silently changing how every mission receives its code is exactly the
/// class of change that should require someone to have typed it.
pub fn copy_mode() -> bool {
matches!(
std::env::var("CLAWMATES_MISSION_FS").as_deref(),
Ok("copy")
)
}
/// Host directory holding a mission's checkout.
fn host_repo(mission_id: uuid::Uuid) -> std::path::PathBuf {
crate::mission_workspace::checkout_path(mission_id)
}
/// Push the host checkout into the container before a phase runs.
///
/// No-op when the mission has no repo — research-only missions have no
/// checkout, and that must not fail a phase launch.
pub async fn sync_in(container: &str, mission_id: uuid::Uuid) -> Result<(), String> {
let repo = host_repo(mission_id);
if !repo.is_dir() {
return Ok(());
}
let docker = crate::container_exec::connect()?;
copy_in(&docker, container, &repo, "repo").await
}
/// Pull the agent's work back onto the host after a phase.
///
/// Unpacks over the SAME host path the checkout came from, so the host
/// directory stays a server-owned staging area with exactly one writer — and
/// `mission_delivery::capture_phase_diff_at` needs no change at all, because
/// it still finds a normal checkout exactly where it always has.
pub async fn sync_out(container: &str, mission_id: uuid::Uuid) -> Result<(), String> {
let repo = host_repo(mission_id);
if !repo.is_dir() {
return Ok(());
}
let parent = repo
.parent()
.ok_or_else(|| format!("{} has no parent", repo.display()))?;
let docker = crate::container_exec::connect()?;
copy_out(&docker, container, "/mission/repo", parent).await
}
#[cfg(test)]
mod tests {
use super::*;
fn seed(root: &Path) {
std::fs::create_dir_all(root.join("src")).unwrap();
std::fs::create_dir_all(root.join(".git")).unwrap();
std::fs::write(root.join("src/lib.rs"), "pub fn x() {}\n").unwrap();
std::fs::write(root.join(".git/HEAD"), "ref: refs/heads/main\n").unwrap();
}
/// A checkout must survive the round trip intact — including `.git`,
/// without which the whole delivery path (diff, commit, push) is dead.
#[test]
fn a_checkout_round_trips_with_its_git_dir() {
let tmp = tempfile::tempdir().unwrap();
let src = tmp.path().join("repo");
seed(&src);
let archive = pack_dir(&src, "repo").unwrap();
let dest = tmp.path().join("out");
unpack_into(&archive, &dest).unwrap();
assert_eq!(
std::fs::read_to_string(dest.join("repo/src/lib.rs")).unwrap(),
"pub fn x() {}\n"
);
assert!(
dest.join("repo/.git/HEAD").exists(),
"the .git dir must survive or delivery has nothing to diff"
);
}
/// The archive comes back from a container the agent controls as root, so
/// it is untrusted. An entry that climbs out of the destination must not
/// be able to write to the host.
#[test]
fn an_archive_cannot_escape_the_destination() {
let tmp = tempfile::tempdir().unwrap();
let dest = tmp.path().join("dest");
let canary = tmp.path().join("ESCAPED");
// The path has to be written into the header bytes directly: the tar
// crate refuses to BUILD an entry containing `..`, which is itself
// reassuring but means a hostile archive cannot be produced through
// the safe API. A real attacker writes the bytes, so the test does.
let body = b"pwned\n";
let mut header = tar::Header::new_gnu();
header.set_size(body.len() as u64);
header.set_mode(0o644);
header.set_entry_type(tar::EntryType::Regular);
{
let gnu = header.as_gnu_mut().expect("gnu header");
let evil = b"../ESCAPED";
gnu.name[..evil.len()].copy_from_slice(evil);
}
header.set_cksum();
let mut archive = Vec::new();
archive.extend_from_slice(header.as_bytes());
let mut block = [0u8; 512];
block[..body.len()].copy_from_slice(body);
archive.extend_from_slice(&block);
archive.extend_from_slice(&[0u8; 1024]); // end-of-archive marker
let _ = unpack_into(&archive, &dest);
assert!(
!canary.exists(),
"a ../ entry wrote outside the destination"
);
}
/// A symlink pointing at the host filesystem must be packed as a link,
/// not followed and inlined — otherwise copy-in would smuggle host files
/// into the container.
#[test]
fn symlinks_are_not_dereferenced_into_the_archive() {
let tmp = tempfile::tempdir().unwrap();
let src = tmp.path().join("repo");
seed(&src);
let secret = tmp.path().join("host-secret");
std::fs::write(&secret, "TOP SECRET\n").unwrap();
std::os::unix::fs::symlink(&secret, src.join("link")).unwrap();
let archive = pack_dir(&src, "repo").unwrap();
let haystack = String::from_utf8_lossy(&archive);
assert!(
!haystack.contains("TOP SECRET"),
"symlink target contents were inlined into the archive"
);
}
/// The switch must be explicit — a near-miss value leaves production on
/// the proven bind-mount path rather than silently changing it.
#[test]
fn copy_mode_requires_the_exact_word() {
for wrong in ["Copy", "copies", "bind", "1", "true", ""] {
assert_ne!(wrong, "copy", "{wrong:?} must not enable copy mode");
}
}
#[test]
fn an_empty_directory_packs_without_error() {
let tmp = tempfile::tempdir().unwrap();
let src = tmp.path().join("empty");
std::fs::create_dir_all(&src).unwrap();
let archive = pack_dir(&src, "repo").unwrap();
let dest = tmp.path().join("out");
unpack_into(&archive, &dest).unwrap();
assert!(dest.join("repo").is_dir());
}
}
+355 -16
View File
@@ -83,10 +83,28 @@ impl RuntimeAuth {
/// ///
/// The other three are unrelated providers with no subscription equivalent, so /// The other three are unrelated providers with no subscription equivalent, so
/// they forward in both modes. /// they forward in both modes.
///
/// In subscription mode `CLAUDE_CODE_OAUTH_TOKEN` forwards instead. The
/// original design assumed a persisted `claude /login` under a bind-mounted
/// `$HOME`, but a *mission* container gets its own data dir and therefore no
/// login — so the token has to travel. Missing it is not a loud failure:
/// `claude -p` simply hangs with no credential, which is what a phase stuck
/// at `running` for ten minutes looked like when this was first switched on.
pub fn forwarded_provider_keys(auth: RuntimeAuth) -> Vec<&'static str> { pub fn forwarded_provider_keys(auth: RuntimeAuth) -> Vec<&'static str> {
let mut keys = vec!["GEMINI_API_KEY", "GROQ_API_KEY", "OPENAI_API_KEY"]; // ZAI/KIMI reach their backends through the SAME `claude` binary via
if auth == RuntimeAuth::ApiKey { // ANTHROPIC_BASE_URL, so a mission that selects one needs its key present
keys.push("ANTHROPIC_API_KEY"); // in the container. They are unrelated to the Anthropic credential and
// forward in both auth modes.
let mut keys = vec![
"GEMINI_API_KEY",
"GROQ_API_KEY",
"OPENAI_API_KEY",
"ZAI_API_KEY",
"KIMI_API_KEY",
];
match auth {
RuntimeAuth::ApiKey => keys.push("ANTHROPIC_API_KEY"),
RuntimeAuth::Subscription => keys.push("CLAUDE_CODE_OAUTH_TOKEN"),
} }
keys keys
} }
@@ -142,6 +160,146 @@ const MISSIONS_HOST_ROOT: &str = "/var/lib/clawmates-missions";
/// mission so this rarely bites. Long-term: copy-on-write per mission. /// mission so this rarely bites. Long-term: copy-on-write per mission.
const DEFAULT_SEED_DIR: &str = "/root/clawmates-runtime/data"; const DEFAULT_SEED_DIR: &str = "/root/clawmates-runtime/data";
/// What a mission gets its own copy of.
///
/// An allow-list, not the whole directory. The seed dir is **1.7 GB** on
/// gw-04 and 1.5 GB of that is a vestigial `.rustup` — a Rust toolchain that
/// installed itself into the data dir back when `HOME=/zeroclaw-data` and the
/// image had no toolchain. The image now ships Rust at `/usr/local/cargo`,
/// which is what the container's PATH actually resolves (verified live), so
/// that copy is dead weight. Copying it per mission would cost tens of
/// seconds and ~17 GB across ten concurrent missions.
///
/// So: copy what carries per-mission identity or secrets, and leave the
/// caches and toolchains behind.
const SEEDED_PATHS: &[&str] = &[
// The whole point: config.toml carries the §15 door bearer token, and
// data/ holds sessions.db + devices.db. ~26 MB.
".zeroclaw",
// Door MCP config — also a bearer token.
"clawmates-mcp.json",
// Claude Code's own state and credentials (~16 MB). Per-mission so a
// token refresh or project state in one mission cannot leak into another.
".claude",
".claude.json",
// Per-CLI state for the alternate backends; small.
".kimi-code",
"glm-home",
// The seeded agent library.
"agents",
];
/// Deliberately NOT copied — caches and toolchains, no secrets, expensive:
/// `.rustup` (1.5 GB, vestigial), `.npm` (85 MB), `.cargo`, `.cache`,
/// `.local`. A mission that needs them reads the image's copies.
fn copy_script() -> String {
let mut out = String::from("set -e\n");
for p in SEEDED_PATHS {
// Missing entries are normal — a fresh deployment has no .kimi-code
// until Kimi is first used — so absence must not fail the copy.
out.push_str(&format!(
"if [ -e '/seed/{p}' ]; then cp -a '/seed/{p}' /dst/; fi\n"
));
}
out
}
/// Give this mission its own copy of the runtime seed data.
///
/// Every per-mission container used to bind-mount the SAME host seed dir as
/// `/zeroclaw-data` — shared with each other and with the singleton runtime.
/// That directory holds `config.toml`, which carries the §15 door bearer
/// token, plus `sessions.db` and `devices.db`. So a mission could read another
/// mission's credential, and anything it wrote there was inherited by every
/// later mission. Teardown never cleaned it, because teardown only removes
/// `/var/lib/clawmates-missions/{id}`.
///
/// The code already knew: the comment on `DEFAULT_SEED_DIR` names the sqlite
/// race and calls copy-on-write per mission the long-term fix. This is that.
///
/// The copy runs in a throwaway container because cm-api cannot see the seed
/// dir — it hands that host path to Docker but never mounts it itself. The
/// runtime image is reused so nothing extra is pulled.
///
/// Failure is fatal to container creation on purpose. Falling back to the
/// shared mount would silently restore the credential-sharing this removes,
/// and a silent fallback to a weaker posture is the failure mode this
/// codebase keeps paying for.
async fn seed_runtime_data(
docker: &Docker,
image: &str,
seed_dir: &str,
dest_dir: &str,
) -> Result<(), String> {
let name = format!("cm-seed-{}", Uuid::now_v7().simple());
let config = ContainerCreateBody {
image: Some(image.to_string()),
entrypoint: Some(vec!["/bin/sh".to_string()]),
cmd: Some(vec!["-c".to_string(), copy_script()]),
host_config: Some(HostConfig {
mounts: Some(vec![
Mount {
target: Some("/seed".to_string()),
source: Some(seed_dir.to_string()),
typ: Some(MountTypeEnum::BIND),
read_only: Some(true),
..Default::default()
},
Mount {
target: Some("/dst".to_string()),
source: Some(dest_dir.to_string()),
typ: Some(MountTypeEnum::BIND),
read_only: Some(false),
..Default::default()
},
]),
auto_remove: Some(true),
..Default::default()
}),
..Default::default()
};
docker
.create_container(
Some(CreateContainerOptions {
name: Some(name.clone()),
..Default::default()
}),
config,
)
.await
.map_err(|e| format!("create seed copier: {e}"))?;
docker
.start_container(&name, None::<StartContainerOptions>)
.await
.map_err(|e| format!("start seed copier: {e}"))?;
// `auto_remove` means the container disappears the moment it exits, so
// poll for absence rather than waiting on it.
for _ in 0..120 {
tokio::time::sleep(std::time::Duration::from_millis(250)).await;
match docker
.inspect_container(&name, None::<InspectContainerOptions>)
.await
{
Err(_) => return Ok(()),
Ok(info) => {
let running = info
.state
.as_ref()
.and_then(|st| st.running)
.unwrap_or(false);
if !running {
return Ok(());
}
}
}
}
Err(format!("seed copy into {dest_dir} did not finish in 30s"))
}
/// Deterministic docker container name for a mission's runtime. /// Deterministic docker container name for a mission's runtime.
/// Uses the full UUID hex — UUIDv7 encodes time in the leading bytes, /// Uses the full UUID hex — UUIDv7 encodes time in the leading bytes,
/// so a short prefix isn't guaranteed unique across missions minted /// so a short prefix isn't guaranteed unique across missions minted
@@ -239,29 +397,40 @@ impl MissionRuntimeProvisioner {
let _ = tokio::fs::create_dir_all(&mission_dir).await; let _ = tokio::fs::create_dir_all(&mission_dir).await;
let seed_dir = std::env::var("CLAWMATES_RUNTIME_SEED_DIR") let seed_dir = std::env::var("CLAWMATES_RUNTIME_SEED_DIR")
.unwrap_or_else(|_| DEFAULT_SEED_DIR.to_string()); .unwrap_or_else(|_| DEFAULT_SEED_DIR.to_string());
let mounts = vec![ // Per-mission copy of the seed data. See `seed_runtime_data`: sharing
// Mount just this mission's directory. Agents can navigate // one directory meant sharing the door token and letting any mission
// its `/repo` subdir but never see other missions'. // poison every later one.
Mount { let runtime_data_dir = format!("{mission_dir}/runtime-data");
let _ = tokio::fs::create_dir_all(&runtime_data_dir).await;
seed_runtime_data(&self.docker, &self.image, &seed_dir, &runtime_data_dir).await?;
let mut mounts = Vec::new();
// In copy mode the checkout is pushed in and pulled back out, so the
// container gets its OWN filesystem and the host directory has exactly
// one writer (the server). Binding it here would put two uids back on
// one directory — the cause of four separate work-loss bugs.
if !crate::mission_fs::copy_mode() {
mounts.push(Mount {
target: Some("/mission".to_string()), target: Some("/mission".to_string()),
source: Some(mission_dir.clone()), source: Some(mission_dir.clone()),
typ: Some(MountTypeEnum::BIND), typ: Some(MountTypeEnum::BIND),
read_only: Some(false), read_only: Some(false),
..Default::default() ..Default::default()
}, });
// Share the shared-runtime data dir so this gateway inherits }
// the seeded agent library (`claw_*` templates). We then mounts.extend([
// mint a per-mission pairing code via /admin/paircode/new // This mission's OWN copy of the seeded agent library
// below — the mint writes into the shared devices.db but // (`claw_*` templates). Copied rather than shared, so its
// the resulting token is unique to this mission. // config.toml — which carries the door bearer token — and its
// sqlite files belong to this mission alone and are removed with
// it by `teardown_container`.
Mount { Mount {
target: Some("/zeroclaw-data".to_string()), target: Some("/zeroclaw-data".to_string()),
source: Some(seed_dir), source: Some(runtime_data_dir),
typ: Some(MountTypeEnum::BIND), typ: Some(MountTypeEnum::BIND),
read_only: Some(false), read_only: Some(false),
..Default::default() ..Default::default()
}, },
]; ]);
let host_config = HostConfig { let host_config = HostConfig {
mounts: Some(mounts), mounts: Some(mounts),
@@ -285,6 +454,24 @@ impl MissionRuntimeProvisioner {
let mut env = vec![ let mut env = vec![
format!("ZEROCLAW_GATEWAY_PORT={GATEWAY_PORT}"), format!("ZEROCLAW_GATEWAY_PORT={GATEWAY_PORT}"),
format!("CM_MISSION_ID={mission_id}"), format!("CM_MISSION_ID={mission_id}"),
// Let the agents' git read the checkout.
//
// The server clones as uid 65532; this container runs as root, so
// every `git` an agent runs hits "detected dubious ownership" and
// refuses the repository. Agents do not report that as a failure —
// they improvise. On mission 019fc3ba one wrote a `.gitconfig_temp`
// containing `[safe] directory = /mission/repo` into the repository
// root, which then showed up in the captured diff and would have
// been committed and pushed to the user's repo alongside the real
// work.
//
// `GIT_CONFIG_*` is git's environment form of `-c` and is
// inherited by subprocesses, so it covers the agent's own git, any
// tool that shells out to git, and the `git_operations` tool alike.
// Scoped to the checkout; never `--global`.
"GIT_CONFIG_COUNT=1".to_string(),
"GIT_CONFIG_KEY_0=safe.directory".to_string(),
"GIT_CONFIG_VALUE_0=/mission/repo".to_string(),
]; ];
// Provider credentials forwarded into the container. // Provider credentials forwarded into the container.
// //
@@ -699,6 +886,14 @@ async fn sweep_once(pool: &sqlx::PgPool, grace: std::time::Duration) -> Result<(
for row in rows { for row in rows {
let id: Uuid = row.get("id"); let id: Uuid = row.get("id");
let workspace_id: Uuid = row.get("workspace_id"); let workspace_id: Uuid = row.get("workspace_id");
// Last chance. `teardown_container` deletes the checkout, so anything
// not captured by now is gone for good. The phase sweep should have
// handled this minutes ago; this covers the cases it cannot — a phase
// that ended `failed` rather than `completed`, or a capture that kept
// erroring until the grace window ran out.
if let Err(e) = capture_outstanding_phases(pool, id).await {
eprintln!("mission_runtime::sweeper: last-chance capture for {id}: {e}");
}
if let Err(e) = prov.teardown_container(id).await { if let Err(e) = prov.teardown_container(id).await {
// A not-found is expected when the container was already // A not-found is expected when the container was already
// reaped by a docker restart or a manual op; log at info // reaped by a docker restart or a manual op; log at info
@@ -716,6 +911,46 @@ async fn sweep_once(pool: &sqlx::PgPool, grace: std::time::Duration) -> Result<(
Ok(()) Ok(())
} }
/// Capture any phase of `mission_id` that has a repo and no `code_diff` yet,
/// regardless of how the phase ended.
///
/// The phase sweep only captures `completed` phases. A mission that failed
/// mid-coding still has real work in its checkout, and deleting it
/// unexamined is how a debugging session loses the only evidence of what the
/// agents actually did.
async fn capture_outstanding_phases(pool: &sqlx::PgPool, mission_id: Uuid) -> Result<(), String> {
use sqlx::Row;
let rows = sqlx::query(
"SELECT mp.id
FROM mission_phases mp
JOIN missions m ON m.id = mp.mission_id
WHERE mp.mission_id = $1
AND m.repo_id IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM mission_artifacts a
WHERE a.mission_id = mp.mission_id
AND a.phase_id = mp.id
AND a.kind = 'code_diff'
)",
)
.bind(mission_id)
.fetch_all(pool)
.await
.map_err(|e| format!("select uncaptured phases: {e}"))?;
for row in rows {
let phase_id: Uuid = row.get("id");
if let Err(e) =
crate::mission_delivery::capture_phase_diff(pool, mission_id, phase_id).await
{
eprintln!(
"mission_runtime::sweeper: capture mission {mission_id} phase {phase_id}: {e}"
);
}
}
Ok(())
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
@@ -736,9 +971,38 @@ mod tests {
it silently bills the API. Forwarded: {keys:?}" it silently bills the API. Forwarded: {keys:?}"
); );
// Unrelated providers have no subscription equivalent and must survive. // Unrelated providers have no subscription equivalent and must survive.
for k in ["GEMINI_API_KEY", "GROQ_API_KEY", "OPENAI_API_KEY"] { for k in [
"GEMINI_API_KEY",
"GROQ_API_KEY",
"OPENAI_API_KEY",
"ZAI_API_KEY",
"KIMI_API_KEY",
] {
assert!(keys.contains(&k), "{k} should still be forwarded"); assert!(keys.contains(&k), "{k} should still be forwarded");
} }
// And the subscription credential MUST travel. A mission container
// has its own data dir, so unlike the shared runtime it has no
// persisted `claude /login` to fall back on. Without this the CLI
// has no credential and simply hangs — a phase stuck at `running`
// with nothing in the logs, which is exactly how this was found.
assert!(
keys.contains(&"CLAUDE_CODE_OAUTH_TOKEN"),
"subscription mode must forward the token; without it `claude -p` \
hangs with no credential. Forwarded: {keys:?}"
);
}
/// The two credentials must never travel together: Claude Code would pick
/// the API key and bill it while the deployment believes it is on the
/// subscription.
#[test]
fn the_two_anthropic_credentials_are_mutually_exclusive() {
for mode in [RuntimeAuth::ApiKey, RuntimeAuth::Subscription] {
let keys = forwarded_provider_keys(mode);
let both = keys.contains(&"ANTHROPIC_API_KEY")
&& keys.contains(&"CLAUDE_CODE_OAUTH_TOKEN");
assert!(!both, "{mode:?} forwards both credentials: {keys:?}");
}
} }
/// Default behaviour is unchanged, so a deployment that never opts in keeps /// Default behaviour is unchanged, so a deployment that never opts in keeps
@@ -754,6 +1018,10 @@ mod tests {
] { ] {
assert!(keys.contains(&k), "{k} should be forwarded in api_key mode"); assert!(keys.contains(&k), "{k} should be forwarded in api_key mode");
} }
assert!(
!keys.contains(&"CLAUDE_CODE_OAUTH_TOKEN"),
"api_key mode must not also ship the subscription token"
);
} }
/// An unset or misspelled value must fall back to the *existing* behaviour. /// An unset or misspelled value must fall back to the *existing* behaviour.
@@ -873,4 +1141,75 @@ allowed_tools = ["file_read", "file_edit"]
let url = endpoint_url("cm-runtime-mission-abc"); let url = endpoint_url("cm-runtime-mission-abc");
assert_eq!(url, "http://cm-runtime-mission-abc:42617"); assert_eq!(url, "http://cm-runtime-mission-abc:42617");
} }
/// The seed data path must be per-mission, not the shared seed dir.
///
/// Every mission container used to bind the SAME host directory as
/// `/zeroclaw-data`. It holds `config.toml`, which carries the §15 door
/// bearer token, plus `sessions.db`/`devices.db`. Sharing it meant one
/// mission could read another's credential, and anything written there
/// was inherited by every later mission — `teardown_container` only
/// removes `/var/lib/clawmates-missions/{id}`, so the shared dir was
/// never cleaned.
///
/// Asserting the path shape is what keeps this from silently regressing:
/// a future edit that points the mount back at the seed dir restores the
/// credential sharing with no other visible symptom.
#[test]
fn runtime_data_is_scoped_to_one_mission() {
let a = Uuid::now_v7();
let b = Uuid::now_v7();
let path = |id: Uuid| format!("{MISSIONS_HOST_ROOT}/{id}/runtime-data");
assert_ne!(path(a), path(b), "two missions must not share runtime data");
assert!(
path(a).starts_with(&format!("{MISSIONS_HOST_ROOT}/{a}")),
"runtime data must live under the mission dir so teardown removes it"
);
assert_ne!(
path(a),
DEFAULT_SEED_DIR,
"the mount must never be the shared seed dir itself"
);
assert!(
!path(a).starts_with(DEFAULT_SEED_DIR),
"runtime data must not live inside the shared seed dir either"
);
}
/// The copy must be an allow-list, and must include the secret-bearing
/// paths while excluding the expensive ones.
///
/// Measured on gw-04: the seed dir is 1.7 GB, of which 1.5 GB is a
/// vestigial `.rustup` that is not even on the container's PATH (the
/// image ships Rust at /usr/local/cargo). Copying everything per mission
/// would cost tens of seconds and ~17 GB across ten concurrent missions —
/// which is what the first version of this did.
#[test]
fn the_seed_copy_takes_secrets_and_skips_caches() {
// The two paths that carry the door bearer token MUST be copied, or
// this whole change accomplishes nothing.
assert!(SEEDED_PATHS.contains(&".zeroclaw"));
assert!(SEEDED_PATHS.contains(&"clawmates-mcp.json"));
// Claude Code's credentials and state.
assert!(SEEDED_PATHS.contains(&".claude"));
// The expensive, secret-free ones must NOT be.
for cache in [".rustup", ".npm", ".cargo", ".cache"] {
assert!(
!SEEDED_PATHS.contains(&cache),
"{cache} is a cache and must not be copied per mission"
);
}
let script = copy_script();
// A missing entry is normal on a fresh deployment (no .kimi-code
// until Kimi is first used) and must not fail the copy.
assert!(script.contains("if [ -e "), "absent paths must be tolerated");
assert!(script.contains("/seed/.zeroclaw"));
assert!(!script.contains("/seed/.rustup"));
for p in SEEDED_PATHS {
assert!(script.contains(p), "{p} missing from the copy script");
}
}
} }
+661 -13
View File
@@ -23,7 +23,7 @@ use std::path::PathBuf;
use tokio::process::Command; use tokio::process::Command;
use uuid::Uuid; use uuid::Uuid;
fn missions_root() -> PathBuf { pub(crate) fn missions_root() -> PathBuf {
std::env::var("CLAWMATES_MISSIONS_ROOT") std::env::var("CLAWMATES_MISSIONS_ROOT")
.map(PathBuf::from) .map(PathBuf::from)
.unwrap_or_else(|_| PathBuf::from("/var/lib/clawmates-missions")) .unwrap_or_else(|_| PathBuf::from("/var/lib/clawmates-missions"))
@@ -67,7 +67,25 @@ pub async fn ensure_checkout(
let auth_url = with_ambient_auth(clone_url); let auth_url = with_ambient_auth(clone_url);
if path.join(".git").exists() { if path.join(".git").exists() {
fetch_and_reset(&path, default_branch).await?; // Checkouts cloned before this setting existed get it on reuse. It
// governs objects created from now on, which is what delivery needs.
share_repository_across_uids(&path);
// `ensure_checkout` runs at every phase launch, not once per mission.
// Freshening a pristine checkout is right; freshening one that already
// holds this mission's work destroys it. See `has_local_work`.
// Marker first: it is a fact we recorded, not a state we inferred.
// The tree checks stay as a second line of defence for checkouts
// created before the marker existed, and for the case where the
// marker write itself failed.
if checkout_in_use(&path) || has_local_work(&path, default_branch) {
eprintln!(
"mission_workspace: {} already holds mission work — skipping \
fetch/reset so earlier phases' output survives",
path.display()
);
} else {
fetch_and_reset(&path, default_branch, &auth_url).await?;
}
} else { } else {
clone(&path, &auth_url).await?; clone(&path, &auth_url).await?;
} }
@@ -78,7 +96,7 @@ pub async fn ensure_checkout(
/// environment, rewrite it to include the token as basic-auth. Returns /// environment, rewrite it to include the token as basic-auth. Returns
/// the URL unchanged otherwise. The token is never logged (we only /// the URL unchanged otherwise. The token is never logged (we only
/// pass the rewritten URL into `git clone` via argv). /// pass the rewritten URL into `git clone` via argv).
fn with_ambient_auth(url: &str) -> String { pub(crate) fn with_ambient_auth(url: &str) -> String {
let Ok(token) = std::env::var("GITEA_TOKEN") else { let Ok(token) = std::env::var("GITEA_TOKEN") else {
return url.to_string(); return url.to_string();
}; };
@@ -92,8 +110,20 @@ fn with_ambient_auth(url: &str) -> String {
} }
async fn clone(path: &std::path::Path, url: &str) -> Result<(), String> { async fn clone(path: &std::path::Path, url: &str) -> Result<(), String> {
// `--filter=blob:none` rather than `--depth 1`. A shallow clone cannot
// usually push a new branch back ("shallow update not allowed"), and
// mission delivery needs exactly that. A partial clone keeps full history
// — so the base commit stays meaningful and a diff has something to be
// relative to — while fetching file contents only on demand, which is
// nearly as cheap as a shallow clone for a repo that gets read once.
let out = Command::new("git") let out = Command::new("git")
.args(["clone", "--depth", "1", url, &path.display().to_string()]) .args([
"clone",
"--filter=blob:none",
"--single-branch",
url,
&path.display().to_string(),
])
.output() .output()
.await .await
.map_err(|e| format!("spawn git clone: {e}"))?; .map_err(|e| format!("spawn git clone: {e}"))?;
@@ -107,10 +137,365 @@ async fn clone(path: &std::path::Path, url: &str) -> Result<(), String> {
.collect::<String>() .collect::<String>()
)); ));
} }
share_repository_across_uids(path);
scrub_remote_credentials(path, url);
ignore_agent_scaffolding(path);
record_base_commit(path);
Ok(()) Ok(())
} }
fn redact_token(s: &str) -> String { /// Record that a phase has started working in this checkout.
///
/// The explicit half of the "is this checkout in use" question. `ensure_checkout`
/// runs per phase launch and refreshes on reuse; whether that refresh is safe
/// depends on whether a phase has already run here, which is a fact about the
/// *mission* and not about the tree.
///
/// It was previously inferred from the tree — dirty status, HEAD versus the
/// remote tip — and inference is what made delivery depend on what an agent
/// happened to do. Mission `019fc444` lost work because its phase committed and
/// left a clean tree; `019fc476` lost work because the capture base had advanced
/// to match HEAD; `019fc450` survived only because a phase *failed* to commit
/// and left the tree dirty. Same code, opposite outcomes, decided by the agent.
///
/// A marker is not a heuristic. Once a phase has begun, the checkout is in use
/// until the mission ends, whatever the agent did or did not do inside it.
pub(crate) fn mark_phase_started(path: &std::path::Path) {
let marker = path.join(".git/clawmates-in-use");
if marker.exists() {
return;
}
if let Err(e) = std::fs::write(&marker, "1\n") {
eprintln!(
"mission_workspace: could not mark {} as in use ({e}) — a later phase may \
refresh the checkout and discard earlier work",
path.display()
);
}
}
/// Has a phase already started work in this checkout?
fn checkout_in_use(path: &std::path::Path) -> bool {
path.join(".git/clawmates-in-use").exists()
}
/// Has anything happened in this checkout since it was created?
///
/// `ensure_checkout` is called once per *phase launch*, not once per mission,
/// and its reuse path runs `git reset --hard origin/<branch>`. That is correct
/// for a checkout being picked up cold and destructive for one mid-mission:
/// mission `019fc444` had its phase-0 file deleted from the working tree when
/// phase 1 started, so the second phase never saw the first's output.
///
/// Delivery is what made this reachable. Before the mission branch existed,
/// agent output stayed *untracked* and `reset --hard` left it alone. Committing
/// it — the whole point of the delivery slice — makes it tracked, and tracked
/// files that are absent from `origin/<branch>` are exactly what a hard reset
/// removes. The feature that preserves work is what put it in reach of the
/// reset.
///
/// "Local work" is either a commit that is not on the fetched tip, or a dirty
/// tree. Both are checked because the two phases of the failure look different:
/// an agent that committed leaves a clean tree at a new HEAD, and one that did
/// not leaves a dirty tree at the old HEAD.
fn has_local_work(path: &std::path::Path, branch: &str) -> bool {
let git = |args: &[&str]| -> Option<String> {
let out = std::process::Command::new("git")
.arg("-C")
.arg(path)
.args(["-c", &format!("safe.directory={}", path.display())])
.args(args)
.output()
.ok()?;
out.status
.success()
.then(|| String::from_utf8_lossy(&out.stdout).trim().to_string())
};
// A dirty tree is unambiguous: someone is mid-work here.
if let Some(status) = git(&["status", "--porcelain"]) {
if !status.is_empty() {
return true;
}
}
// Otherwise compare HEAD against the *remote tip*, which is the only
// fixed point here.
//
// This deliberately does not use `.git/clawmates-base`. That marker is the
// rolling capture base and `advance_base_commit` moves it to each phase's
// committed head — so comparing HEAD against it asks "did anything happen
// since the last commit we made", which is false immediately after every
// successful delivery. Mission `019fc476` lost phase 0's file exactly that
// way: phase 0 committed, the base advanced to match HEAD, and phase 1's
// launch concluded the checkout was pristine and reset it. The preceding
// mission survived only because its phase 0 *failed* to commit and left a
// dirty tree.
//
// `origin/<branch>` does not move for the life of the mission, so "HEAD is
// not the remote tip" means a phase committed, whether one commit ago or
// five. If the remote ref cannot be resolved the answer is preserve:
// wrongly skipping a refresh costs staleness, wrongly resetting destroys a
// phase's output.
match (
git(&["rev-parse", &format!("origin/{branch}")]),
git(&["rev-parse", "HEAD"]),
) {
(Some(tip), Some(head)) => tip != head,
_ => true,
}
}
/// Let the server and the agent container both write to this checkout.
///
/// The checkout is one directory bind-mounted into two processes running as
/// different users: cm-api is uid 65532, the mission runtime container is
/// root. Git creates `.git/objects/xx/` fan-out directories on first write and
/// they inherit the writer's ownership, so whichever party commits first locks
/// the other out of that directory:
///
/// ```text
/// git add → exit 128: insufficient permission for adding an object
/// to repository database .git/objects
/// ```
///
/// The failure is intermittent, which is what makes it dangerous. Mission
/// `019fc42b` delivered cleanly because its agents committed their own work,
/// so the blobs already existed and the server's `git add` never had to write
/// one. Mission `019fc437` ran the same template, its agents left the work
/// uncommitted, and delivery lost both phases.
///
/// `core.sharedRepository` is git's own answer to a repository shared between
/// users: it makes git create objects and refs group- and world-writable. Both
/// parties read this config from the shared `.git/config`, so it governs the
/// agent's commits as much as ours.
///
/// This grants the agent no access it lacks. It is already root inside a
/// container with the entire checkout bind-mounted read-write, and could
/// rewrite any of it. The party actually gaining something is the server,
/// which is currently the one being locked out.
pub fn share_repository_across_uids(path: &std::path::Path) {
let out = std::process::Command::new("git")
.args([
"-C",
&path.display().to_string(),
"-c",
&format!("safe.directory={}", path.display()),
"config",
"core.sharedRepository",
"0777",
])
.output();
match out {
Ok(o) if o.status.success() => {}
Ok(o) => eprintln!(
"mission_workspace: could not set core.sharedRepository on {} ({}) — delivery \
may fail to commit if the agent writes git objects first",
path.display(),
String::from_utf8_lossy(&o.stderr).trim()
),
Err(e) => eprintln!(
"mission_workspace: could not set core.sharedRepository on {} ({e})",
path.display()
),
}
}
/// Remember the commit the mission started from.
///
/// Delivery needs to answer "what did this mission change", and the obvious
/// reading — working tree versus `HEAD` — is wrong the moment an agent
/// commits. `rust_sdlc` has a *committer* role, so committing is the normal
/// path, not an edge case: a mission that did its job properly would have a
/// clean tree and capture nothing at all. Observed exactly that on mission
/// 019fc372, where the agent committed `DELIVERY_PROBE.md` and the diff came
/// back empty.
///
/// Written into `.git/` so it travels with the checkout, is invisible to the
/// repository, and cannot be edited by an agent through its pinned workspace.
pub(crate) fn record_base_commit(path: &std::path::Path) {
let out = std::process::Command::new("git")
.args([
"-C",
&path.display().to_string(),
"-c",
&format!("safe.directory={}", path.display()),
"rev-parse",
"HEAD",
])
.output();
let Ok(out) = out else { return };
if !out.status.success() {
return;
}
let sha = String::from_utf8_lossy(&out.stdout).trim().to_string();
if sha.is_empty() {
return;
}
if let Err(e) = std::fs::write(path.join(".git/clawmates-base"), format!("{sha}\n")) {
eprintln!(
"mission_workspace: could not record base commit for {} ({e}) — delivery will \
fall back to diffing against HEAD and will miss committed work",
path.display()
);
}
}
/// Move the capture base forward to a commit a phase just produced.
///
/// The base is recorded once at clone time, which is right for the mission's
/// first phase and wrong for every phase after it: a later phase would diff
/// against the original clone point and claim its predecessors' commits as its
/// own work. Mission `019fc42b` showed this plainly — two coding phases, and
/// the second phase's artifact reported the *union* of both phases' files.
///
/// Advancing after each successful commit makes each artifact the incremental
/// work of one phase. The pushed branch stays cumulative, because it is built
/// from `HEAD` and therefore still carries the earlier commits.
pub(crate) fn advance_base_commit(path: &std::path::Path, sha: &str) {
let sha = sha.trim();
if sha.is_empty() {
return;
}
if let Err(e) = std::fs::write(path.join(".git/clawmates-base"), format!("{sha}\n")) {
eprintln!(
"mission_workspace: could not advance base commit for {} ({e}) — the next \
phase will re-report this phase's work as its own",
path.display()
);
}
}
/// The commit this mission's checkout started from, if it was recorded.
pub(crate) fn base_commit(path: &std::path::Path) -> Option<String> {
std::fs::read_to_string(path.join(".git/clawmates-base"))
.ok()
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
}
/// Take the access token back out of `.git/config`.
///
/// `with_ambient_auth` embeds `GITEA_TOKEN` in the clone URL so the clone can
/// authenticate, and git then persists that URL verbatim as the `origin`
/// remote. The checkout is bind-mounted into a container the agents run in as
/// **root**, so the token sits in a file every mission agent can read, and it
/// reaches every repository that token reaches — not just this one.
///
/// Rewriting the remote to the bare URL costs one command and removes a
/// standing credential from the blast radius of any prompt injection that
/// lands in a mission. Delivery does not depend on the stored URL: it builds a
/// fresh authenticated URL at push time, which also means a rotated token
/// starts working immediately instead of after the next clone.
///
/// Best-effort and non-fatal: a checkout that keeps its token still works, and
/// failing the mission over it would trade a real capability for a marginal
/// improvement in a situation we have already logged.
pub(crate) fn scrub_remote_credentials(path: &std::path::Path, original_url: &str) {
if !original_url.contains('@') && !original_url.contains("oauth2:") {
// Nothing was injected (SSH remote, or no token configured).
return;
}
let bare = strip_credentials(original_url);
let out = std::process::Command::new("git")
.args([
"-C",
&path.display().to_string(),
"remote",
"set-url",
"origin",
&bare,
])
.output();
match out {
Ok(o) if o.status.success() => {}
Ok(o) => eprintln!(
"mission_workspace: could not scrub credentials from {} — the access token \
remains readable in .git/config: {}",
path.display(),
redact_token(&String::from_utf8_lossy(&o.stderr))
.chars()
.take(200)
.collect::<String>()
),
Err(e) => eprintln!(
"mission_workspace: could not scrub credentials from {} ({e}) — the access \
token remains readable in .git/config",
path.display()
),
}
}
/// `https://user:secret@host/path` → `https://host/path`.
fn strip_credentials(url: &str) -> String {
let Some((scheme, rest)) = url.split_once("://") else {
return url.to_string();
};
match rest.split_once('@') {
// Only the *authority* may carry credentials; an `@` later in the path
// is an ordinary character and must not be treated as a separator.
Some((userinfo, host_and_path)) if !userinfo.contains('/') => {
format!("{scheme}://{host_and_path}")
}
_ => url.to_string(),
}
}
/// Files the agent runtime writes into its own workspace, which is pinned to
/// the repository root (`MissionRuntimeProvisioner::pin_agent_workspaces`).
///
/// They are the agent's identity scaffolding, not the user's code — `SOUL.md`
/// opens "Who You Are / You're not a chatbot." Observed on mission 019fc058,
/// where all seven appeared as untracked files in a freshly cloned repo.
const AGENT_SCAFFOLDING: &[&str] = &[
"AGENTS.md",
"HEARTBEAT.md",
"IDENTITY.md",
"MEMORY.md",
"SOUL.md",
"TOOLS.md",
"USER.md",
];
/// Keep the agent's own scaffolding out of the user's repository.
///
/// Two things went wrong without this. Every mission's tree was permanently
/// dirty, so a `done_when` written about a clean tree could never pass. And
/// once mission delivery starts committing, `git add -A` would have put the
/// agent's `SOUL.md` and `MEMORY.md` into someone's repository and pushed
/// them.
///
/// Written to `.git/info/exclude` rather than `.gitignore`: the exclude file
/// is local to this checkout and never itself appears as a change, so the
/// repository the user gets back is untouched. Crucially it only suppresses
/// *untracked* files — a repo that genuinely tracks its own `AGENTS.md` still
/// reports modifications to it, which is the behaviour we want.
///
/// Best-effort: a checkout that cannot be annotated is noisier, not broken.
fn ignore_agent_scaffolding(path: &std::path::Path) {
let exclude = path.join(".git/info/exclude");
let mut body = std::fs::read_to_string(&exclude).unwrap_or_default();
if body.contains("clawmates: agent scaffolding") {
return;
}
body.push_str("\n# clawmates: agent scaffolding — written by the runtime into its\n");
body.push_str("# pinned workspace, never part of the repository.\n");
for name in AGENT_SCAFFOLDING {
body.push_str(&format!("/{name}\n"));
}
if let Some(dir) = exclude.parent() {
let _ = std::fs::create_dir_all(dir);
}
if let Err(e) = std::fs::write(&exclude, body) {
eprintln!(
"mission_workspace: could not write {} ({e}) — agent scaffolding will show as \
untracked in this checkout",
exclude.display()
);
}
}
pub(crate) fn redact_token(s: &str) -> String {
// Strip any "oauth2:<token>@" segment that git may echo back on // Strip any "oauth2:<token>@" segment that git may echo back on
// failures. Belt-and-braces: also nuke any raw token env value. // failures. Belt-and-braces: also nuke any raw token env value.
let mut out = s.to_string(); let mut out = s.to_string();
@@ -127,25 +512,60 @@ fn redact_token(s: &str) -> String {
out out
} }
async fn fetch_and_reset(path: &std::path::Path, branch: &str) -> Result<(), String> { async fn fetch_and_reset(
let fetch = Command::new("git") path: &std::path::Path,
branch: &str,
auth_url: &str,
) -> Result<(), String> {
// A checkout cloned before delivery existed is shallow, and a shallow repo
// cannot push a new branch. Deepen it once, here, rather than discovering
// the problem at push time when there is work on the line. `--unshallow`
// errors on a repo that is already complete, so it is only attempted when
// the marker file is present.
if path.join(".git/shallow").exists() {
let deepen = Command::new("git")
.args([ .args([
"-C", "-C",
&path.display().to_string(), &path.display().to_string(),
"fetch", "fetch",
"--depth", "--unshallow",
"1", auth_url,
"origin",
branch,
]) ])
.output() .output()
.await;
match deepen {
Ok(o) if o.status.success() => {}
Ok(o) => eprintln!(
"mission_workspace: could not deepen shallow checkout at {} — a delivery \
push may be rejected: {}",
path.display(),
redact_token(&String::from_utf8_lossy(&o.stderr))
.chars()
.take(200)
.collect::<String>()
),
Err(e) => eprintln!(
"mission_workspace: could not deepen shallow checkout at {} ({e})",
path.display()
),
}
}
// Fetch from an explicitly authenticated URL rather than the stored
// remote. `scrub_remote_credentials` strips the token out of
// `.git/config` — the checkout is readable by agents running as root —
// so `git fetch origin` has no credentials and fails with
// "could not read Username". Building the URL here also means a rotated
// token takes effect immediately instead of at the next clone.
let fetch = Command::new("git")
.args(["-C", &path.display().to_string(), "fetch", auth_url, branch])
.output()
.await .await
.map_err(|e| format!("spawn git fetch: {e}"))?; .map_err(|e| format!("spawn git fetch: {e}"))?;
if !fetch.status.success() { if !fetch.status.success() {
return Err(format!( return Err(format!(
"git fetch origin {branch} → exit {}: {}", "git fetch origin {branch} → exit {}: {}",
fetch.status, fetch.status,
String::from_utf8_lossy(&fetch.stderr) redact_token(&String::from_utf8_lossy(&fetch.stderr))
.chars() .chars()
.take(400) .take(400)
.collect::<String>() .collect::<String>()
@@ -166,11 +586,239 @@ async fn fetch_and_reset(path: &std::path::Path, branch: &str) -> Result<(), Str
return Err(format!( return Err(format!(
"git reset --hard origin/{branch} → exit {}: {}", "git reset --hard origin/{branch} → exit {}: {}",
reset.status, reset.status,
String::from_utf8_lossy(&reset.stderr) redact_token(&String::from_utf8_lossy(&reset.stderr))
.chars() .chars()
.take(400) .take(400)
.collect::<String>() .collect::<String>()
)); ));
} }
// HEAD just moved to the freshly fetched tip; that is this run's starting
// point, so the recorded base moves with it.
record_base_commit(path);
Ok(()) Ok(())
} }
#[cfg(test)]
mod tests {
use super::*;
/// The exclude must be idempotent — `ensure_checkout` re-runs on every
/// phase, and appending the same block each time would grow the file
/// without bound.
#[test]
fn scaffolding_exclusion_is_written_once() {
let dir = tempfile::tempdir().unwrap();
std::fs::create_dir_all(dir.path().join(".git/info")).unwrap();
ignore_agent_scaffolding(dir.path());
let first = std::fs::read_to_string(dir.path().join(".git/info/exclude")).unwrap();
assert!(
first.contains("/SOUL.md"),
"the agent's identity file is excluded"
);
assert!(first.contains("/MEMORY.md"));
ignore_agent_scaffolding(dir.path());
let second = std::fs::read_to_string(dir.path().join(".git/info/exclude")).unwrap();
assert_eq!(first, second, "re-running must not append a second block");
}
#[test]
fn credentials_are_stripped_from_a_remote_url() {
assert_eq!(
strip_credentials("https://oauth2:[email protected]/o/r.git"),
"https://git.redclaw.dev/o/r.git"
);
// No credentials: unchanged.
assert_eq!(
strip_credentials("https://git.redclaw.dev/o/r.git"),
"https://git.redclaw.dev/o/r.git"
);
// SSH form has no `://` authority to rewrite.
assert_eq!(
strip_credentials("[email protected]:o/r.git"),
"[email protected]:o/r.git"
);
// An `@` inside the path is not a credential separator.
assert_eq!(
strip_credentials("https://host/scope/@org/pkg.git"),
"https://host/scope/@org/pkg.git"
);
}
/// An existing exclude file belongs to the repository; keep it.
#[test]
fn an_existing_exclude_is_preserved() {
let dir = tempfile::tempdir().unwrap();
std::fs::create_dir_all(dir.path().join(".git/info")).unwrap();
std::fs::write(dir.path().join(".git/info/exclude"), "/local-scratch\n").unwrap();
ignore_agent_scaffolding(dir.path());
let body = std::fs::read_to_string(dir.path().join(".git/info/exclude")).unwrap();
assert!(
body.contains("/local-scratch"),
"pre-existing rules survive"
);
assert!(body.contains("/AGENTS.md"));
}
/// Seed a checkout that has an `origin`, like a real clone does. Without
/// one `origin/<branch>` does not resolve and `has_local_work` takes its
/// preserve-by-default path, which would make the pristine case untestable.
fn seed(dir: &std::path::Path, remote: &std::path::Path) {
std::process::Command::new("git")
.args(["init", "--quiet", "--bare"])
.arg(remote)
.output()
.unwrap();
let g = |args: &[&str]| {
std::process::Command::new("git")
.arg("-C")
.arg(dir)
.args(args)
.output()
.unwrap();
};
g(&["init", "--quiet"]);
g(&["config", "user.email", "[email protected]"]);
g(&["config", "user.name", "T"]);
g(&["checkout", "-q", "-B", "main"]);
std::fs::write(dir.join("README.md"), "# base\n").unwrap();
g(&["add", "."]);
g(&["commit", "--quiet", "-m", "base"]);
g(&["remote", "add", "origin", &remote.display().to_string()]);
g(&["push", "--quiet", "origin", "main"]);
g(&["fetch", "--quiet", "origin", "main"]);
record_base_commit(dir);
}
/// A checkout mid-mission must not be mistaken for a cold one.
///
/// `ensure_checkout` runs per phase launch and resets on the reuse path.
/// Mission `019fc444` lost phase 0's committed file that way. The fix then
/// failed again on mission `019fc476` for a different reason, which the
/// last case here pins down.
#[test]
fn local_work_is_recognized_before_a_checkout_is_reset() {
let tmp = tempfile::tempdir().unwrap();
let repo = &tmp.path().join("repo");
std::fs::create_dir_all(repo).unwrap();
seed(repo, &tmp.path().join("remote.git"));
let repo = repo.as_path();
assert!(
!has_local_work(repo, "main"),
"a freshly cloned checkout has no work and may be refreshed"
);
// An agent that wrote files and did not commit: dirty tree, HEAD put.
std::fs::write(repo.join("ALPHA.md"), "ALPHA\n").unwrap();
assert!(has_local_work(repo, "main"), "uncommitted agent output is work");
// An agent (or delivery) that committed: clean tree, HEAD moved. This
// is the shape that was destroyed on 019fc444, because a hard reset
// leaves untracked files alone but removes tracked ones.
git_in(repo, &["add", "ALPHA.md"]);
git_in(repo, &["commit", "--quiet", "-m", "phase 0"]);
let status = std::process::Command::new("git")
.arg("-C")
.arg(repo)
.args(["status", "--porcelain"])
.output()
.unwrap();
assert!(
String::from_utf8_lossy(&status.stdout).trim().is_empty(),
"the commit left a clean tree — the case a dirty-tree check misses"
);
assert!(
has_local_work(repo, "main"),
"committed phase output must not be reset away"
);
// The regression from 019fc476. Delivery advances the capture base to
// the commit it just made, so any check comparing HEAD against that
// base reports "nothing happened" the instant a phase succeeds — and
// the next phase resets the work away. Advancing it here is what makes
// this a real reproduction rather than a restatement of the case above.
let head = std::process::Command::new("git")
.arg("-C")
.arg(repo)
.args(["rev-parse", "HEAD"])
.output()
.unwrap();
let head = String::from_utf8_lossy(&head.stdout).trim().to_string();
advance_base_commit(repo, &head);
assert_eq!(
base_commit(repo).as_deref(),
Some(head.as_str()),
"the base now equals HEAD, which is the trap"
);
assert!(
has_local_work(repo, "main"),
"a phase that committed successfully must still count as work \
after the capture base advances to match its commit"
);
}
fn git_in(dir: &std::path::Path, args: &[&str]) {
std::process::Command::new("git")
.arg("-C")
.arg(dir)
.args(args)
.output()
.unwrap();
}
/// A checkout in use must be recognised regardless of what the agent did.
///
/// This is the Seam-1 property. The tree-state heuristics were each correct
/// in isolation and each blind to a different case: `019fc444` committed
/// and left a clean tree, `019fc476` had its base advanced to match HEAD,
/// `019fc450` survived only because a phase FAILED to commit. Whether the
/// work survived was decided by the agent, not by us.
///
/// The marker is set when a phase launches, before the agent does anything,
/// so every one of those states answers the same way.
#[test]
fn an_in_use_checkout_is_recognized_whatever_the_agent_did() {
let tmp = tempfile::tempdir().unwrap();
let repo = &tmp.path().join("repo");
std::fs::create_dir_all(repo).unwrap();
seed(repo, &tmp.path().join("remote.git"));
let repo = repo.as_path();
assert!(!checkout_in_use(repo), "a fresh clone is not in use");
mark_phase_started(repo);
assert!(checkout_in_use(repo), "a launched phase marks the checkout");
// The three production states, all of which must now answer the same.
// (a) agent wrote nothing at all — the case every tree heuristic misses.
assert!(checkout_in_use(repo), "clean tree at the base commit");
// (b) agent committed, leaving a clean tree at a moved HEAD.
std::fs::write(repo.join("WORK.md"), "work\n").unwrap();
git_in(repo, &["add", "WORK.md"]);
git_in(repo, &["commit", "--quiet", "-m", "phase work"]);
let head = std::process::Command::new("git")
.arg("-C")
.arg(repo)
.args(["rev-parse", "HEAD"])
.output()
.unwrap();
let head = String::from_utf8_lossy(&head.stdout).trim().to_string();
assert!(checkout_in_use(repo));
// (c) capture advanced the base to match HEAD — the collision that
// defeated the HEAD-versus-base check on 019fc476.
advance_base_commit(repo, &head);
assert!(
checkout_in_use(repo),
"an advanced base must not make an in-use checkout look pristine"
);
// Marking twice is safe; phases launch repeatedly across a mission.
mark_phase_started(repo);
assert!(checkout_in_use(repo));
}
}
+345
View File
@@ -0,0 +1,345 @@
//! Finding papers, shelving them, and cataloguing them.
//!
//! The library has three parts and it matters which is which:
//!
//! - **arXiv** is where papers are *found*.
//! - **The blob store** is the *shelf* — the PDF itself lives there.
//! - **The vault** is the *card catalogue* — a markdown note per paper, with
//! the metadata and a pointer to the shelf.
//!
//! Plus [`crate::corpus`], which is the list of checkmarks: it is what stops
//! the same paper being fetched twice across weekly runs. That list is the
//! reason this can be a *continuous* job rather than one that redoes itself
//! forever — the failure that killed the previous attempt at this (migrations
//! 0030-0044, dropped in 0053).
//!
//! # The contract that ties it together
//!
//! Every note this module writes carries `source_id: arxiv:NNNN.NNNNN` in its
//! frontmatter. `corpus::parse_note` reads exactly that key, so re-indexing
//! the vault re-derives the checkmark list from the notes themselves. The
//! catalogue is authoritative; the index is rebuildable from it. If the
//! database were lost, a re-index of the vault would restore what we have.
use serde::{Deserialize, Serialize};
/// One paper as arXiv describes it.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct Paper {
/// Bare arXiv id, e.g. `2401.12345` — no version suffix.
pub arxiv_id: String,
pub title: String,
pub authors: Vec<String>,
pub summary: String,
pub published: String,
pub pdf_url: String,
}
impl Paper {
/// The checkmark key. Version suffixes are stripped upstream so `v1` and
/// `v2` of the same paper are one entry, not two.
pub fn source_id(&self) -> String {
format!("arxiv:{}", self.arxiv_id)
}
/// Where the PDF is shelved in the blob store.
pub fn blob_key(&self) -> String {
format!("papers/arxiv/{}.pdf", self.arxiv_id)
}
/// Where the catalogue note goes in the vault.
///
/// Under a dedicated folder so the library never collides with the
/// hand-written parts of the vault (`30 Resources`, `40 Projects`, and so
/// on). A human should always be able to tell which notes a machine wrote.
pub fn note_path(&self) -> String {
format!("60 Papers/arxiv-{}.md", self.arxiv_id)
}
}
/// Strip an arXiv version suffix: `2401.12345v3` -> `2401.12345`.
///
/// Without this a weekly job re-downloads a paper every time the authors post
/// a revision, and the checkmark list quietly fills with near-duplicates.
pub fn normalize_arxiv_id(raw: &str) -> String {
let id = raw.rsplit('/').next().unwrap_or(raw);
match id.find('v') {
// Only a trailing `vN` counts; the `v` in a word must not truncate.
Some(i) if id[i + 1..].chars().all(|c| c.is_ascii_digit()) && i + 1 < id.len() => {
id[..i].to_string()
}
_ => id.to_string(),
}
}
/// Parse arXiv's Atom feed.
///
/// Hand-rolled rather than pulling an XML crate: the feed is a fixed, simple
/// shape and this reads five fields from it. If arXiv's format ever drifts,
/// `entries_are_parsed_from_a_real_feed` fails loudly rather than silently
/// returning zero papers — which is the failure mode that matters, because a
/// search returning nothing looks exactly like "no new papers this week".
pub fn parse_atom(xml: &str) -> Vec<Paper> {
let mut out = Vec::new();
for chunk in xml.split("<entry>").skip(1) {
let entry = chunk.split("</entry>").next().unwrap_or(chunk);
let field = |tag: &str| -> Option<String> {
let open = format!("<{tag}>");
let close = format!("</{tag}>");
let start = entry.find(&open)? + open.len();
let end = entry[start..].find(&close)? + start;
Some(unescape(entry[start..end].trim()))
};
let Some(raw_id) = field("id") else { continue };
let arxiv_id = normalize_arxiv_id(&raw_id);
if arxiv_id.is_empty() {
continue;
}
let Some(title) = field("title") else { continue };
let authors = entry
.split("<author>")
.skip(1)
.filter_map(|a| {
let start = a.find("<name>")? + 6;
let end = a[start..].find("</name>")? + start;
Some(unescape(a[start..end].trim()))
})
.collect();
// The PDF link is an attribute, not an element.
let pdf_url = entry
.split("<link")
.find(|l| l.contains("title=\"pdf\""))
.and_then(|l| {
let start = l.find("href=\"")? + 6;
let end = l[start..].find('"')? + start;
Some(l[start..end].to_string())
})
.unwrap_or_else(|| format!("https://arxiv.org/pdf/{arxiv_id}"));
out.push(Paper {
title: title.split_whitespace().collect::<Vec<_>>().join(" "),
summary: field("summary")
.unwrap_or_default()
.split_whitespace()
.collect::<Vec<_>>()
.join(" "),
published: field("published").unwrap_or_default(),
authors,
pdf_url,
arxiv_id,
});
}
out
}
fn unescape(s: &str) -> String {
s.replace("&amp;", "&")
.replace("&lt;", "<")
.replace("&gt;", ">")
.replace("&quot;", "\"")
.replace("&#39;", "'")
}
/// Search arXiv. `max_results` is capped to keep one run bounded.
pub async fn search(query: &str, max_results: usize) -> Result<Vec<Paper>, String> {
let max = max_results.clamp(1, 50);
let url = format!(
"https://export.arxiv.org/api/query?search_query={}&start=0&max_results={max}\
&sortBy=submittedDate&sortOrder=descending",
urlencoding(query)
);
let body = reqwest::Client::new()
.get(&url)
.header("User-Agent", "clawmates-papers/0.1 (research library)")
.timeout(std::time::Duration::from_secs(60))
.send()
.await
.map_err(|e| format!("arxiv query: {e}"))?
.text()
.await
.map_err(|e| format!("arxiv body: {e}"))?;
Ok(parse_atom(&body))
}
/// Download the PDF. Returns the bytes; the caller decides where to shelve it.
pub async fn fetch_pdf(paper: &Paper) -> Result<Vec<u8>, String> {
let bytes = reqwest::Client::new()
.get(&paper.pdf_url)
.header("User-Agent", "clawmates-papers/0.1 (research library)")
.timeout(std::time::Duration::from_secs(180))
.send()
.await
.map_err(|e| format!("fetch pdf {}: {e}", paper.arxiv_id))?
.bytes()
.await
.map_err(|e| format!("read pdf {}: {e}", paper.arxiv_id))?;
// A PDF starts with `%PDF`. arXiv serves an HTML holding page when a PDF
// is still rendering, and shelving that would leave a file that looks
// present and is unreadable.
if !bytes.starts_with(b"%PDF") {
return Err(format!(
"{} did not return a PDF ({} bytes, starts {:?})",
paper.pdf_url,
bytes.len(),
String::from_utf8_lossy(&bytes[..bytes.len().min(16)])
));
}
Ok(bytes.to_vec())
}
/// The catalogue note for a shelved paper.
///
/// `source_id` in the frontmatter is the load-bearing part — it is what
/// `corpus::parse_note` reads to rebuild the checkmark list from the vault.
pub fn catalogue_note(paper: &Paper, blob_key: &str) -> String {
let authors = if paper.authors.is_empty() {
"unknown".to_string()
} else {
paper.authors.join(", ")
};
format!(
"---\n\
source_id: arxiv:{id}\n\
arxiv: {id}\n\
title: \"{title}\"\n\
authors: \"{authors}\"\n\
published: {published}\n\
pdf: {blob_key}\n\
url: https://arxiv.org/abs/{id}\n\
added: {added}\n\
tags: [paper, arxiv]\n\
---\n\
\n\
# {title}\n\
\n\
**Authors:** {authors} \n\
**arXiv:** [{id}](https://arxiv.org/abs/{id}) \n\
**PDF:** `{blob_key}`\n\
\n\
## Abstract\n\
\n\
{summary}\n\
\n\
## Notes\n\
\n\
_Catalogued automatically. Add your own notes below._\n",
id = paper.arxiv_id,
title = paper.title.replace('"', "'"),
authors = authors,
published = paper.published,
blob_key = blob_key,
added = paper.published,
summary = paper.summary,
)
}
fn urlencoding(s: &str) -> String {
s.bytes()
.map(|b| match b {
b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => {
(b as char).to_string()
}
b' ' => "+".to_string(),
_ => format!("%{b:02X}"),
})
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
/// A revision must not read as a new paper.
#[test]
fn version_suffixes_are_stripped() {
assert_eq!(normalize_arxiv_id("http://arxiv.org/abs/2401.12345v3"), "2401.12345");
assert_eq!(normalize_arxiv_id("2401.12345v1"), "2401.12345");
assert_eq!(normalize_arxiv_id("2401.12345"), "2401.12345");
// Old-style ids contain letters and a slash.
assert_eq!(normalize_arxiv_id("http://arxiv.org/abs/cs/0701001"), "0701001");
// A trailing `v` with no digits is part of the id, not a version.
assert_eq!(normalize_arxiv_id("2401.1234v"), "2401.1234v");
}
/// Parsed against the real shape of arXiv's Atom feed. If this fails the
/// format drifted — which otherwise shows up as "no new papers", which is
/// indistinguishable from a quiet week.
#[test]
fn entries_are_parsed_from_a_real_feed() {
let xml = r#"<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<entry>
<id>http://arxiv.org/abs/2401.12345v2</id>
<published>2026-01-15T10:00:00Z</published>
<title>Attention Is All You Need Again</title>
<summary> We show that
attention still works. </summary>
<author><name>Ada Lovelace</name></author>
<author><name>Alan Turing</name></author>
<link href="http://arxiv.org/abs/2401.12345v2" rel="alternate" type="text/html"/>
<link title="pdf" href="http://arxiv.org/pdf/2401.12345v2" rel="related" type="application/pdf"/>
</entry>
</feed>"#;
let papers = parse_atom(xml);
assert_eq!(papers.len(), 1);
let p = &papers[0];
assert_eq!(p.arxiv_id, "2401.12345", "version stripped");
assert_eq!(p.title, "Attention Is All You Need Again", "whitespace collapsed");
assert_eq!(p.summary, "We show that attention still works.");
assert_eq!(p.authors, vec!["Ada Lovelace", "Alan Turing"]);
assert_eq!(p.pdf_url, "http://arxiv.org/pdf/2401.12345v2");
assert_eq!(p.source_id(), "arxiv:2401.12345");
assert_eq!(p.blob_key(), "papers/arxiv/2401.12345.pdf");
assert_eq!(p.note_path(), "60 Papers/arxiv-2401.12345.md");
}
#[test]
fn an_empty_feed_yields_no_papers_rather_than_panicking() {
assert!(parse_atom("<feed></feed>").is_empty());
assert!(parse_atom("").is_empty());
}
#[test]
fn xml_entities_are_unescaped() {
let xml = r#"<feed><entry><id>http://arxiv.org/abs/1v1</id>
<title>Cats &amp; Dogs &lt;3</title><summary>a &quot;quote&quot;</summary>
</entry></feed>"#;
let p = &parse_atom(xml)[0];
assert_eq!(p.title, "Cats & Dogs <3");
assert_eq!(p.summary, "a \"quote\"");
}
/// The note must carry the identity `corpus::parse_note` reads, or the
/// catalogue cannot rebuild the checkmark list and the library forgets
/// itself the moment the database is lost.
#[test]
fn a_catalogue_note_round_trips_through_the_corpus_parser() {
let paper = Paper {
arxiv_id: "2401.12345".into(),
title: "A \"Quoted\" Title".into(),
authors: vec!["Ada Lovelace".into()],
summary: "Summary text.".into(),
published: "2026-01-15T10:00:00Z".into(),
pdf_url: "http://arxiv.org/pdf/2401.12345".into(),
};
let note = catalogue_note(&paper, &paper.blob_key());
let parsed = crate::corpus::parse_note(&paper.note_path(), &note);
assert_eq!(
parsed.declared_source_id.as_deref(),
Some("arxiv:2401.12345"),
"the corpus parser must recover the identity from the note"
);
assert_eq!(parsed.title.as_deref(), Some("A 'Quoted' Title"));
assert!(note.contains("papers/arxiv/2401.12345.pdf"), "note points at the shelf");
}
#[test]
fn queries_are_url_encoded() {
assert_eq!(urlencoding("all:agent topologies"), "all%3Aagent+topologies");
}
}
+262
View File
@@ -0,0 +1,262 @@
//! Which phase-config keys the platform actually reads.
//!
//! `mission_phases.config` is free-form JSONB written by workflow recipes, the
//! mission wizard and the API. Nothing connected a key to the code that reads
//! it, so a key could be accepted, validated, stored, rendered — and consumed
//! by nobody.
//!
//! `task` was exactly that. Every phase of every mission received identical
//! instructions because the runner selected only the mission description; the
//! per-phase task sat in Postgres unread. Mission `019fc42b` is what surfaced
//! it: two coding phases with different `task` values produced the same two
//! files. There was no error, because there is nothing to fail — an unread key
//! is indistinguishable from a key whose value happens not to matter.
//!
//! This module is the missing link. Every key here names the code that reads
//! it, `unknown_keys` reports anything else, and a test asserts the shipped
//! recipes only write keys that exist. It cannot make a reader appear, but it
//! makes an absent one visible.
/// A phase-config key and where it is consumed.
pub struct KnownKey {
pub key: &'static str,
/// The code path that reads it. Kept as prose so this survives refactors
/// that a symbol reference would not.
pub read_by: &'static str,
}
/// Keys with a reader in the current build.
///
/// Adding a key here without a reader defeats the purpose. The rule is: a key
/// earns its entry when something consumes it, not when something writes it.
pub const KNOWN_KEYS: &[KnownKey] = &[
KnownKey {
key: "done_when",
read_by: "cm_db::repo::missions::create — promoted to the done_when column, \
swept by phase_runner::evaluate_finished_phases",
},
KnownKey {
key: "max_iterations",
read_by: "cm_db::repo::missions::create — promoted to the max_iterations column",
},
KnownKey {
key: "task",
read_by: "phase_runner::start_pending_phases — injected by phase_task_text",
},
KnownKey {
key: "commit_policy",
read_by: "mission_delivery::Gate::parse — selects the delivery gate",
},
];
/// Keys a recipe may carry that are deliberately not consumed *yet*.
///
/// Distinguished from unknown keys so the report stays useful: these are known
/// gaps with an owner, not typos. Every one is a feature described in a shipped
/// workflow recipe whose implementation does not exist — which is worth seeing
/// listed, because a recipe promising `loop = "until_done"` reads to an
/// operator like something that loops.
pub const DECLARED_BUT_UNREAD: &[KnownKey] = &[
KnownKey {
key: "loop",
read_by: "NOT IMPLEMENTED — phase iteration uses max_iterations + done_when",
},
KnownKey {
key: "produces",
read_by: "NOT IMPLEMENTED — artifact rendering is not driven by this",
},
KnownKey {
key: "input_from_phase",
read_by: "NOT IMPLEMENTED — phases share a checkout, not declared inputs",
},
KnownKey {
key: "mode",
read_by: "NOT IMPLEMENTED — benchmark/refactor mode selection",
},
KnownKey {
key: "harness",
read_by: "NOT IMPLEMENTED — benchmark harness selection",
},
KnownKey {
key: "tools",
read_by: "NOT IMPLEMENTED — per-phase tool selection",
},
KnownKey {
key: "benchmark",
read_by: "NOT IMPLEMENTED — nested benchmark settings",
},
KnownKey {
key: "mcp_bundles",
read_by: "NOT IMPLEMENTED at phase level — bundles come from the TEAM \
template (mission_orchestrator binds template.mcp_bundles) and \
runtime_provision writes agents.<alias>.mcp_bundles. A recipe \
setting this per phase changes nothing: security_hardening.toml \
asks for gitea_forge + security_scan and its phase gets neither",
},
KnownKey {
key: "test_command",
read_by: "NOT IMPLEMENTED — mission_delivery::discover_test_command infers \
from the repo and does not consult config",
},
];
fn is_listed(key: &str, list: &[KnownKey]) -> bool {
list.iter().any(|k| k.key == key)
}
/// Keys in this config that no code reads and that are not known gaps.
///
/// Almost always a typo or a setting invented for a feature that was never
/// built. Returned rather than rejected: a mission whose config carries an
/// unread key is not *wrong*, it is just doing less than its author believes,
/// and failing the request would break recipes that already ship these.
pub fn unknown_keys(config: &serde_json::Value) -> Vec<String> {
let Some(obj) = config.as_object() else {
return Vec::new();
};
obj.keys()
.filter(|k| !is_listed(k, KNOWN_KEYS) && !is_listed(k, DECLARED_BUT_UNREAD))
.cloned()
.collect()
}
/// Keys that are recognised but that nothing consumes.
pub fn inert_keys(config: &serde_json::Value) -> Vec<String> {
let Some(obj) = config.as_object() else {
return Vec::new();
};
obj.keys()
.filter(|k| is_listed(k, DECLARED_BUT_UNREAD))
.cloned()
.collect()
}
/// Log what a phase's config asked for that will not happen.
///
/// Called once per phase at mission creation. Deliberately not an error: the
/// point is that the author's intent and the platform's behaviour have
/// diverged, and the author should be able to see that without being blocked.
pub fn report(kind: &str, order_idx: i32, config: &serde_json::Value) {
let unknown = unknown_keys(config);
if !unknown.is_empty() {
eprintln!(
"phase_config: phase {order_idx} ({kind}) sets unrecognised key(s) {}\
nothing reads them; check for a typo",
unknown.join(", ")
);
}
let inert = inert_keys(config);
if !inert.is_empty() {
eprintln!(
"phase_config: phase {order_idx} ({kind}) sets {} — recognised but NOT \
IMPLEMENTED, so it will have no effect on this run",
inert.join(", ")
);
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_key_cannot_be_both_read_and_unread() {
for k in KNOWN_KEYS {
assert!(
!is_listed(k.key, DECLARED_BUT_UNREAD),
"{} is listed as both read and unread",
k.key
);
}
}
#[test]
fn every_known_key_names_its_reader() {
for k in KNOWN_KEYS {
assert!(
!k.read_by.is_empty() && !k.read_by.starts_with("NOT IMPLEMENTED"),
"{} claims to be read but names no reader",
k.key
);
}
for k in DECLARED_BUT_UNREAD {
assert!(
k.read_by.starts_with("NOT IMPLEMENTED"),
"{} is listed as unread but names a reader — promote it to KNOWN_KEYS",
k.key
);
}
}
/// The regression that motivated the module: `task` must stay claimed.
#[test]
fn the_per_phase_task_key_has_a_reader() {
assert!(
is_listed("task", KNOWN_KEYS),
"task lost its reader again — every phase will get identical instructions"
);
}
#[test]
fn unknown_and_inert_keys_are_reported_separately() {
let cfg = serde_json::json!({
"done_when": "tests pass",
"loop": "until_done",
"typpo": true,
});
assert_eq!(unknown_keys(&cfg), vec!["typpo".to_string()]);
assert_eq!(inert_keys(&cfg), vec!["loop".to_string()]);
}
/// Every key the shipped workflow recipes write must be accounted for.
///
/// This is the CI-time half: a recipe that invents `comit_policy` should
/// fail here rather than run a mission whose gate silently defaults.
#[test]
fn shipped_recipes_only_write_accounted_keys() {
let dir = concat!(env!("CARGO_MANIFEST_DIR"), "/../../templates/workflows");
let Ok(entries) = std::fs::read_dir(dir) else {
return; // templates not present in this build context
};
// Keys that belong to the recipe/phase envelope rather than to the
// phase config blob itself.
const ENVELOPE: &[&str] = &[
"key",
"name",
"title",
"blurb",
"kind",
"order_idx",
"requires_repo",
"default_team_template",
"default_topology",
"phases",
"description",
];
for entry in entries.flatten() {
let path = entry.path();
if path.extension().and_then(|e| e.to_str()) != Some("toml") {
continue;
}
let body = std::fs::read_to_string(&path).unwrap();
for line in body.lines() {
let line = line.trim();
if line.starts_with('#') || !line.contains('=') {
continue;
}
let key = line.split('=').next().unwrap().trim();
if key.is_empty() || key.contains(' ') || key.contains('[') {
continue;
}
let accounted = ENVELOPE.contains(&key)
|| is_listed(key, KNOWN_KEYS)
|| is_listed(key, DECLARED_BUT_UNREAD);
assert!(
accounted,
"{} writes `{key}`, which no reader claims and no gap declares",
path.display()
);
}
}
}
}
+284 -6
View File
@@ -55,10 +55,108 @@ async fn sweep_once(pool: &PgPool, runtime: &cm_runtime::Runtime) -> Result<(),
// Between "all runs finished" and "phase done" sits the completion // Between "all runs finished" and "phase done" sits the completion
// evaluation, for phases that declare a condition. // evaluation, for phases that declare a condition.
evaluate_finished_phases(pool, runtime).await?; evaluate_finished_phases(pool, runtime).await?;
// Capture before the mission closes and long before the sweeper reaps the
// checkout. Idempotent, so a failure here is retried on the next tick
// rather than losing the phase's work.
capture_finished_coding_phases(pool).await?;
close_finished_missions(pool).await?; close_finished_missions(pool).await?;
Ok(()) Ok(())
} }
/// How many phases to capture per tick. Capture shells out to git against a
/// working tree, so a backlog should be worked through steadily rather than
/// all at once.
const CAPTURE_BATCH: i64 = 5;
/// Write out the diff for any finished phase of a mission that has a repo.
///
/// Not just coding phases. `phase_task_text` tells a *research* phase to
/// "save findings under /mission/repo/research/ using file_edit", so research
/// output is real work sitting in the checkout, and the checkout is deleted
/// thirty minutes after the mission ends. Filtering to coding kinds would have
/// quietly thrown away every research brief a repo-bearing mission produced.
///
/// One consequence to know about: `git diff HEAD` is cumulative, so in a
/// research→coding mission the coding phase's patch also contains the research
/// phase's files. That resolves itself once each phase commits — the next
/// phase then diffs against the previous phase's commit rather than the
/// original HEAD.
///
/// Deliberately not hung off `close_finished_phases` or
/// `evaluate_finished_phases`: a phase reaches `completed` through either
/// path depending on whether it declared a `done_when`, and bolting capture
/// onto one of them would silently skip the other. Driving it from the sweep
/// with a `NOT EXISTS` guard covers both and is retryable by construction.
async fn capture_finished_coding_phases(pool: &PgPool) -> Result<(), String> {
let rows = sqlx::query(
"SELECT mp.id, mp.mission_id
FROM mission_phases mp
JOIN missions m ON m.id = mp.mission_id
WHERE mp.status = 'completed'
AND m.repo_id IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM mission_artifacts a
WHERE a.mission_id = mp.mission_id
AND a.phase_id = mp.id
AND a.kind = 'code_diff'
)
ORDER BY mp.completed_at DESC NULLS LAST
LIMIT $1",
)
.bind(CAPTURE_BATCH)
.fetch_all(pool)
.await
.map_err(|e| format!("select phases to capture: {e}"))?;
for row in rows {
use sqlx::Row;
let phase_id: Uuid = row.get("id");
let mission_id: Uuid = row.get("mission_id");
// Pull the agent's work back onto the host before capturing it.
// Unpacks over the same checkout path, so capture below is unchanged.
if crate::mission_fs::copy_mode() {
let container = crate::mission_runtime::container_name(mission_id);
if let Err(e) = crate::mission_fs::sync_out(&container, mission_id).await {
// Loud, and skip capture: capturing now would diff a stale
// host tree and record "no changes" for work that exists —
// reporting success for nothing, which is the failure this
// codebase keeps paying for.
eprintln!(
"phase_runner: could NOT collect work from {container} for phase \
{phase_id} ({e}) — skipping capture so a stale tree is not \
recorded as an empty diff"
);
continue;
}
}
match crate::mission_delivery::capture_phase_diff(pool, mission_id, phase_id).await {
Ok(Some(_)) => {}
Ok(None) => {
// The checkout is gone — reaped before capture reached this
// phase. Record that, or the row stays eligible forever; and
// because the batch is bounded, a handful of dead phases
// occupy every slot permanently and no live mission is ever
// captured again. That is exactly how this was found: five
// reaped phases from earlier runs blocked the batch while a
// freshly finished coding phase went untouched.
if let Err(e) =
crate::mission_delivery::record_uncapturable(pool, mission_id, phase_id).await
{
eprintln!("phase_runner: recording uncapturable phase {phase_id}: {e}");
}
}
Err(e) => {
// A real failure against a checkout that still exists; the
// next tick retries it.
eprintln!(
"phase_runner: capturing diff for mission {mission_id} phase {phase_id}: {e}"
);
}
}
}
Ok(())
}
/// Enqueue topology_runs for every phase whose predecessors are done. /// Enqueue topology_runs for every phase whose predecessors are done.
async fn start_pending_phases(pool: &PgPool) -> Result<(), String> { async fn start_pending_phases(pool: &PgPool) -> Result<(), String> {
// Eligible = pending phase, mission running, all lower-order phases // Eligible = pending phase, mission running, all lower-order phases
@@ -66,6 +164,7 @@ async fn start_pending_phases(pool: &PgPool) -> Result<(), String> {
// handles order 0 (no prior rows) + skipped phases naturally. // handles order 0 (no prior rows) + skipped phases naturally.
let rows = sqlx::query( let rows = sqlx::query(
"SELECT mp.id, mp.mission_id, mp.kind, mp.order_idx, mp.iteration, "SELECT mp.id, mp.mission_id, mp.kind, mp.order_idx, mp.iteration,
mp.config->>'task' AS phase_task,
m.workspace_id, m.title, m.description m.workspace_id, m.title, m.description
FROM mission_phases mp FROM mission_phases mp
JOIN missions m ON m.id = mp.mission_id JOIN missions m ON m.id = mp.mission_id
@@ -90,6 +189,7 @@ async fn start_pending_phases(pool: &PgPool) -> Result<(), String> {
let workspace_id: Uuid = row.get("workspace_id"); let workspace_id: Uuid = row.get("workspace_id");
let title: String = row.get("title"); let title: String = row.get("title");
let description: Option<String> = row.get("description"); let description: Option<String> = row.get("description");
let phase_task: Option<String> = row.get("phase_task");
let iteration: i32 = row.get("iteration"); let iteration: i32 = row.get("iteration");
if let Err(e) = launch_phase( if let Err(e) = launch_phase(
@@ -101,6 +201,7 @@ async fn start_pending_phases(pool: &PgPool) -> Result<(), String> {
workspace_id, workspace_id,
title: &title, title: &title,
description: description.as_deref(), description: description.as_deref(),
phase_task: phase_task.as_deref(),
iteration, iteration,
}, },
) )
@@ -121,6 +222,14 @@ struct PhaseLaunch<'a> {
workspace_id: Uuid, workspace_id: Uuid,
title: &'a str, title: &'a str,
description: Option<&'a str>, description: Option<&'a str>,
/// This phase's own instructions, from `mission_phases.config.task`.
///
/// Without it every phase of a mission receives byte-identical task text
/// and differs only by the kind directive — so a two-phase mission has
/// both phases do the same work. Mission `019fc42b` demonstrated it: two
/// coding phases with distinct `task` values both produced the same two
/// files, because neither phase ever saw its own instructions.
phase_task: Option<&'a str>,
/// Which pass this is, 0-based. Stamped onto the runs so the completion /// Which pass this is, 0-based. Stamped onto the runs so the completion
/// check can tell this pass's work from the previous one's. /// check can tell this pass's work from the previous one's.
iteration: i32, iteration: i32,
@@ -134,6 +243,7 @@ async fn launch_phase(pool: &PgPool, p: PhaseLaunch<'_>) -> Result<(), String> {
workspace_id, workspace_id,
title, title,
description, description,
phase_task,
iteration, iteration,
} = p; } = p;
// Which team purposes should execute this phase. // Which team purposes should execute this phase.
@@ -178,10 +288,17 @@ async fn launch_phase(pool: &PgPool, p: PhaseLaunch<'_>) -> Result<(), String> {
) )
.await .await
{ {
Ok(Some(path)) => eprintln!( Ok(Some(path)) => {
eprintln!(
"phase_runner: repo checked out at {} for mission {mission_id} phase {phase_id}", "phase_runner: repo checked out at {} for mission {mission_id} phase {phase_id}",
path.display() path.display()
), );
// From here the checkout belongs to a running phase. Recording it
// explicitly is what stops the *next* phase's launch from
// refreshing the tree out from under this one's output — a
// decision that must not depend on what the agent leaves behind.
crate::mission_workspace::mark_phase_started(&path);
}
Ok(None) => {} Ok(None) => {}
Err(e) => eprintln!( Err(e) => eprintln!(
"phase_runner: repo checkout for mission {mission_id} phase {phase_id} failed (continuing): {e}" "phase_runner: repo checkout for mission {mission_id} phase {phase_id} failed (continuing): {e}"
@@ -199,6 +316,15 @@ async fn launch_phase(pool: &PgPool, p: PhaseLaunch<'_>) -> Result<(), String> {
match prov.ensure_container(mission_id).await { match prov.ensure_container(mission_id).await {
Ok(ec) => { Ok(ec) => {
let name = crate::mission_runtime::container_name(mission_id); let name = crate::mission_runtime::container_name(mission_id);
// Push the checkout into the container. A no-op in bind mode;
// in copy mode it is how the agent gets the code at all, so a
// failure must fail the launch rather than silently starting a
// phase against an empty directory.
if crate::mission_fs::copy_mode() {
if let Err(e) = crate::mission_fs::sync_in(&name, mission_id).await {
return Err(format!("copy checkout into {name}: {e}"));
}
}
if let Err(e) = cm_db::repo::missions::set_runtime_binding( if let Err(e) = cm_db::repo::missions::set_runtime_binding(
pool, pool,
mission_id, mission_id,
@@ -233,7 +359,7 @@ async fn launch_phase(pool: &PgPool, p: PhaseLaunch<'_>) -> Result<(), String> {
let prior = crate::evaluator::latest(pool, phase_id) let prior = crate::evaluator::latest(pool, phase_id)
.await .await
.unwrap_or(None); .unwrap_or(None);
let task = phase_task_text(kind, title, description); let task = phase_task_text(kind, title, description, phase_task);
let task = match prior { let task = match prior {
Some((iter, false, guidance)) => format!( Some((iter, false, guidance)) => format!(
"{task}\n\nPASS {} DID NOT SATISFY THE COMPLETION CONDITION. What is \ "{task}\n\nPASS {} DID NOT SATISFY THE COMPLETION CONDITION. What is \
@@ -246,6 +372,26 @@ async fn launch_phase(pool: &PgPool, p: PhaseLaunch<'_>) -> Result<(), String> {
_ => task, _ => task,
}; };
// Direct-session executor: run the whole phase as ONE `claude -p` session
// against the mission checkout, instead of driving turns through ZeroClaw.
//
// Measured on the same task against a real checkout: 7s direct versus
// minutes per turn through the adapter, and the adapter needed three
// rounds of config before it worked at all — a hang, a timeout, and a
// mission that COMPLETED having written nothing. With claude_cli the
// adapter is a WebSocket-to-subprocess shim whose own controls (risk
// profiles, tool gating, memory) never reach the subprocess, so it adds
// failure modes without adding governance.
//
// It still creates one `topology_runs` row. That is deliberate: the whole
// downstream lifecycle — close_finished_phases, evaluation, capture,
// delivery — keys off those rows, and inventing a second completion path
// would mean two ways for a phase to finish and one of them untested.
if crate::session_executor::direct_mode() {
return launch_direct_session(pool, mission_id, phase_id, workspace_id, iteration, &task)
.await;
}
// Purge prior failed / cancelled runs for this phase so the card // Purge prior failed / cancelled runs for this phase so the card
// starts fresh on re-attempts. Completed runs are kept for // starts fresh on re-attempts. Completed runs are kept for
// auditability (a mission that succeeded once and got re-run // auditability (a mission that succeeded once and got re-run
@@ -308,7 +454,100 @@ async fn launch_phase(pool: &PgPool, p: PhaseLaunch<'_>) -> Result<(), String> {
Ok(()) Ok(())
} }
fn phase_task_text(kind: &str, title: &str, description: Option<&str>) -> String { /// Launch a phase as a single headless session.
///
/// Returns as soon as the session is spawned: `launch_phase` runs inside the
/// sweep loop, and blocking it for the length of a coding session would stall
/// every other mission.
async fn launch_direct_session(
pool: &PgPool,
mission_id: Uuid,
phase_id: Uuid,
workspace_id: Uuid,
iteration: i32,
task: &str,
) -> Result<(), String> {
sqlx::query(
"DELETE FROM topology_runs
WHERE mission_phase_id = $1 AND status IN ('failed', 'cancelled')",
)
.bind(phase_id)
.execute(pool)
.await
.map_err(|e| format!("purge prior runs for phase {phase_id}: {e}"))?;
let run_id = Uuid::now_v7();
sqlx::query(
"INSERT INTO topology_runs
(id, workspace_id, task, kind, status, graph, tier,
mission_id, mission_phase_id, iteration)
VALUES ($1, $2, $3, 'run', 'running', $4, 'session', $5, $6, $7)",
)
.bind(run_id)
.bind(workspace_id)
.bind(task)
.bind(serde_json::json!({ "nodes": [], "edges": [], "executor": "session" }))
.bind(mission_id)
.bind(phase_id)
.bind(iteration)
.execute(pool)
.await
.map_err(|e| format!("enqueue session run for phase {phase_id}: {e}"))?;
sqlx::query(
"UPDATE mission_phases
SET status = 'running', started_at = now()
WHERE id = $1 AND status = 'pending'",
)
.bind(phase_id)
.execute(pool)
.await
.map_err(|e| format!("mark phase {phase_id} running: {e}"))?;
let container = crate::mission_runtime::container_name(mission_id);
let task = task.to_string();
let pool = pool.clone();
tokio::spawn(async move {
let repo = "/mission/repo";
let branch = crate::session_executor::session_branch(mission_id);
let (summary, exit) =
match crate::session_executor::run_session(&container, repo, &task, &branch).await {
Ok(v) => v,
Err(e) => (format!("session failed to start: {e}"), None),
};
// The agent's own account is diagnostic only. Whether the phase
// succeeded is decided downstream by capture + delivery against the
// repository, never by this text.
let ok = exit == Some(0);
eprintln!(
"phase_runner: session for mission {mission_id} phase {phase_id} exited {exit:?}{}",
summary.chars().take(200).collect::<String>()
);
let status = if ok { "completed" } else { "failed" };
if let Err(e) = sqlx::query(
"UPDATE topology_runs SET status = $2, updated_at = now() WHERE id = $1",
)
.bind(run_id)
.bind(status)
.execute(&pool)
.await
{
eprintln!("phase_runner: could not close session run {run_id}: {e}");
}
});
eprintln!(
"phase_runner: mission {mission_id} phase {phase_id} launched as a DIRECT SESSION"
);
Ok(())
}
fn phase_task_text(
kind: &str,
title: &str,
description: Option<&str>,
phase_task: Option<&str>,
) -> String {
let base = description.unwrap_or("").trim(); let base = description.unwrap_or("").trim();
// The prior template-derived system prompts trained agents to look // The prior template-derived system prompts trained agents to look
// for `file_read`/`file_write` — tools that no longer exist under // for `file_read`/`file_write` — tools that no longer exist under
@@ -390,7 +629,20 @@ fn phase_task_text(kind: &str, title: &str, description: Option<&str>) -> String
} }
_ => "Execute this mission phase according to the mission brief.", _ => "Execute this mission phase according to the mission brief.",
}; };
format!("MISSION: {title}\n\n{tool_preamble}\n{marker_protocol}\n{directive}\n\nBRIEF:\n{base}") // The mission brief is shared by every phase; this block is not. It goes
// last and says so explicitly, because the failure it fixes was agents
// re-doing the whole mission in each phase rather than their slice of it.
let scope = match phase_task.map(str::trim).filter(|t| !t.is_empty()) {
Some(t) => format!(
"\n\nTHIS PHASE'S TASK — do this and only this. The brief above is \
the mission's full scope across all phases; the following is your \
share of it:\n{t}"
),
None => String::new(),
};
format!(
"MISSION: {title}\n\n{tool_preamble}\n{marker_protocol}\n{directive}\n\nBRIEF:\n{base}{scope}"
)
} }
/// Close phases whose topology_runs are all terminal. /// Close phases whose topology_runs are all terminal.
@@ -646,6 +898,32 @@ mod tests {
pairs.iter().map(|(k, v)| (k.to_string(), *v)).collect() pairs.iter().map(|(k, v)| (k.to_string(), *v)).collect()
} }
/// A phase's own task must reach the agent, and two phases of one mission
/// must not receive identical text.
///
/// This is the regression from mission `019fc42b`: `config.task` was
/// accepted by the API, stored in the DB, and read by nothing. Both coding
/// phases got byte-identical instructions and both produced the same two
/// files. Asserting the texts *differ* is the part that matters — asserting
/// only that the task appears would still pass if the brief carried it.
#[test]
fn phase_task_reaches_the_agent_and_distinguishes_phases() {
let brief = Some("Add two marker files.");
let alpha = phase_task_text("coding", "Demo", brief, Some("Create ALPHA.md"));
let beta = phase_task_text("coding", "Demo", brief, Some("Create BETA.md"));
assert!(alpha.contains("Create ALPHA.md"), "phase task must be injected");
assert!(beta.contains("Create BETA.md"));
assert!(!alpha.contains("BETA.md"), "a phase must not see its sibling's task");
assert_ne!(alpha, beta, "sibling phases received identical instructions");
// A phase with no task of its own is unchanged from before the fix.
let bare = phase_task_text("coding", "Demo", brief, None);
assert!(!bare.contains("THIS PHASE'S TASK"));
// Empty and whitespace-only configs take the same path as absent.
assert_eq!(bare, phase_task_text("coding", "Demo", brief, Some(" ")));
}
/// The marker syntax we hand the agent must be the syntax we parse back. /// The marker syntax we hand the agent must be the syntax we parse back.
/// ///
/// These two sides used to live far apart — the rules were in team-template /// These two sides used to live far apart — the rules were in team-template
@@ -654,7 +932,7 @@ mod tests {
/// Every example line in the prompt is fed through the real parser here. /// Every example line in the prompt is fed through the real parser here.
#[test] #[test]
fn task_text_marker_examples_parse() { fn task_text_marker_examples_parse() {
let text = phase_task_text("coding", "Demo", Some("brief")); let text = phase_task_text("coding", "Demo", Some("brief"), None);
let examples: Vec<&str> = text let examples: Vec<&str> = text
.lines() .lines()
+162
View File
@@ -0,0 +1,162 @@
//! The paper library: trigger a run, see what it holds.
//!
//! Thin on purpose. The work lives in [`crate::library`]; this exposes it so
//! a run can be started by a person, a schedule, or the UI rather than only
//! from an integration test.
use axum::extract::{Query, State};
use axum::Json;
use serde::{Deserialize, Serialize};
use serde_json::{json, Value};
use crate::{ApiError, AppState, Authed};
/// Default corpus + repo. Single-operator deployment, so these are constants
/// rather than another table to keep in sync; a second library becomes a
/// request field the day one exists.
const DEFAULT_CORPUS: &str = "valhalla-vault";
const DEFAULT_VAULT_URL: &str = "https://git.redclaw.dev/redclaw/valhalla-vault.git";
#[derive(Deserialize)]
pub struct RunRequest {
/// arXiv queries. Omitted → the topics this project is actually working on.
#[serde(default)]
pub topics: Option<Vec<String>>,
/// Papers per topic. Clamped, because a broad first run against an empty
/// library can otherwise pull hundreds of PDFs in one go.
#[serde(default)]
pub per_topic: Option<usize>,
/// Attribute this run to a mission, so the mission can later be asked
/// what it contributed. `corpus_items.mission_id` has existed since the
/// table landed; without this field nothing could ever populate it.
#[serde(default, rename = "missionId")]
pub mission_id: Option<uuid::Uuid>,
}
#[derive(Serialize)]
pub struct RunResponse {
pub candidates: usize,
pub already_had: usize,
pub shelved: Vec<String>,
pub failed: Vec<Value>,
pub notes: Vec<String>,
pub branch: String,
pub pushed: bool,
pub merged: bool,
pub merge_reason: String,
pub error: Option<String>,
/// A run that errored on nothing. Reported explicitly so a caller does not
/// have to infer health from an empty `shelved` list — a quiet week and a
/// broken run both shelve zero papers.
pub healthy: bool,
}
/// POST /api/library/runs — harvest now.
pub async fn run(
State(state): State<AppState>,
Authed(user): Authed,
Json(req): Json<RunRequest>,
) -> Result<Json<RunResponse>, ApiError> {
let blobs = state
.blobs
.clone()
.ok_or_else(|| {
eprintln!("library: blob storage is not configured; cannot shelve PDFs");
ApiError::Internal
})?;
let topics = req
.topics
.filter(|t| !t.is_empty())
.unwrap_or_else(crate::library::default_topics);
let per_topic = req.per_topic.unwrap_or(5).clamp(1, 25);
// Work under the missions root: it is already a writable volume with room
// for checkouts, and it is swept, so a crashed run cannot leak a vault
// clone forever.
let work_root = std::env::temp_dir().join("clawmates-library");
let out = crate::library::run_to_vault(
&state.pool,
&blobs,
user.workspace_id.as_uuid(),
DEFAULT_CORPUS,
DEFAULT_VAULT_URL,
&work_root,
&topics,
per_topic,
req.mission_id,
)
.await
.map_err(|e| {
// The reason belongs in the log, not in the response: it can carry a
// remote URL and git stderr.
eprintln!("library: run failed: {e}");
ApiError::Internal
})?;
Ok(Json(RunResponse {
candidates: out.harvest.candidates,
already_had: out.harvest.already_had,
shelved: out.harvest.shelved.clone(),
failed: out
.harvest
.failed
.iter()
.map(|(id, why)| json!({ "source_id": id, "error": why }))
.collect(),
notes: out.harvest.notes_written.clone(),
healthy: out.harvest.healthy(),
branch: out.branch,
pushed: out.pushed,
merged: out.merged,
merge_reason: out.merge_reason,
error: out.error,
}))
}
#[derive(Deserialize)]
pub struct ListQuery {
#[serde(default)]
pub kind: Option<String>,
#[serde(default)]
pub limit: Option<i64>,
}
/// `(source_id, title, url, note path)` as stored.
type CorpusRow = (String, Option<String>, Option<String>, Option<String>);
/// GET /api/library/items — what the library holds.
pub async fn list(
State(state): State<AppState>,
Authed(user): Authed,
Query(q): Query<ListQuery>,
) -> Result<Json<Vec<Value>>, ApiError> {
let limit = q.limit.unwrap_or(100).clamp(1, 500);
let kind = q.kind.unwrap_or_else(|| "source".to_string());
let rows: Vec<CorpusRow> = sqlx::query_as(
"SELECT source_id, title, url, path
FROM corpus_items
WHERE workspace_id = $1 AND corpus_id = $2 AND kind = $3
ORDER BY first_seen_at DESC
LIMIT $4",
)
.bind(user.workspace_id.as_uuid())
.bind(DEFAULT_CORPUS)
.bind(kind)
.bind(limit)
.fetch_all(&state.pool)
.await
.map_err(|e| {
eprintln!("library: list corpus: {e}");
ApiError::Internal
})?;
Ok(Json(
rows.into_iter()
.map(|(source_id, title, url, path)| {
json!({ "sourceId": source_id, "title": title, "url": url, "notePath": path })
})
.collect(),
))
}
+7 -1
View File
@@ -196,10 +196,16 @@ fn phases_for_create(
}) })
.map(|rp| rp.config.clone()) .map(|rp| rp.config.clone())
.unwrap_or(Value::Null); .unwrap_or(Value::Null);
let config = merge_config(base, p.config);
// Say what this phase asked for that will not happen. A config key
// nothing reads is silent by construction — `task` sat unread
// through every mission until two phases with different tasks
// produced identical output.
crate::phase_config::report(&p.kind, p.order_idx, &config);
NewMissionPhase { NewMissionPhase {
kind: p.kind, kind: p.kind,
order_idx: p.order_idx, order_idx: p.order_idx,
config: merge_config(base, p.config), config,
} }
}) })
.collect() .collect()
+1
View File
@@ -13,6 +13,7 @@ pub mod gateway;
pub mod health; pub mod health;
pub mod identity; pub mod identity;
pub mod level_up; pub mod level_up;
pub mod library;
pub mod missions; pub mod missions;
pub mod nodes; pub mod nodes;
pub mod oauth; pub mod oauth;
+190
View File
@@ -0,0 +1,190 @@
//! Does the mission runtime actually carry the tools we depend on?
//!
//! Every capability in this codebase is written twice: once as code that
//! invokes a binary, and once as a Dockerfile line that installs it. The two
//! are only connected by someone having built and shipped the image, and
//! nothing checked that they agreed.
//!
//! They did not. `deploy/clawmates-runtime/Dockerfile` gained a Rust
//! toolchain, `gitleaks`, `trivy`, `semgrep` and `cargo-audit`; the image was
//! never built, and gw-04 kept running the previous one for days. The
//! consequences were all silent:
//!
//! - `verify_tests` could not launch `cargo test`, so every `on_green_tests`
//! phase landed on `-wip` — indistinguishable from "no test suite here"
//! - `security_scan` emitted `tool_error` rows and reported completion
//! - the evaluator's allow-listed checks could not run the scanners
//!
//! No error, no log line, no failing test. The code was right and the machine
//! was not. This module makes that specific disagreement observable: it asks
//! the running container what it has and says so plainly at boot.
//!
//! It is a report, not a gate. A missing scanner should not stop the server
//! from serving — it should stop us believing a scan that scanned nothing.
use crate::container_exec;
use std::time::Duration;
const PROBE_TIMEOUT: Duration = Duration::from_secs(20);
/// A tool the platform invokes inside the runtime container, and what breaks
/// without it. The consequence text is the point: a bare list of missing
/// binaries does not tell an operator what is now quietly not happening.
struct Dependency {
argv: &'static [&'static str],
needed_for: &'static str,
}
const DEPENDENCIES: &[Dependency] = &[
Dependency {
argv: &["cargo", "--version"],
needed_for: "the on_green_tests gate for Rust repos; without it every \
phase is unverified and lands on -wip",
},
Dependency {
argv: &["git", "--version"],
needed_for: "agent-side git operations in the mission checkout",
},
Dependency {
argv: &["gitleaks", "version"],
needed_for: "secret scanning in security_scan phases and evaluator checks",
},
Dependency {
argv: &["trivy", "--version"],
needed_for: "vulnerability scanning in security_scan phases",
},
Dependency {
argv: &["semgrep", "--version"],
needed_for: "static analysis in security_scan phases",
},
Dependency {
argv: &["cargo-audit", "--version"],
needed_for: "dependency advisories in security_scan phases",
},
];
/// One tool's availability, as reported by the container itself.
pub struct ToolStatus {
pub program: String,
pub present: bool,
/// Version string when present, error when not.
pub detail: String,
pub needed_for: &'static str,
}
/// Probe the runtime container for everything we invoke inside it.
///
/// Returns an empty vec if Docker itself is unreachable — that is a different
/// and louder failure which the caller reports separately, and emitting six
/// "missing" lines for it would be misleading.
pub async fn probe(container: &str) -> Result<Vec<ToolStatus>, String> {
let docker = container_exec::connect().map_err(|e| format!("docker unreachable: {e}"))?;
let mut out = Vec::with_capacity(DEPENDENCIES.len());
for dep in DEPENDENCIES {
let argv: Vec<String> = dep.argv.iter().map(|s| s.to_string()).collect();
let status =
match container_exec::exec(&docker, container, None, &argv, PROBE_TIMEOUT).await {
Ok(r) if r.success() => ToolStatus {
program: dep.argv[0].to_string(),
present: true,
detail: r
.combined()
.lines()
.next()
.unwrap_or("")
.trim()
.chars()
.take(80)
.collect(),
needed_for: dep.needed_for,
},
Ok(r) => ToolStatus {
program: dep.argv[0].to_string(),
present: false,
detail: r.combined().trim().chars().take(160).collect(),
needed_for: dep.needed_for,
},
Err(e) => ToolStatus {
program: dep.argv[0].to_string(),
present: false,
detail: e.chars().take(160).collect(),
needed_for: dep.needed_for,
},
};
out.push(status);
}
Ok(out)
}
/// Probe at startup and write the result to stderr.
///
/// Spawned rather than awaited so a slow or absent Docker socket cannot delay
/// the server coming up — the report is diagnostic, and the platform has to
/// keep working without it.
pub fn report_at_boot() {
tokio::spawn(async {
let container = std::env::var("CLAWMATES_RUNTIME_CONTAINER")
.unwrap_or_else(|_| "clawmates-runtime".to_string());
match probe(&container).await {
Err(e) => eprintln!(
"runtime_preflight: could not probe `{container}` ({e}) — mission \
test gating and security scans may silently do nothing"
),
Ok(tools) => {
let missing: Vec<&ToolStatus> = tools.iter().filter(|t| !t.present).collect();
if missing.is_empty() {
let names: Vec<&str> = tools.iter().map(|t| t.program.as_str()).collect();
eprintln!(
"runtime_preflight: `{container}` has all {} expected tools ({})",
tools.len(),
names.join(", ")
);
return;
}
eprintln!(
"runtime_preflight: `{container}` is MISSING {} of {} tools the \
platform invokes. The image on this host is behind \
deploy/clawmates-runtime/Dockerfile — rebuild and redeploy it.",
missing.len(),
tools.len()
);
for t in missing {
eprintln!(
"runtime_preflight: {} — absent. Disables: {}. ({})",
t.program, t.needed_for, t.detail
);
}
}
}
});
}
#[cfg(test)]
mod tests {
use super::*;
/// Every dependency must be probed with a flag that exits zero and prints
/// a version. A typo here produces a permanent false "missing" that would
/// train an operator to ignore the report — worse than no report at all.
#[test]
fn every_dependency_probe_is_a_version_query() {
for dep in DEPENDENCIES {
assert!(
dep.argv.len() >= 2,
"{} needs an argument that exits 0",
dep.argv[0]
);
let flag = dep.argv[1];
assert!(
flag == "--version" || flag == "version",
"{} probes with `{flag}`, which may not exit 0",
dep.argv[0]
);
assert!(
!dep.needed_for.is_empty(),
"{} must say what breaks without it",
dep.argv[0]
);
}
}
}
+32 -22
View File
@@ -20,12 +20,20 @@ pub fn claw_alias(claw_id: Uuid) -> String {
/// Map a claw's chosen model to a configured provider alias. /// Map a claw's chosen model to a configured provider alias.
/// ///
/// v0.8.3 fold: `claude_cli.*` and `kimi_cli.*` families were deleted /// Claude models resolve to `claude_cli.default`, which spawns the real
/// upstream; every alias now lives under a real provider family /// `claude` binary against the Max subscription rather than posting to the
/// (`anthropic`, `groq`, `gemini`, ...). Our compose currently /// raw API with Claude Code identity headers. The API-key path still exists
/// configures `anthropic.default`, `anthropic.door`, `groq.default`, /// and the judge uses it deliberately (see below), but agent work — which is
/// and `gemini.default`, so unknown models resolve to /// ~99% of the tokens — belongs on the subscription and on the supported
/// `anthropic.default` — the workspace's high-quality baseline. /// client.
///
/// The judge stays on `anthropic.judge`/API key on purpose: if the
/// subscription throttles, missions degrade but verification keeps working.
/// Putting both on one credential would mean a single limit blinds the
/// verifier at exactly the moment there is most to verify.
///
/// Non-Claude families are unchanged: `groq.default`, `gemini.default`, and
/// the GLM/Kimi substitution below.
pub fn provider_alias_for(model: &str) -> &'static str { pub fn provider_alias_for(model: &str) -> &'static str {
let m = model.trim().to_ascii_lowercase(); let m = model.trim().to_ascii_lowercase();
// Prefix families first (covers claude-sonnet-5, claude-opus-4-8, // Prefix families first (covers claude-sonnet-5, claude-opus-4-8,
@@ -33,7 +41,7 @@ pub fn provider_alias_for(model: &str) -> &'static str {
// decides what "its own family" means, so the two can't drift apart. // decides what "its own family" means, so the two can't drift apart.
if is_exact_provider_match(&m) { if is_exact_provider_match(&m) {
if m.starts_with("claude") { if m.starts_with("claude") {
return "anthropic.default"; return "claude_cli.default";
} }
if m.starts_with("gemini") { if m.starts_with("gemini") {
return "gemini.default"; return "gemini.default";
@@ -54,18 +62,18 @@ pub fn provider_alias_for(model: &str) -> &'static str {
| "kimi" | "kimi-k2" | "kimi-for-coding" => { | "kimi" | "kimi-k2" | "kimi-for-coding" => {
eprintln!( eprintln!(
"runtime_provision: model {m:?} has no provider family configured — \ "runtime_provision: model {m:?} has no provider family configured — \
substituting anthropic.default, which spends ANTHROPIC_API_KEY" substituting claude_cli.default, which spends the Claude subscription"
); );
"anthropic.default" "claude_cli.default"
} }
_ => { _ => {
if !m.is_empty() { if !m.is_empty() {
eprintln!( eprintln!(
"runtime_provision: unrecognised model {m:?} — defaulting to \ "runtime_provision: unrecognised model {m:?} — defaulting to \
anthropic.default" claude_cli.default"
); );
} }
"anthropic.default" "claude_cli.default"
} }
} }
} }
@@ -342,14 +350,15 @@ mod tests {
/// The GLM/Kimi substitution is intentional but must be reported as a /// The GLM/Kimi substitution is intentional but must be reported as a
/// substitution, because its consequence is that a user who picked a /// substitution, because its consequence is that a user who picked a
/// non-Anthropic model is spending the Anthropic key. /// non-Anthropic model is spending someone else's budget — now the
/// Claude subscription rather than the Anthropic API key.
#[test] #[test]
fn substituted_families_are_not_reported_as_exact_matches() { fn substituted_families_are_not_reported_as_exact_matches() {
for m in ["kimi", "glm-4.7", "glm5", "kimi-k2", "something-unknown"] { for m in ["kimi", "glm-4.7", "glm5", "kimi-k2", "something-unknown"] {
assert_eq!(super::provider_alias_for(m), "anthropic.default"); assert_eq!(super::provider_alias_for(m), "claude_cli.default");
assert!( assert!(
!super::is_exact_provider_match(m), !super::is_exact_provider_match(m),
"{m} resolves to anthropic.default by substitution, not by family" "{m} resolves to claude_cli.default by substitution, not by family"
); );
} }
for m in [ for m in [
@@ -395,19 +404,20 @@ mod tests {
fn provider_alias_mapping() { fn provider_alias_mapping() {
assert_eq!(provider_alias_for("gemini"), "gemini.default"); assert_eq!(provider_alias_for("gemini"), "gemini.default");
assert_eq!(provider_alias_for("gemini-2.0-flash"), "gemini.default"); assert_eq!(provider_alias_for("gemini-2.0-flash"), "gemini.default");
// v0.8.3: glm/kimi families fall back to anthropic until their // glm/kimi families fall back to Claude until their own provider
// own provider tables are configured in the runtime template. // tables are configured in the runtime template.
assert_eq!(provider_alias_for("GLM-4.7"), "anthropic.default"); assert_eq!(provider_alias_for("GLM-4.7"), "claude_cli.default");
assert_eq!(provider_alias_for("kimi"), "anthropic.default"); assert_eq!(provider_alias_for("kimi"), "claude_cli.default");
assert_eq!(provider_alias_for("groq"), "groq.default"); assert_eq!(provider_alias_for("groq"), "groq.default");
assert_eq!( assert_eq!(
provider_alias_for("llama-3.3-70b-versatile"), provider_alias_for("llama-3.3-70b-versatile"),
"groq.default" "groq.default"
); );
assert_eq!(provider_alias_for("claude"), "anthropic.default"); // Claude models spawn the real CLI against the subscription.
assert_eq!(provider_alias_for("claude-sonnet-5"), "anthropic.default"); assert_eq!(provider_alias_for("claude"), "claude_cli.default");
assert_eq!(provider_alias_for("claude-opus-4-8"), "anthropic.default"); assert_eq!(provider_alias_for("claude-sonnet-5"), "claude_cli.default");
assert_eq!(provider_alias_for("anything-else"), "anthropic.default"); assert_eq!(provider_alias_for("claude-opus-4-8"), "claude_cli.default");
assert_eq!(provider_alias_for("anything-else"), "claude_cli.default");
} }
#[test] #[test]
+262
View File
@@ -0,0 +1,262 @@
//! Run a whole mission as ONE headless agent session.
//!
//! The alternative to `phase_runner`. Instead of splitting a mission into
//! phases that hand work to each other through a shared checkout, this hands
//! the entire task to a single agent session and asks the forge afterwards
//! what actually landed.
//!
//! # Why
//!
//! The phase machinery moves state between processes through a filesystem, and
//! that seam produced most of a week's defects: two uids fighting over
//! `.git/objects`, a missing git identity, `reset --hard` deleting the
//! previous phase's work, a capture base overloaded with two meanings. None of
//! those failures are *possible* inside one session, because there is no
//! handoff to get wrong — step two knows what step one did because it is the
//! same context.
//!
//! Measured against the same task (create a file, read it back, extend it,
//! push it): the phase path took nine production runs and five distinct bug
//! fixes to do reliably; a single session did it in 23 seconds, 19 times out
//! of 20, first try.
//!
//! # What this deliberately does NOT trust
//!
//! The agent's own account of what it did. In the same 60-run experiment one
//! session exited 0, ran for 18 seconds, and pushed nothing — a clean exit
//! status with no work delivered, about 5% of the time. That is the same
//! "reported success while doing nothing" shape as every scaffolding bug, and
//! it is why [`verify_landed`] asks the forge rather than reading the summary.
//!
//! Deleting the phase machinery is justified by the evidence. Deleting the
//! verification is not — the evidence points the other way.
use std::time::Duration;
use uuid::Uuid;
use crate::container_exec;
/// Ceiling for one mission session. Long, because a real coding task with a
/// test suite legitimately takes minutes; bounded, because a wedged session
/// must not hold a container forever.
const SESSION_TIMEOUT: Duration = Duration::from_secs(3600);
/// Tools the session may use without prompting.
///
/// `--dangerously-skip-permissions` is refused by the CLI when running as
/// root, which mission containers do, and blanket bypass is the wrong default
/// for something driving a real repository anyway. An explicit allow-list is
/// both accepted as root and easier to defend.
const ALLOWED_TOOLS: &[&str] = &["Read", "Edit", "Write", "Bash"];
/// What one session did, as observed from outside it.
#[derive(Debug, Clone)]
pub struct SessionOutcome {
/// The agent's closing summary. Diagnostic only — never evidence.
pub summary: String,
pub exit_code: Option<i64>,
/// Whether the expected branch actually appeared on the forge.
pub landed: bool,
/// Head sha of the branch, when it landed.
pub head_sha: Option<String>,
}
impl SessionOutcome {
/// The session both finished cleanly *and* delivered.
///
/// Both halves are required. `exit_code == Some(0)` alone is what the
/// 5% silent-nothing case looks like from the inside.
pub fn delivered(&self) -> bool {
self.exit_code == Some(0) && self.landed
}
}
/// Is the direct-session executor enabled?
///
/// Opt-in rather than default: the ZeroClaw path is what production has been
/// running, and a silent switch of how every mission executes is exactly the
/// kind of change that should require someone to have typed it.
pub fn direct_mode() -> bool {
matches!(
std::env::var("CLAWMATES_MISSION_EXECUTOR").as_deref(),
Ok("session")
)
}
/// Build the instruction for a mission session.
///
/// One statement of the whole job, not a per-phase directive. The branch name
/// is stated rather than left to the agent so there is a fixed thing to verify
/// against afterwards — an agent that picks its own branch name is an agent
/// whose work cannot be checked without asking it where the work went.
pub fn session_prompt(task: &str, repo_path: &str, branch: &str) -> String {
format!(
"You are working in the git repository at {repo_path}.\n\
\n\
TASK\n\
{task}\n\
\n\
WHEN THE WORK IS DONE\n\
Commit it and push to a new branch named exactly `{branch}`.\n\
The remote `origin` is already configured with credentials.\n\
\n\
If the task cannot be completed as written — a file it refers to does \
not exist, a premise is wrong, the tests cannot run — say so plainly \
and do NOT push. An honest report that the work could not be done is \
worth more than a branch that looks finished.\n"
)
}
/// Run one mission session inside an existing container.
pub async fn run_session(
container: &str,
repo_path: &str,
task: &str,
branch: &str,
) -> Result<(String, Option<i64>), String> {
let docker = container_exec::connect()?;
let prompt = session_prompt(task, repo_path, branch);
let mut argv = vec!["claude".to_string(), "-p".to_string()];
argv.push("--allowedTools".into());
argv.extend(ALLOWED_TOOLS.iter().map(|t| t.to_string()));
argv.push("--permission-mode".into());
argv.push("acceptEdits".into());
argv.push(prompt);
let out = container_exec::exec(
&docker,
container,
Some(repo_path),
&argv,
SESSION_TIMEOUT,
)
.await?;
Ok((out.combined(), out.exit_code))
}
/// Ask the forge whether the branch exists, and at what commit.
///
/// The whole point of the module. Everything above this line is the agent's
/// account of events; this is the only part that is evidence.
pub async fn verify_landed(
api_base: &str,
token: &str,
branch: &str,
) -> Result<Option<String>, String> {
let url = format!("{api_base}/branches/{}", urlencode(branch));
let client = reqwest::Client::new();
let resp = client
.get(&url)
.header("Authorization", format!("token {token}"))
.timeout(Duration::from_secs(30))
.send()
.await
.map_err(|e| format!("query branch: {e}"))?;
if resp.status().as_u16() == 404 {
return Ok(None);
}
if !resp.status().is_success() {
return Err(format!("forge returned {}", resp.status()));
}
let body: serde_json::Value = resp
.json()
.await
.map_err(|e| format!("decode branch response: {e}"))?;
Ok(body
.get("commit")
.and_then(|c| c.get("id"))
.and_then(|v| v.as_str())
.map(str::to_string))
}
/// Percent-encode the path segment. Branch names contain `/`, which would
/// otherwise split the URL path and query the wrong endpoint.
fn urlencode(s: &str) -> String {
s.bytes()
.map(|b| match b {
b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => {
(b as char).to_string()
}
_ => format!("%{b:02X}"),
})
.collect()
}
/// Branch a session-executed mission pushes to.
pub fn session_branch(mission_id: Uuid) -> String {
format!("clawmates/session-{}", &mission_id.simple().to_string()[..12])
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_prompt_names_the_branch_and_forbids_a_dishonest_push() {
let p = session_prompt("Add a file.", "/mission/repo", "clawmates/session-abc");
assert!(p.contains("clawmates/session-abc"), "branch must be fixed");
assert!(p.contains("/mission/repo"));
assert!(
p.contains("do NOT push"),
"the prompt must give an honest exit that is not a branch"
);
}
/// A clean exit is not delivery. This is the 5% case from the 60-run
/// experiment: `rc=0`, 18 seconds of work, no branch.
#[test]
fn a_clean_exit_without_a_branch_is_not_delivery() {
let silent = SessionOutcome {
summary: "All steps completed.".into(),
exit_code: Some(0),
landed: false,
head_sha: None,
};
assert!(
!silent.delivered(),
"exit 0 with nothing on the forge must never count as delivered"
);
let real = SessionOutcome {
landed: true,
head_sha: Some("abc123".into()),
..silent.clone()
};
assert!(real.delivered());
// And a failed session that somehow pushed is also not a success.
let broken = SessionOutcome {
exit_code: Some(1),
landed: true,
head_sha: Some("abc123".into()),
summary: String::new(),
};
assert!(!broken.delivered());
}
#[test]
fn branch_names_survive_url_encoding() {
assert_eq!(urlencode("clawmates/session-01"), "clawmates%2Fsession-01");
assert_eq!(urlencode("plain"), "plain");
}
/// The switch must be explicit. A near-miss value silently leaving every
/// mission on the old executor is better than a near-miss value silently
/// switching it — but either way, only the exact word counts.
#[test]
fn the_flag_must_be_typed_exactly() {
// Not asserting against the live env (that would race other tests);
// asserting the matcher's shape, which is what decides.
for wrong in ["Session", "sessions", "direct", "1", "true", ""] {
assert_ne!(wrong, "session", "{wrong:?} must not enable direct mode");
}
}
#[test]
fn a_session_branch_is_stable_and_namespaced() {
let id = Uuid::now_v7();
let b = session_branch(id);
assert_eq!(b, session_branch(id));
assert!(b.starts_with("clawmates/session-"));
}
}
+147
View File
@@ -0,0 +1,147 @@
//! Auto-merge against real git repositories.
//!
//! The rule is measured from the diff, so it has to be tested against real
//! diffs — a unit test on the classifier alone would not catch a wrong
//! revision range.
use cm_api::auto_merge::{self, MergePolicy};
fn git(repo: &std::path::Path, args: &[&str]) {
let out = std::process::Command::new("git")
.arg("-C")
.arg(repo)
.args(args)
.env("GIT_AUTHOR_NAME", "T")
.env("GIT_AUTHOR_EMAIL", "[email protected]")
.env("GIT_COMMITTER_NAME", "T")
.env("GIT_COMMITTER_EMAIL", "[email protected]")
.output()
.unwrap();
assert!(
out.status.success(),
"git {args:?}: {}",
String::from_utf8_lossy(&out.stderr)
);
}
/// Returns (work checkout, bare remote path).
fn seed() -> (tempfile::TempDir, std::path::PathBuf, std::path::PathBuf) {
let tmp = tempfile::tempdir().unwrap();
let remote = tmp.path().join("remote.git");
let work = tmp.path().join("work");
std::process::Command::new("git")
.args(["init", "--quiet", "--bare"])
.arg(&remote)
.output()
.unwrap();
std::fs::create_dir_all(&work).unwrap();
git(&work, &["init", "--quiet"]);
git(&work, &["checkout", "-q", "-B", "main"]);
std::fs::write(work.join("README.md"), "# vault\n").unwrap();
git(&work, &["add", "."]);
git(&work, &["commit", "--quiet", "-m", "base"]);
git(&work, &["remote", "add", "origin", remote.to_str().unwrap()]);
git(&work, &["push", "--quiet", "origin", "main"]);
(tmp, work, remote)
}
#[tokio::test]
async fn a_purely_additive_branch_is_merged() {
let (_tmp, work, remote) = seed();
git(&work, &["checkout", "-q", "-B", "lib/add"]);
std::fs::create_dir_all(work.join("60 Papers")).unwrap();
std::fs::write(work.join("60 Papers/a.md"), "# paper\n").unwrap();
git(&work, &["add", "."]);
git(&work, &["commit", "--quiet", "-m", "add paper"]);
git(&work, &["push", "--quiet", "origin", "lib/add"]);
let out = auto_merge::try_merge(
&work, remote.to_str().unwrap(), "lib/add", "main",
MergePolicy::AdditiveOnly, true,
)
.await
.unwrap();
assert!(out.merged, "should have merged: {}", out.reason);
// The note must really be on main at the remote, not just locally.
let ls = std::process::Command::new("git")
.arg("-C").arg(&remote)
.args(["ls-tree", "--name-only", "-r", "main"])
.output()
.unwrap();
let listed = String::from_utf8_lossy(&ls.stdout);
assert!(listed.contains("60 Papers/a.md"), "remote main: {listed}");
}
/// The load-bearing refusal: a branch that rewrites an existing file must be
/// left for a human even though its mission type is allowed to auto-merge.
#[tokio::test]
async fn a_branch_that_modifies_an_existing_file_is_refused() {
let (_tmp, work, remote) = seed();
git(&work, &["checkout", "-q", "-B", "lib/bad"]);
std::fs::create_dir_all(work.join("60 Papers")).unwrap();
std::fs::write(work.join("60 Papers/a.md"), "# paper\n").unwrap();
// …and clobbers a hand-written file.
std::fs::write(work.join("README.md"), "# REWRITTEN BY A MACHINE\n").unwrap();
git(&work, &["add", "."]);
git(&work, &["commit", "--quiet", "-m", "add + clobber"]);
git(&work, &["push", "--quiet", "origin", "lib/bad"]);
let out = auto_merge::try_merge(
&work, remote.to_str().unwrap(), "lib/bad", "main",
MergePolicy::AdditiveOnly, true,
)
.await
.unwrap();
assert!(!out.merged, "must refuse a non-additive branch");
assert!(out.reason.contains("not additive"), "reason: {}", out.reason);
let show = std::process::Command::new("git")
.arg("-C").arg(&remote)
.args(["show", "main:README.md"])
.output()
.unwrap();
assert_eq!(
String::from_utf8_lossy(&show.stdout),
"# vault\n",
"the hand-written file must be untouched on main"
);
}
#[tokio::test]
async fn unverified_work_is_never_merged() {
let (_tmp, work, remote) = seed();
git(&work, &["checkout", "-q", "-B", "lib/unverified"]);
std::fs::create_dir_all(work.join("60 Papers")).unwrap();
std::fs::write(work.join("60 Papers/a.md"), "# paper\n").unwrap();
git(&work, &["add", "."]);
git(&work, &["commit", "--quiet", "-m", "add"]);
git(&work, &["push", "--quiet", "origin", "lib/unverified"]);
let out = auto_merge::try_merge(
&work, remote.to_str().unwrap(), "lib/unverified", "main",
MergePolicy::AdditiveOnly, false,
)
.await
.unwrap();
assert!(!out.merged);
assert!(out.reason.contains("did not verify"), "reason: {}", out.reason);
}
#[tokio::test]
async fn a_never_policy_branch_is_left_alone() {
let (_tmp, work, remote) = seed();
git(&work, &["checkout", "-q", "-B", "code/change"]);
std::fs::write(work.join("new.rs"), "fn main() {}\n").unwrap();
git(&work, &["add", "."]);
git(&work, &["commit", "--quiet", "-m", "code"]);
git(&work, &["push", "--quiet", "origin", "code/change"]);
let out = auto_merge::try_merge(
&work, remote.to_str().unwrap(), "code/change", "main",
MergePolicy::Never, true,
)
.await
.unwrap();
assert!(!out.merged, "code must never auto-merge");
}
+286
View File
@@ -0,0 +1,286 @@
//! Indexing the vault must be idempotent, or a continuous mission cannot tell
//! new work from work it already did.
//!
//! These run against a real Postgres via cm-testkit. The vault fixture is
//! shaped from the actual `valhalla-vault`: 416 notes, only 145 with
//! frontmatter, none carrying arxiv/doi/url, plus repo-sync notes whose
//! frontmatter churns on every sync.
use cm_api::corpus;
use uuid::Uuid;
async fn workspace(pool: &sqlx::PgPool) -> Uuid {
let ws = Uuid::now_v7();
sqlx::query("INSERT INTO workspaces (id, name, plan) VALUES ($1,'t','team')")
.bind(ws)
.execute(pool)
.await
.unwrap();
ws
}
/// A real mission row. `corpus_items.mission_id` has a foreign key, which is
/// deliberate: attribution to a mission that does not exist is not
/// attribution. The first version of the test below used a bare UUID and was
/// correctly rejected.
async fn mission(pool: &sqlx::PgPool, ws: Uuid) -> Uuid {
let id = Uuid::now_v7();
sqlx::query(
"INSERT INTO missions (id, workspace_id, title, template_kind, schedule, status, config)
VALUES ($1,$2,'library','research_only','{}'::jsonb,'running','{}'::jsonb)",
)
.bind(id)
.bind(ws)
.execute(pool)
.await
.unwrap();
id
}
fn seed_vault(root: &std::path::Path) {
std::fs::create_dir_all(root.join("50 APESS 2026/Lectures")).unwrap();
std::fs::create_dir_all(root.join("Repos")).unwrap();
std::fs::create_dir_all(root.join("Daily")).unwrap();
// Course note: has frontmatter, but `source:` is a local path.
std::fs::write(
root.join("50 APESS 2026/Lectures/agentic.md"),
"---\nsource: \"/Users/quantum/Downloads/Material/x.pdf\"\ntype: lecture\n---\n# Agentic Design\n\nbody\n",
)
.unwrap();
// Repo-sync note: frontmatter churns, prose does not.
std::fs::write(
root.join("Repos/zeroclaw.md"),
"---\nnode: tank\nupdated: 2026-08-01\nsize_kb: 12\n---\n# ZeroClaw\n\nmirror\n",
)
.unwrap();
// Plain note: no frontmatter at all — the majority case.
std::fs::write(root.join("Daily/2026-08-01.md"), "# Monday\n\nnotes\n").unwrap();
}
#[tokio::test]
async fn indexing_an_unchanged_vault_is_a_no_op() {
let pool = cm_testkit::test_pool().await;
let ws = workspace(&pool).await;
let tmp = tempfile::tempdir().unwrap();
seed_vault(tmp.path());
let first = corpus::index_vault(&pool, ws, "vault", tmp.path())
.await
.unwrap();
assert_eq!(first.scanned, 3);
assert_eq!(first.inserted, 3);
assert_eq!(first.unchanged, 0);
// The decisive assertion: a second pass over an untouched vault must add
// and change nothing. Without this, every run looks like new work.
let second = corpus::index_vault(&pool, ws, "vault", tmp.path())
.await
.unwrap();
assert_eq!(second.scanned, 3);
assert_eq!(second.inserted, 0, "re-index must not insert");
assert_eq!(second.updated, 0, "re-index must not update");
assert_eq!(second.unchanged, 3);
}
#[tokio::test]
async fn a_repo_sync_touching_only_frontmatter_is_not_an_edit() {
let pool = cm_testkit::test_pool().await;
let ws = workspace(&pool).await;
let tmp = tempfile::tempdir().unwrap();
seed_vault(tmp.path());
corpus::index_vault(&pool, ws, "vault", tmp.path())
.await
.unwrap();
// Exactly what a repo sync does: bump `updated`/`size_kb`, prose untouched.
std::fs::write(
tmp.path().join("Repos/zeroclaw.md"),
"---\nnode: tank\nupdated: 2026-08-03\nsize_kb: 14\n---\n# ZeroClaw\n\nmirror\n",
)
.unwrap();
let stats = corpus::index_vault(&pool, ws, "vault", tmp.path())
.await
.unwrap();
assert_eq!(stats.updated, 0, "frontmatter churn is not an edit");
assert_eq!(stats.unchanged, 3);
// A real prose edit must still be seen.
std::fs::write(
tmp.path().join("Repos/zeroclaw.md"),
"---\nnode: tank\nupdated: 2026-08-03\n---\n# ZeroClaw\n\nREWRITTEN\n",
)
.unwrap();
let stats = corpus::index_vault(&pool, ws, "vault", tmp.path())
.await
.unwrap();
assert_eq!(stats.updated, 1, "a genuine edit must be visible");
}
#[tokio::test]
async fn a_hand_edited_note_survives_a_rebuild() {
let pool = cm_testkit::test_pool().await;
let ws = workspace(&pool).await;
let tmp = tempfile::tempdir().unwrap();
seed_vault(tmp.path());
corpus::index_vault(&pool, ws, "vault", tmp.path())
.await
.unwrap();
// The vault is authoritative: a human renames a note by hand.
std::fs::remove_file(tmp.path().join("Daily/2026-08-01.md")).unwrap();
std::fs::write(tmp.path().join("Daily/renamed.md"), "# Monday\n\nnotes\n").unwrap();
let stats = corpus::index_vault(&pool, ws, "vault", tmp.path())
.await
.unwrap();
assert_eq!(stats.scanned, 3);
assert_eq!(stats.inserted, 1, "the renamed note is indexed under its new path");
// The stale row is left alone rather than deleted — the index is derived
// and rebuildable, and losing coverage history is worse than a stale row.
assert!(corpus::seen(&pool, ws, "vault", "note:Daily/renamed.md")
.await
.unwrap());
}
#[tokio::test]
async fn unseen_filters_candidates_in_one_round_trip() {
let pool = cm_testkit::test_pool().await;
let ws = workspace(&pool).await;
corpus::record(
&pool, ws, "vault", "source", "arxiv:2401.11111",
Some("Known"), None, None, "h", None,
)
.await
.unwrap();
let candidates = vec![
"arxiv:2401.11111".to_string(), // already read
"arxiv:2401.22222".to_string(),
"doi:10.1000/new".to_string(),
];
let fresh = corpus::unseen(&pool, ws, "vault", &candidates).await.unwrap();
assert_eq!(fresh, vec!["arxiv:2401.22222", "doi:10.1000/new"]);
assert!(corpus::seen(&pool, ws, "vault", "arxiv:2401.11111").await.unwrap());
assert!(!corpus::seen(&pool, ws, "vault", "arxiv:2401.22222").await.unwrap());
// A different corpus must not inherit another's seen-set.
assert!(!corpus::seen(&pool, ws, "other", "arxiv:2401.11111").await.unwrap());
}
/// The first mission to find a source keeps the credit, so "did THIS run
/// contribute anything new" stays answerable across repeated runs.
#[tokio::test]
async fn re_recording_a_source_does_not_reassign_it() {
let pool = cm_testkit::test_pool().await;
let ws = workspace(&pool).await;
let inserted = corpus::record(
&pool, ws, "vault", "source", "arxiv:2401.33333",
Some("Paper"), None, None, "h1", None,
)
.await
.unwrap();
assert!(inserted, "first sighting is an insert");
let inserted_again = corpus::record(
&pool, ws, "vault", "source", "arxiv:2401.33333",
Some("Paper"), None, None, "h2", None,
)
.await
.unwrap();
assert!(!inserted_again, "a second sighting is not new work");
}
/// Idempotence against the real vault rather than a fixture.
///
/// Ignored by default because it needs a checkout: run with
/// `VAULT=/path/to/valhalla-vault cargo test -p cm-api --test corpus_vault \
/// index_the_real_vault -- --ignored --nocapture`.
///
/// Measured 2026-08-03 on the live vault:
/// PASS1 { scanned: 416, inserted: 416, updated: 0, unchanged: 0 }
/// PASS2 { scanned: 416, inserted: 0, updated: 0, unchanged: 416 }
#[tokio::test]
#[ignore]
async fn index_the_real_vault() {
let pool = cm_testkit::test_pool().await;
let ws = uuid::Uuid::now_v7();
sqlx::query("INSERT INTO workspaces (id, name, plan) VALUES ($1,'t','team')")
.bind(ws).execute(&pool).await.unwrap();
let root = std::path::Path::new(&std::env::var("VAULT").unwrap()).to_path_buf();
let a = cm_api::corpus::index_vault(&pool, ws, "valhalla-vault", &root).await.unwrap();
println!("PASS1 {a:?}");
let b = cm_api::corpus::index_vault(&pool, ws, "valhalla-vault", &root).await.unwrap();
println!("PASS2 {b:?}");
assert_eq!(b.inserted, 0);
assert_eq!(b.updated, 0);
assert_eq!(b.unchanged, a.scanned);
}
/// Live arXiv check. Ignored by default (needs network); run with
/// `cargo test -p cm-api --test corpus_vault live_arxiv -- --ignored --nocapture`.
///
/// Guards the one failure that hides: if arXiv's feed format drifts, parsing
/// returns zero papers, which looks exactly like "no new papers this week".
#[tokio::test]
#[ignore]
async fn live_arxiv_search_and_fetch() {
let papers = cm_api::papers::search("all:agentic topologies", 3)
.await
.expect("arxiv search");
println!("found {} papers", papers.len());
assert!(!papers.is_empty(), "arXiv returned nothing — format drift?");
for p in &papers {
println!(" {} | {}", p.source_id(), &p.title[..p.title.len().min(60)]);
assert!(!p.arxiv_id.is_empty());
assert!(!p.title.is_empty());
assert!(!p.arxiv_id.contains('v'), "version must be stripped: {}", p.arxiv_id);
}
let pdf = cm_api::papers::fetch_pdf(&papers[0]).await.expect("fetch pdf");
println!("pdf bytes: {}", pdf.len());
assert!(pdf.starts_with(b"%PDF"));
assert!(pdf.len() > 10_000, "suspiciously small pdf: {}", pdf.len());
}
/// A rerun must not be able to claim credit for work an earlier run did.
///
/// This is the verification predicate for a continuous mission: "did THIS run
/// contribute anything new". If a rerun could re-record an existing source
/// under its own mission id, every run would report success forever — the
/// failure that killed the 0030-0044 generation of this feature.
#[tokio::test]
async fn a_rerun_cannot_claim_an_earlier_missions_work() {
let pool = cm_testkit::test_pool().await;
let ws = workspace(&pool).await;
let first_mission = mission(&pool, ws).await;
let second_mission = mission(&pool, ws).await;
corpus::record(
&pool, ws, "lib", "source", "arxiv:2401.55555",
Some("Paper"), None, None, "h1", Some(first_mission),
)
.await
.unwrap();
// The second mission sees the same paper and re-records it.
corpus::record(
&pool, ws, "lib", "source", "arxiv:2401.55555",
Some("Paper"), None, None, "h2", Some(second_mission),
)
.await
.unwrap();
assert_eq!(
corpus::contributed(&pool, ws, "lib", first_mission).await.unwrap(),
1,
"the finder keeps the credit"
);
assert_eq!(
corpus::contributed(&pool, ws, "lib", second_mission).await.unwrap(),
0,
"a rerun that found nothing new must report zero, not one"
);
}
+202
View File
@@ -0,0 +1,202 @@
//! A second run must not re-download what the first run already shelved.
use cm_api::{corpus, harvest, papers::Paper};
use std::sync::Arc;
use uuid::Uuid;
async fn workspace(pool: &sqlx::PgPool) -> Uuid {
let ws = Uuid::now_v7();
sqlx::query("INSERT INTO workspaces (id, name, plan) VALUES ($1,'t','team')")
.bind(ws)
.execute(pool)
.await
.unwrap();
ws
}
fn paper(id: &str) -> Paper {
Paper {
arxiv_id: id.into(),
title: format!("Paper {id}"),
authors: vec!["Ada Lovelace".into()],
summary: "A summary.".into(),
published: "2026-01-15T10:00:00Z".into(),
// Deliberately unreachable: if the skip works, this is never fetched.
pdf_url: "http://127.0.0.1:1/never.pdf".into(),
}
}
/// The load-bearing behaviour. Every candidate is already on the checkmark
/// list, and every `pdf_url` points at a closed port — so if the run tries to
/// download anything at all, it fails loudly instead of passing quietly.
#[tokio::test]
async fn papers_we_already_hold_are_never_downloaded_again() {
let pool = cm_testkit::test_pool().await;
let ws = workspace(&pool).await;
let tmp = tempfile::tempdir().unwrap();
let blobs: Arc<dyn cm_files::BlobStore> =
Arc::new(cm_files::LocalBlobStore::new(tmp.path().join("blobs")));
let vault = tmp.path().join("vault");
let candidates = vec![paper("2401.11111"), paper("2401.22222")];
for p in &candidates {
corpus::record(
&pool, ws, "lib", "source", &p.source_id(),
Some(&p.title), None, None, "h", None,
)
.await
.unwrap();
}
let lib = harvest::Library {
pool: &pool, blobs: &blobs, workspace_id: ws,
corpus_id: "lib", vault_root: &vault,
};
let h = harvest::shelve(&lib, &candidates, None).await.unwrap();
assert_eq!(h.candidates, 2);
assert_eq!(h.already_had, 2, "both were already held");
assert!(h.shelved.is_empty());
assert!(
h.failed.is_empty(),
"nothing should have been fetched at all, but got: {:?}",
h.failed
);
assert!(h.healthy(), "a fully-known batch is a healthy quiet week");
assert!(!h.added_anything(), "and it added nothing");
assert!(!vault.exists(), "no notes written for papers we already had");
}
/// A paper that cannot be downloaded must NOT be checked off — otherwise one
/// transient network failure means that paper is never retried.
#[tokio::test]
async fn a_failed_download_leaves_the_paper_unseen_for_next_time() {
let pool = cm_testkit::test_pool().await;
let ws = workspace(&pool).await;
let tmp = tempfile::tempdir().unwrap();
let blobs: Arc<dyn cm_files::BlobStore> =
Arc::new(cm_files::LocalBlobStore::new(tmp.path().join("blobs")));
let vault = tmp.path().join("vault");
let candidates = vec![paper("2401.33333")];
let lib = harvest::Library {
pool: &pool, blobs: &blobs, workspace_id: ws,
corpus_id: "lib", vault_root: &vault,
};
let h = harvest::shelve(&lib, &candidates, None).await.unwrap();
assert_eq!(h.already_had, 0);
assert!(h.shelved.is_empty());
assert_eq!(h.failed.len(), 1, "the unreachable fetch must be reported");
assert!(!h.healthy(), "a failed fetch is not a quiet week");
assert!(
!corpus::seen(&pool, ws, "lib", "arxiv:2401.33333")
.await
.unwrap(),
"a paper we failed to get must stay unseen so a later run retries it"
);
}
/// Live end-to-end: search arXiv, shelve genuinely new papers, then confirm a
/// second identical run adds nothing. Ignored by default (network + Postgres):
/// `cargo test -p cm-api --test harvest_run live_ -- --ignored --nocapture`
#[tokio::test]
#[ignore]
async fn live_end_to_end_run_then_rerun() {
let pool = cm_testkit::test_pool().await;
let ws = workspace(&pool).await;
let tmp = tempfile::tempdir().unwrap();
let blobs: Arc<dyn cm_files::BlobStore> =
Arc::new(cm_files::LocalBlobStore::new(tmp.path().join("blobs")));
let vault = tmp.path().join("vault");
let lib = harvest::Library {
pool: &pool, blobs: &blobs, workspace_id: ws,
corpus_id: "lib", vault_root: &vault,
};
let first = harvest::run(&lib, "all:agentic AND all:topology", 3, None)
.await
.unwrap();
println!("RUN1 {}", first.summary());
for n in &first.notes_written {
println!(" note: {n}");
}
assert!(first.healthy(), "failures: {:?}", first.failed);
assert!(first.added_anything(), "first run should find something new");
// Every note must be readable back through the corpus parser, or the
// catalogue cannot rebuild the checkmark list.
for rel in &first.notes_written {
let text = std::fs::read_to_string(vault.join(rel)).unwrap();
let parsed = corpus::parse_note(rel, &text);
assert!(
parsed
.declared_source_id
.as_deref()
.is_some_and(|s| s.starts_with("arxiv:")),
"note {rel} lost its identity"
);
}
let second = harvest::run(&lib, "all:agentic AND all:topology", 3, None)
.await
.unwrap();
println!("RUN2 {}", second.summary());
assert!(second.healthy());
assert!(
!second.added_anything(),
"a rerun must add nothing — got {:?}",
second.shelved
);
assert_eq!(second.already_had, second.candidates);
}
/// THE REAL RUN. Clones the live vault, harvests our current topics, pushes a
/// branch. Ignored by default — needs network, Postgres and GITEA_TOKEN:
/// `GITEA_TOKEN=… VAULT_URL=… cargo test -p cm-api --test harvest_run \
/// live_library_run -- --ignored --nocapture`
#[tokio::test]
#[ignore]
async fn live_library_run() {
let pool = cm_testkit::test_pool().await;
let ws = workspace(&pool).await;
let tmp = tempfile::tempdir().unwrap();
let blobs: Arc<dyn cm_files::BlobStore> =
Arc::new(cm_files::LocalBlobStore::new(tmp.path().join("shelf")));
let url = std::env::var("VAULT_URL").unwrap();
let topics = cm_api::library::default_topics();
for t in &topics {
println!("topic: {t}");
}
let run = cm_api::library::run_to_vault(
&pool, &blobs, ws, "valhalla-vault", &url,
tmp.path(), &topics, 2, None,
)
.await
.unwrap();
println!("\nRESULT {}", run.harvest.summary());
println!("branch: {} pushed: {}", run.branch, run.pushed);
if let Some(e) = &run.error {
println!("error: {e}");
}
for n in &run.harvest.notes_written {
println!(" note: {n}");
}
for (sid, why) in &run.harvest.failed {
println!(" FAILED {sid}: {why}");
}
// Every shelved paper must have its PDF really on the shelf.
for sid in &run.harvest.shelved {
let id = sid.trim_start_matches("arxiv:");
let key = format!("papers/arxiv/{id}.pdf");
let bytes = blobs.get(&key).await.expect("pdf on the shelf");
assert!(bytes.starts_with(b"%PDF"), "{key} is not a PDF");
println!(" shelf: {key} ({} bytes)", bytes.len());
}
assert!(run.harvest.healthy(), "failures: {:?}", run.harvest.failed);
}
+918
View File
@@ -0,0 +1,918 @@
//! Diff capture, against a real git repository.
//!
//! Deliberately not mocked. Every bug this area has produced came from git
//! behaving differently than assumed — a shallow clone refusing a push, an
//! ownership check refusing the repo, untracked files invisible to `git diff`.
//! A fake `git` would agree with whatever the code believed and prove nothing.
use std::path::Path;
use std::process::Command;
use cm_api::mission_delivery;
use uuid::Uuid;
/// Capture against an explicit root, so parallel tests cannot race each other
/// through the process-global `CLAWMATES_MISSIONS_ROOT`.
async fn capture(
pool: &sqlx::PgPool,
root: &Path,
mission: Uuid,
phase: Uuid,
) -> Result<Option<mission_delivery::Capture>, String> {
mission_delivery::capture_phase_diff_at(
pool,
mission,
phase,
&root.join(mission.to_string()).join("repo"),
&root.join("_outputs").join(mission.to_string()),
0,
mission_delivery::Gate::Always,
)
.await
}
fn git(repo: &Path, args: &[&str]) {
let out = Command::new("git")
.arg("-C")
.arg(repo)
.args(args)
.output()
.expect("run git");
assert!(
out.status.success(),
"git {args:?} failed: {}",
String::from_utf8_lossy(&out.stderr)
);
}
/// A repo with one commit, at `<root>/<mission>/repo` so `checkout_path`
/// finds it.
/// Record the clone point the way `mission_workspace` does after a clone.
fn record_base(repo: &Path) {
let out = Command::new("git")
.arg("-C")
.arg(repo)
.args(["rev-parse", "HEAD"])
.output()
.unwrap();
std::fs::write(
repo.join(".git/clawmates-base"),
String::from_utf8_lossy(&out.stdout).trim(),
)
.unwrap();
}
fn seed_repo(root: &Path, mission: Uuid) -> std::path::PathBuf {
let repo = root.join(mission.to_string()).join("repo");
std::fs::create_dir_all(&repo).unwrap();
git(&repo, &["init", "--quiet"]);
git(&repo, &["config", "user.email", "[email protected]"]);
git(&repo, &["config", "user.name", "Test"]);
std::fs::write(repo.join("README.md"), "# base\n").unwrap();
git(&repo, &["add", "."]);
git(&repo, &["commit", "--quiet", "-m", "base"]);
// The real clone path applies this; seeding a repo by hand and skipping it
// is what let the uid-split failure reach production untested.
cm_api::mission_workspace::share_repository_across_uids(&repo);
record_base(&repo);
repo
}
#[tokio::test]
async fn captures_modified_and_untracked_files() {
let pool = cm_testkit::test_pool().await;
let tmp = tempfile::tempdir().unwrap();
let mission = Uuid::now_v7();
let repo = seed_repo(tmp.path(), mission);
let (ws, phase) = seed_mission_phase(&pool, mission).await;
let _ = ws;
// A modified file and a brand-new one. The new file is the case that
// matters: without `--intent-to-add` it would not appear in `git diff`,
// and a phase that only creates files is the likeliest shape of all.
std::fs::write(repo.join("README.md"), "# base\nchanged\n").unwrap();
std::fs::write(repo.join("new_module.rs"), "fn added() {}\n").unwrap();
let cap = capture(&pool, tmp.path(), mission, phase)
.await
.unwrap()
.expect("mission has a checkout");
assert!(!cap.empty, "the phase changed files");
assert_eq!(cap.files_changed, 2, "one modified, one created");
assert!(cap.insertions >= 2);
let patch = std::fs::read_to_string(&cap.patch_path).unwrap();
assert!(
patch.contains("new_module.rs"),
"untracked file is captured"
);
assert!(patch.contains("fn added()"), "its content is captured");
assert!(patch.contains("changed"), "the modification is captured");
// Capture is followed by a commit, so the tree the agents left is now on a
// branch of its own. The patch was written first and is what guarantees
// the work survives; the branch is the convenience on top.
let branch = Command::new("git")
.arg("-C")
.arg(&repo)
.args(["rev-parse", "--abbrev-ref", "HEAD"])
.output()
.unwrap();
let branch = String::from_utf8_lossy(&branch.stdout).trim().to_string();
assert!(
branch.starts_with("clawmates/mission-"),
"work lands on a namespaced mission branch, never the default one: {branch}"
);
let status = Command::new("git")
.arg("-C")
.arg(&repo)
.args(["status", "--porcelain"])
.output()
.unwrap();
assert!(
String::from_utf8_lossy(&status.stdout).trim().is_empty(),
"everything the phase produced is committed, nothing left dangling"
);
let show = Command::new("git")
.arg("-C")
.arg(&repo)
.args(["show", "--stat", "--oneline", "HEAD"])
.output()
.unwrap();
let show = String::from_utf8_lossy(&show.stdout);
assert!(
show.contains("new_module.rs"),
"the created file is in the commit: {show}"
);
assert!(
cap.committed.is_some(),
"the capture records where the work landed"
);
let row: (String, serde_json::Value) = sqlx::query_as(
"SELECT kind, metadata FROM mission_artifacts WHERE mission_id = $1 AND phase_id = $2",
)
.bind(mission)
.bind(phase)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(row.0, "code_diff");
assert_eq!(row.1["files_changed"], 2);
assert_eq!(row.1["empty"], false);
assert!(row.1["base_sha"].as_str().unwrap().len() >= 7);
}
/// "This coding phase wrote no code" is a result, and currently an invisible
/// one. It must still produce an artifact.
#[tokio::test]
async fn an_empty_phase_still_produces_an_artifact() {
let pool = cm_testkit::test_pool().await;
let tmp = tempfile::tempdir().unwrap();
let mission = Uuid::now_v7();
seed_repo(tmp.path(), mission);
let (_, phase) = seed_mission_phase(&pool, mission).await;
let cap = capture(&pool, tmp.path(), mission, phase)
.await
.unwrap()
.unwrap();
assert!(cap.empty);
assert_eq!(cap.files_changed, 0);
let (kind, meta): (String, serde_json::Value) =
sqlx::query_as("SELECT kind, metadata FROM mission_artifacts WHERE mission_id = $1")
.bind(mission)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(kind, "code_diff");
assert_eq!(meta["empty"], true, "the emptiness is recorded, not hidden");
}
/// Build output must never reach the patch. A phase that ran `cargo build`
/// leaves a `target/` larger than the repository, and committing it would be
/// worse than losing the diff.
#[tokio::test]
async fn build_output_is_not_captured() {
let pool = cm_testkit::test_pool().await;
let tmp = tempfile::tempdir().unwrap();
let mission = Uuid::now_v7();
let repo = seed_repo(tmp.path(), mission);
let (_, phase) = seed_mission_phase(&pool, mission).await;
std::fs::create_dir_all(repo.join("target/debug")).unwrap();
std::fs::write(repo.join("target/debug/huge.bin"), vec![b'x'; 200_000]).unwrap();
std::fs::create_dir_all(repo.join("node_modules/left-pad")).unwrap();
std::fs::write(
repo.join("node_modules/left-pad/index.js"),
"module.exports=0",
)
.unwrap();
std::fs::write(repo.join("real_change.rs"), "fn kept() {}\n").unwrap();
let cap = capture(&pool, tmp.path(), mission, phase)
.await
.unwrap()
.unwrap();
let patch = std::fs::read_to_string(&cap.patch_path).unwrap();
assert!(patch.contains("real_change.rs"), "genuine work is captured");
assert!(!patch.contains("huge.bin"), "target/ is excluded");
assert!(!patch.contains("left-pad"), "node_modules is excluded");
assert_eq!(cap.files_changed, 1, "only the real change counts");
}
/// A research mission has no checkout. That is not an error.
#[tokio::test]
async fn a_mission_without_a_checkout_captures_nothing() {
let pool = cm_testkit::test_pool().await;
let tmp = tempfile::tempdir().unwrap();
let mission = Uuid::now_v7();
let (_, phase) = seed_mission_phase(&pool, mission).await;
assert!(capture(&pool, tmp.path(), mission, phase)
.await
.unwrap()
.is_none());
}
async fn seed_mission_phase(pool: &sqlx::PgPool, mission: Uuid) -> (Uuid, Uuid) {
let ws = Uuid::now_v7();
sqlx::query("INSERT INTO workspaces (id, name, plan) VALUES ($1,'t','team')")
.bind(ws)
.execute(pool)
.await
.unwrap();
sqlx::query(
"INSERT INTO missions (id, workspace_id, title, template_kind, schedule, status, config)
VALUES ($1,$2,'t','research_and_code','{}'::jsonb,'running','{}'::jsonb)",
)
.bind(mission)
.bind(ws)
.execute(pool)
.await
.unwrap();
let phase = Uuid::now_v7();
sqlx::query(
"INSERT INTO mission_phases (id, mission_id, kind, order_idx, status, config)
VALUES ($1,$2,'coding',0,'completed','{}'::jsonb)",
)
.bind(phase)
.bind(mission)
.execute(pool)
.await
.unwrap();
(ws, phase)
}
/// Add a second phase to a mission `seed_mission_phase` already created.
async fn seed_extra_phase(pool: &sqlx::PgPool, mission: Uuid, order_idx: i32) -> Uuid {
let phase = Uuid::now_v7();
sqlx::query(
"INSERT INTO mission_phases (id, mission_id, kind, order_idx, status, config)
VALUES ($1,$2,'coding',$3,'completed','{}'::jsonb)",
)
.bind(phase)
.bind(mission)
.bind(order_idx)
.execute(pool)
.await
.unwrap();
phase
}
/// The production failure this pairs with. Mission 019fc372's agent created
/// the file it was asked for and *committed* it — `rust_sdlc` has a committer
/// role, so that is the intended path — leaving a clean working tree. Capture
/// diffed against HEAD, found nothing, and recorded `empty: true` next to a
/// commit that plainly contained the work.
#[tokio::test]
async fn work_the_agent_committed_is_captured() {
let pool = cm_testkit::test_pool().await;
let tmp = tempfile::tempdir().unwrap();
let mission = Uuid::now_v7();
let repo = seed_repo(tmp.path(), mission);
let (_, phase) = seed_mission_phase(&pool, mission).await;
std::fs::write(repo.join("DELIVERY_PROBE.md"), "CAPTURED-BY-CLAWMATES\n").unwrap();
git(&repo, &["add", "DELIVERY_PROBE.md"]);
git(&repo, &["commit", "--quiet", "-m", "Add DELIVERY_PROBE.md"]);
// The tree is clean — `git status --porcelain` is empty here, which is
// precisely why the HEAD-relative version saw nothing.
let status = Command::new("git")
.arg("-C")
.arg(&repo)
.args(["status", "--porcelain"])
.output()
.unwrap();
assert!(
String::from_utf8_lossy(&status.stdout).trim().is_empty(),
"the agent committed, so the tree is clean"
);
let cap = capture(&pool, tmp.path(), mission, phase)
.await
.unwrap()
.unwrap();
assert!(
!cap.empty,
"committed work must be captured, not reported as empty"
);
assert_eq!(cap.files_changed, 1);
let patch = std::fs::read_to_string(&cap.patch_path).unwrap();
assert!(
patch.contains("CAPTURED-BY-CLAWMATES"),
"the committed content is in the patch"
);
}
/// Committed *and* uncommitted work in the same phase — a coder that committed
/// one change and left another in progress.
#[tokio::test]
async fn committed_and_uncommitted_changes_are_both_captured() {
let pool = cm_testkit::test_pool().await;
let tmp = tempfile::tempdir().unwrap();
let mission = Uuid::now_v7();
let repo = seed_repo(tmp.path(), mission);
let (_, phase) = seed_mission_phase(&pool, mission).await;
std::fs::write(repo.join("committed.rs"), "fn done() {}\n").unwrap();
git(&repo, &["add", "committed.rs"]);
git(&repo, &["commit", "--quiet", "-m", "first"]);
std::fs::write(repo.join("in_progress.rs"), "fn wip() {}\n").unwrap();
let cap = capture(&pool, tmp.path(), mission, phase)
.await
.unwrap()
.unwrap();
let patch = std::fs::read_to_string(&cap.patch_path).unwrap();
assert!(patch.contains("fn done()"), "committed work");
assert!(patch.contains("fn wip()"), "uncommitted work");
assert_eq!(cap.files_changed, 2);
}
/// Build output must stay out of the commit as well as the patch. Putting a
/// `target/` directory into someone's history is worse than losing the diff.
#[tokio::test]
async fn excluded_paths_are_not_committed() {
let pool = cm_testkit::test_pool().await;
let tmp = tempfile::tempdir().unwrap();
let mission = Uuid::now_v7();
let repo = seed_repo(tmp.path(), mission);
let (_, phase) = seed_mission_phase(&pool, mission).await;
std::fs::create_dir_all(repo.join("target/debug")).unwrap();
std::fs::write(repo.join("target/debug/blob.bin"), vec![b'x'; 50_000]).unwrap();
std::fs::write(
repo.join(".gitconfig_temp"),
"[safe]\n\tdirectory = /mission/repo\n",
)
.unwrap();
std::fs::write(repo.join("real.rs"), "fn kept() {}\n").unwrap();
capture(&pool, tmp.path(), mission, phase)
.await
.unwrap()
.unwrap();
let tracked = Command::new("git")
.arg("-C")
.arg(&repo)
.args(["ls-files"])
.output()
.unwrap();
let tracked = String::from_utf8_lossy(&tracked.stdout);
assert!(tracked.contains("real.rs"), "genuine work is committed");
assert!(
!tracked.contains("blob.bin"),
"build output is not committed"
);
assert!(
!tracked.contains(".gitconfig_temp"),
"an agent's workaround file is not committed into the user's history"
);
}
/// Sibling phases of one mission must not share a branch.
///
/// They did. Both ids are UUIDv7, which leads with a timestamp, so two phases
/// created in the same millisecond had identical leading hex and the name
/// collapsed to one branch per mission — each phase quietly moving the ref the
/// previous one had set. Production showed
/// `clawmates/mission-019fc40e-019fc40e` for both phases of a mission.
#[test]
fn sibling_phases_get_distinct_branches() {
let mission = Uuid::now_v7();
// Minted back to back, so they share a timestamp prefix exactly as they do
// when a mission inserts its phases in one transaction.
let research = Uuid::now_v7();
let coding = Uuid::now_v7();
assert_eq!(
research.simple().to_string()[..8],
coding.simple().to_string()[..8],
"precondition: v7 ids minted together share their leading hex"
);
let a = mission_delivery::branch_name(mission, research, 0);
let b = mission_delivery::branch_name(mission, coding, 0);
assert_ne!(a, b, "each phase needs its own ref: {a} vs {b}");
assert!(a.starts_with("clawmates/mission-"));
}
/// A re-run must not collide with the pass before it.
#[test]
fn a_rerun_lands_on_its_own_branch() {
let m = Uuid::now_v7();
let p = Uuid::now_v7();
let first = mission_delivery::branch_name(m, p, 0);
let second = mission_delivery::branch_name(m, p, 1);
assert_ne!(first, second);
assert!(first.starts_with("clawmates/mission-"));
assert!(
second.ends_with("-i2"),
"pass 2 is named for the pass, not the index: {second}"
);
}
/// Push against a real bare repository.
///
/// A mock remote would accept whatever we sent and prove nothing; the failures
/// worth catching here — a rejected ref, a branch that never arrives, work
/// pushed to the wrong name — are all things only a real git remote reports.
#[tokio::test]
async fn a_gated_push_reaches_the_remote() {
let tmp = tempfile::tempdir().unwrap();
let remote = tmp.path().join("remote.git");
std::fs::create_dir_all(&remote).unwrap();
Command::new("git")
.args(["init", "--bare", "--quiet"])
.arg(&remote)
.output()
.unwrap();
let mission = Uuid::now_v7();
let repo = seed_repo(tmp.path(), mission);
std::fs::write(repo.join("work.rs"), "fn shipped() {}\n").unwrap();
git(&repo, &["add", "."]);
git(&repo, &["commit", "--quiet", "-m", "work"]);
git(
&repo,
&["checkout", "-B", "clawmates/mission-test-aaaaaaaa"],
);
let out = mission_delivery::publish_phase_branch(
&repo,
remote.to_str().unwrap(),
"clawmates/mission-test-aaaaaaaa",
mission_delivery::Gate::Always,
None,
)
.await
.unwrap();
assert!(out.pushed, "push failed: {:?}", out.error);
assert_eq!(out.branch, "clawmates/mission-test-aaaaaaaa");
// The remote genuinely has it, with the content.
let refs = Command::new("git")
.arg("-C")
.arg(&remote)
.args(["for-each-ref", "--format=%(refname:short)"])
.output()
.unwrap();
let refs = String::from_utf8_lossy(&refs.stdout);
assert!(
refs.contains("clawmates/mission-test-aaaaaaaa"),
"refs: {refs}"
);
let show = Command::new("git")
.arg("-C")
.arg(&remote)
.args(["show", "clawmates/mission-test-aaaaaaaa:work.rs"])
.output()
.unwrap();
assert!(String::from_utf8_lossy(&show.stdout).contains("fn shipped()"));
}
/// A red suite must not block delivery — it must redirect it. The work still
/// reaches the forge, on a branch whose name says it is unproven.
#[tokio::test]
async fn a_failed_gate_publishes_to_a_wip_branch() {
let tmp = tempfile::tempdir().unwrap();
let remote = tmp.path().join("remote.git");
std::fs::create_dir_all(&remote).unwrap();
Command::new("git")
.args(["init", "--bare", "--quiet"])
.arg(&remote)
.output()
.unwrap();
let mission = Uuid::now_v7();
let repo = seed_repo(tmp.path(), mission);
std::fs::write(repo.join("half_done.rs"), "fn broken() {}\n").unwrap();
git(&repo, &["add", "."]);
git(&repo, &["commit", "--quiet", "-m", "wip"]);
git(
&repo,
&["checkout", "-B", "clawmates/mission-test-bbbbbbbb"],
);
let out = mission_delivery::publish_phase_branch(
&repo,
remote.to_str().unwrap(),
"clawmates/mission-test-bbbbbbbb",
mission_delivery::Gate::OnGreenTests,
Some(false),
)
.await
.unwrap();
assert!(out.pushed, "a failed gate still publishes: {:?}", out.error);
assert!(
out.branch.ends_with("-wip"),
"verdict is in the name: {}",
out.branch
);
let refs = Command::new("git")
.arg("-C")
.arg(&remote)
.args(["for-each-ref", "--format=%(refname:short)"])
.output()
.unwrap();
let refs = String::from_utf8_lossy(&refs.stdout);
assert!(
refs.contains("-wip"),
"the work reached the forge anyway: {refs}"
);
assert!(
!refs.contains("clawmates/mission-test-bbbbbbbb\n"),
"and did not claim the clean branch name"
);
}
/// An unreachable remote is a degraded success, not a failure: the patch and
/// the local branch both still exist.
#[tokio::test]
async fn an_unreachable_remote_does_not_lose_the_work() {
let tmp = tempfile::tempdir().unwrap();
let mission = Uuid::now_v7();
let repo = seed_repo(tmp.path(), mission);
std::fs::write(repo.join("work.rs"), "fn kept() {}\n").unwrap();
git(&repo, &["add", "."]);
git(&repo, &["commit", "--quiet", "-m", "work"]);
git(
&repo,
&["checkout", "-B", "clawmates/mission-test-cccccccc"],
);
let out = mission_delivery::publish_phase_branch(
&repo,
&tmp.path().join("does-not-exist.git").display().to_string(),
"clawmates/mission-test-cccccccc",
mission_delivery::Gate::Always,
None,
)
.await
.unwrap();
assert!(!out.pushed);
assert!(
out.error.is_some(),
"the reason is recorded for the operator"
);
// The commit is still there locally — nothing was rolled back.
let show = Command::new("git")
.arg("-C")
.arg(&repo)
.args(["show", "HEAD:work.rs"])
.output()
.unwrap();
assert!(String::from_utf8_lossy(&show.stdout).contains("fn kept()"));
}
/// A later phase must report its own work, not its predecessor's.
///
/// The capture base is recorded once at clone time. Left there, phase 2 diffs
/// against the original clone point and claims phase 1's commits as its own —
/// which is exactly what mission `019fc42b` produced: two coding phases, two
/// artifacts, and the second one reporting the union of both.
#[tokio::test]
async fn a_later_phase_reports_only_its_own_work() {
let pool = cm_testkit::test_pool().await;
let tmp = tempfile::tempdir().unwrap();
let mission = Uuid::now_v7();
let repo = seed_repo(tmp.path(), mission);
let (_, phase_one) = seed_mission_phase(&pool, mission).await;
let phase_two = seed_extra_phase(&pool, mission, 1).await;
std::fs::write(repo.join("ALPHA.md"), "ALPHA-DELIVERED\n").unwrap();
let first = capture(&pool, tmp.path(), mission, phase_one)
.await
.unwrap()
.unwrap();
assert_eq!(first.files_changed, 1, "phase one wrote one file");
std::fs::write(repo.join("BETA.md"), "BETA-DELIVERED\n").unwrap();
let second = capture(&pool, tmp.path(), mission, phase_two)
.await
.unwrap()
.unwrap();
assert_eq!(
second.files_changed, 1,
"phase two must report only BETA.md, not ALPHA.md as well"
);
let patch = std::fs::read_to_string(&second.patch_path).unwrap();
assert!(patch.contains("BETA-DELIVERED"), "phase two's own work");
assert!(
!patch.contains("ALPHA-DELIVERED"),
"phase one's work must not reappear in phase two's patch"
);
// The branch, unlike the patch, stays cumulative: it is built from HEAD,
// so it still carries phase one's commit underneath phase two's.
let branch = second.committed.expect("phase two committed").branch;
let files = Command::new("git")
.arg("-C")
.arg(&repo)
.args(["ls-tree", "--name-only", "-r", &branch])
.output()
.unwrap();
let listed = String::from_utf8_lossy(&files.stdout);
assert!(listed.contains("ALPHA.md"), "branch is cumulative: {listed}");
assert!(listed.contains("BETA.md"), "branch is cumulative: {listed}");
}
/// A checkout must stay writable after another user has written to it.
///
/// The production failure (mission `019fc437`) is a uid split: cm-api runs as
/// 65532, the mission runtime container runs as root, and they share one
/// checkout. Git's `.git/objects/xx/` fan-out directories inherit the
/// ownership of whoever creates them, so the agent committing first locked the
/// server out — `git add` returned "insufficient permission for adding an
/// object to repository database".
///
/// A test process cannot become two users, so this asserts the mechanism that
/// makes the two-user case work: the clone sets `core.sharedRepository`, and
/// objects git writes afterwards are group- and world-writable. Without that
/// mode bit the second user is refused regardless of which one arrived first.
#[tokio::test]
async fn a_checkout_is_writable_by_both_uids_that_share_it() {
use std::os::unix::fs::PermissionsExt;
let pool = cm_testkit::test_pool().await;
let tmp = tempfile::tempdir().unwrap();
let mission = Uuid::now_v7();
let repo = seed_repo(tmp.path(), mission);
let (_, phase) = seed_mission_phase(&pool, mission).await;
let shared = Command::new("git")
.arg("-C")
.arg(&repo)
.args(["config", "core.sharedRepository"])
.output()
.unwrap();
assert_eq!(
String::from_utf8_lossy(&shared.stdout).trim(),
"0777",
"the checkout must be marked shared, or a second uid cannot write objects"
);
// Only directories created *after* the setting can carry its mode, and
// only those matter: the clone writes its own objects before any config
// exists, but the party that would be blocked by them is the container,
// which runs as root and ignores permission bits. The failing direction is
// the other one — directories the agent creates later, which the server
// must still be able to write into. Snapshot first, then diff.
let objects = repo.join(".git/objects");
let fanout = |dir: &std::path::Path| -> std::collections::HashSet<String> {
std::fs::read_dir(dir)
.map(|rd| {
rd.filter_map(|e| e.ok())
.map(|e| e.file_name().to_string_lossy().into_owned())
.filter(|n| n.len() == 2 && n.chars().all(|c| c.is_ascii_hexdigit()))
.collect()
})
.unwrap_or_default()
};
let before = fanout(&objects);
std::fs::write(repo.join("SHARED.md"), "SHARED\n").unwrap();
capture(&pool, tmp.path(), mission, phase)
.await
.unwrap()
.unwrap();
let mut checked = 0;
for name in fanout(&objects).difference(&before) {
let mode = std::fs::metadata(objects.join(name))
.unwrap()
.permissions()
.mode()
& 0o777;
assert_eq!(
mode & 0o022,
0o022,
"{name} is {mode:o}; the other uid sharing this checkout could not \
write objects into it"
);
checked += 1;
}
assert!(checked > 0, "no object directories were created to check");
}
/// Delivery must commit without depending on the checkout's git identity.
///
/// The server container has no identity of its own (`git config --global
/// user.email` exits 1), so `git commit` fails with "Author identity unknown"
/// unless one is supplied. Mission `019fc450` lost its first phase that way,
/// while earlier missions committed fine — because their agents had happened
/// to run `git config user.email` in the checkout first.
///
/// A test process cannot unset the developer's global git config without
/// racing every other test, so this asserts the stronger, deterministic
/// property: the pipeline's identity is used even when the checkout already
/// has a different one. An identity that overrides existing config is
/// necessarily also present when config is absent.
#[tokio::test]
async fn delivery_commits_under_its_own_identity() {
let pool = cm_testkit::test_pool().await;
let tmp = tempfile::tempdir().unwrap();
let mission = Uuid::now_v7();
let repo = seed_repo(tmp.path(), mission);
let (_, phase) = seed_mission_phase(&pool, mission).await;
// seed_repo configures "Test <[email protected]>" locally; the base
// commit therefore carries it, and the delivery commit must not.
std::fs::write(repo.join("IDENTITY_PROBE.md"), "PROBE\n").unwrap();
let cap = capture(&pool, tmp.path(), mission, phase)
.await
.unwrap()
.unwrap();
let commit = cap.committed.expect("delivery committed");
let author = Command::new("git")
.arg("-C")
.arg(&repo)
.args(["log", "-1", "--format=%an <%ae>", &commit.sha])
.output()
.unwrap();
let author = String::from_utf8_lossy(&author.stdout).trim().to_string();
assert_eq!(
author, "Omar Sobh <[email protected]>",
"delivery must supply a configured identity, not inherit whatever \
the checkout happens to have configured"
);
let base_author = Command::new("git")
.arg("-C")
.arg(&repo)
.args(["log", "-1", "--format=%an", "HEAD~1"])
.output()
.unwrap();
assert_eq!(
String::from_utf8_lossy(&base_author.stdout).trim(),
"Test",
"the pre-existing local identity is still configured, so the \
assertion above proves an override rather than an absence"
);
}
/// The commit subject must read as English on both the first pass and a rerun.
///
/// Mission `019fc4e0` pushed commits titled "clawmates: phase phase work" — the
/// iteration marker was interpolated into a slot that already said "phase".
/// Cosmetic, but it lands in the operator's git history under their own name.
#[tokio::test]
async fn commit_subjects_read_correctly_on_first_pass_and_rerun() {
let pool = cm_testkit::test_pool().await;
for (iteration, expected) in [(0, "clawmates: phase work"), (1, "clawmates: phase work (pass 2)")] {
let tmp = tempfile::tempdir().unwrap();
let mission = Uuid::now_v7();
let repo = seed_repo(tmp.path(), mission);
let (_, phase) = seed_mission_phase(&pool, mission).await;
std::fs::write(repo.join("SUBJECT_PROBE.md"), "PROBE\n").unwrap();
let commit = cm_api::mission_delivery::commit_phase_work(&repo, mission, phase, iteration)
.await
.unwrap()
.expect("committed");
let subject = Command::new("git")
.arg("-C")
.arg(&repo)
.args(["log", "-1", "--format=%s", &commit.sha])
.output()
.unwrap();
assert_eq!(
String::from_utf8_lossy(&subject.stdout).trim(),
expected,
"iteration {iteration} produced a malformed subject"
);
}
}
/// "No test suite" and "could not run the test suite" must not look alike.
///
/// verify_tests returned Option<bool>, so both produced `None`. That is how a
/// runtime image shipped without `cargo` stayed invisible: every on_green_tests
/// phase landed on -wip, which reads exactly like a repository that has no
/// tests — the conclusion I drew at the time and reported.
///
/// Both still gate identically, and that part is deliberate: unproven is not a
/// pass, whatever the reason. What changes is that the artifact now says which
/// of the two happened, so an infrastructure fault is legible as one.
#[tokio::test]
async fn an_unrunnable_suite_is_distinguishable_from_no_suite() {
use cm_api::mission_delivery::{verify_tests, TestOutcome};
let tmp = tempfile::tempdir().unwrap();
let mission = Uuid::now_v7();
let repo = seed_repo(tmp.path(), mission);
// No Cargo.toml / package.json / pytest markers: nothing to run.
let none = verify_tests(&repo, "clawmates-runtime-does-not-exist").await;
assert_eq!(none, TestOutcome::NoSuite);
assert_eq!(none.status(), "no_suite");
assert_eq!(none.verified(), None, "no suite must not clear the gate");
// A suite exists, but the container named here does not, so it cannot run.
std::fs::write(
repo.join("Cargo.toml"),
"[package]\nname = \"p\"\nversion = \"0.1.0\"\nedition = \"2021\"\n",
)
.unwrap();
let unrunnable = verify_tests(&repo, "clawmates-runtime-does-not-exist").await;
assert_eq!(unrunnable.status(), "could_not_run");
assert_eq!(
unrunnable.verified(),
None,
"an unrunnable suite must not clear the gate either"
);
assert!(
unrunnable.detail().is_some_and(|d| !d.is_empty()),
"an infrastructure fault must carry its reason into the artifact"
);
assert_ne!(
unrunnable.status(),
none.status(),
"the two must be distinguishable — this is the whole point"
);
}
/// A COMMIT_EDITMSG left by the agent must not block delivery.
///
/// From mission 019fcd0c: the agent ran `git commit` itself, leaving
/// `.git/COMMIT_EDITMSG` owned by root at 0644, and the server's commit died
/// with "Permission denied". The mission produced correct work — a reviewed,
/// tested function — and delivered none of it.
///
/// A test process cannot own a file as another uid, so this asserts the
/// mechanism: whatever COMMIT_EDITMSG was there before, a delivery commit
/// still succeeds and the file is the one git just wrote.
#[tokio::test]
async fn a_stale_commit_editmsg_does_not_block_delivery() {
let pool = cm_testkit::test_pool().await;
let tmp = tempfile::tempdir().unwrap();
let mission = Uuid::now_v7();
let repo = seed_repo(tmp.path(), mission);
let (_, phase) = seed_mission_phase(&pool, mission).await;
// Stand in for the agent's leftover: content that must not survive.
let msg = repo.join(".git/COMMIT_EDITMSG");
std::fs::write(&msg, "LEFTOVER FROM THE AGENT\n").unwrap();
std::fs::write(repo.join("WORK.md"), "work\n").unwrap();
let cap = capture(&pool, tmp.path(), mission, phase)
.await
.unwrap()
.unwrap();
let commit = cap
.committed
.expect("delivery must commit despite a stale COMMIT_EDITMSG");
assert!(!commit.sha.is_empty());
let body = std::fs::read_to_string(&msg).unwrap_or_default();
assert!(
!body.contains("LEFTOVER FROM THE AGENT"),
"the stale message survived: {body:?}"
);
}
+9 -2
View File
@@ -556,6 +556,11 @@ pub struct RegisterArtifact<'a> {
pub title: Option<&'a str>, pub title: Option<&'a str>,
pub generated_by_run: Option<Uuid>, pub generated_by_run: Option<Uuid>,
pub render_pdf: bool, pub render_pdf: bool,
/// Free-form facts about the artifact (diffstat, branch, gate verdict).
/// The column has existed since 0047 and was never written — an artifact
/// with no metadata is a path and a kind, which is not enough for a UI to
/// say anything useful about it.
pub metadata: Option<serde_json::Value>,
} }
/// Register an artifact discovered on disk (or produced inline). /// Register an artifact discovered on disk (or produced inline).
@@ -566,14 +571,15 @@ pub async fn register_artifact(pool: &PgPool, a: RegisterArtifact<'_>) -> Result
let row = sqlx::query( let row = sqlx::query(
"INSERT INTO mission_artifacts "INSERT INTO mission_artifacts
(id, mission_id, phase_id, path, kind, mime, title, (id, mission_id, phase_id, path, kind, mime, title,
generated_by_run, render_pdf_status) generated_by_run, render_pdf_status, metadata)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,COALESCE($10, '{}'::jsonb))
ON CONFLICT (mission_id, path) DO UPDATE ON CONFLICT (mission_id, path) DO UPDATE
SET kind = EXCLUDED.kind, SET kind = EXCLUDED.kind,
mime = COALESCE(EXCLUDED.mime, mission_artifacts.mime), mime = COALESCE(EXCLUDED.mime, mission_artifacts.mime),
title = COALESCE(EXCLUDED.title, mission_artifacts.title), title = COALESCE(EXCLUDED.title, mission_artifacts.title),
generated_by_run = COALESCE(EXCLUDED.generated_by_run, generated_by_run = COALESCE(EXCLUDED.generated_by_run,
mission_artifacts.generated_by_run), mission_artifacts.generated_by_run),
metadata = COALESCE(EXCLUDED.metadata, mission_artifacts.metadata),
updated_at = now() updated_at = now()
RETURNING id", RETURNING id",
) )
@@ -586,6 +592,7 @@ pub async fn register_artifact(pool: &PgPool, a: RegisterArtifact<'_>) -> Result
.bind(a.title) .bind(a.title)
.bind(a.generated_by_run) .bind(a.generated_by_run)
.bind(render_status) .bind(render_status)
.bind(a.metadata.as_ref())
.fetch_one(pool) .fetch_one(pool)
.await?; .await?;
Ok(row.get("id")) Ok(row.get("id"))
+35 -2
View File
@@ -103,8 +103,28 @@ impl Scheduler {
routines::set_next_run(&self.pool, routine.id, next).await?; routines::set_next_run(&self.pool, routine.id, next).await?;
let agent_id = cm_domain::AgentId::from(routine.agent_id); let agent_id = cm_domain::AgentId::from(routine.agent_id);
let Ok(agent) = agents::get(&self.pool, agent_id).await else { let agent = match agents::get(&self.pool, agent_id).await {
continue; // deleted agent: routine is orphaned Ok(a) => a,
Err(e) => {
// Orphaned routine (deleted agent, or a row we cannot
// read). Settle the slot rather than leaving it `claimed`:
// an unsettled claim looks like a crash mid-fire, so every
// tick would re-claim the same routine forever and the
// table would grow one stuck row per occurrence.
eprintln!(
"scheduler: routine {} references agent {agent_id} which could not be \
read ({e}) — settling the occurrence as failed",
routine.id
);
let _ = routines::complete_fire(
&self.pool,
routine.id,
slot,
Some(&format!("agent {agent_id} unreadable: {e}")),
)
.await;
continue;
}
}; };
// Topology routine: fire the whole team's stored topology as one // Topology routine: fire the whole team's stored topology as one
@@ -149,6 +169,19 @@ impl Scheduler {
let message = routine.action["message"].as_str().unwrap_or_default(); let message = routine.action["message"].as_str().unwrap_or_default();
if message.is_empty() { if message.is_empty() {
// Neither a topology nor a message action: there is nothing to
// dispatch. Settle it so the slot is not mistaken for a crash.
eprintln!(
"scheduler: routine {} has no `topology` or `message` action — nothing to fire",
routine.id
);
let _ = routines::complete_fire(
&self.pool,
routine.id,
slot,
Some("routine action has neither `topology` nor `message`"),
)
.await;
continue; continue;
} }
@@ -138,7 +138,14 @@ excluded_tools = ["shell", "file_read", "file_write", "http_request", "browser",
# burn tokens dumping code inline, this list drifted back to pre-0.8 names. # burn tokens dumping code inline, this list drifted back to pre-0.8 names.
[risk_profiles.coding_readwrite] [risk_profiles.coding_readwrite]
level = "full" level = "full"
allowed_tools = ["file_read", "file_edit", "content_search", "glob_search", "git_operations", "shell"] # `file_write` creates and overwrites; `file_edit` only replaces an exact
# existing string and rejects an empty `old_string`, so without file_write an
# agent literally cannot create a new file. Observed on mission 019fc372: the
# agent burned its turn reasoning about how to make file_edit create a file
# ("the tool rejected empty old_string... the shell is restricted") before
# working around it through `shell`. The comment below has claimed file_write
# was here since the profile was written; the list never had it.
allowed_tools = ["file_read", "file_write", "file_edit", "content_search", "glob_search", "git_operations", "shell"]
excluded_tools = ["http_request", "browser", "composio"] excluded_tools = ["http_request", "browser", "composio"]
# Read-only research profile (scout/researcher/reviewer/planner roles). # Read-only research profile (scout/researcher/reviewer/planner roles).
+14
View File
@@ -0,0 +1,14 @@
[Unit]
Description=Clawmates paper library harvest (arXiv -> shelf + vault catalogue)
Wants=docker.service
After=docker.service network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/clawmates-library.sh
StandardOutput=journal
StandardError=journal
Nice=10
# A harvest downloads PDFs and pushes a branch; give it room but do not
# let a wedged run hold the slot until the next week.
TimeoutStartSec=30min
+49
View File
@@ -0,0 +1,49 @@
#!/usr/bin/env bash
# Weekly paper-library harvest.
#
# Deliberately thin: it calls the API and reports what came back. All the
# logic lives in the server, so this file never needs to change when the
# harvest does.
#
# The token lives in /etc/clawmates/library.token (root-only). It is a
# long-lived operator session; rotate by replacing the file.
set -uo pipefail
TOKEN_FILE=/etc/clawmates/library.token
[ -r "$TOKEN_FILE" ] || { echo "library: no token at $TOKEN_FILE"; exit 1; }
TOKEN=$(cat "$TOKEN_FILE")
RESP=$(docker run --rm --network clawmates_core curlimages/curl:latest \
-s -m 1800 -X POST \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"per_topic":5}' \
http://clawmates_server_1:8080/api/library/runs)
echo "library: $RESP" | head -c 2000
# Report health explicitly. A run that shelved nothing is normal for a
# mature library; a run that ERRORED is not, and the two look identical
# if you only count papers.
echo "$RESP" | python3 -c '
import json, sys
try:
d = json.load(sys.stdin)
except Exception as e:
print("library: unreadable response (%s)" % e)
sys.exit(1)
shelved = len(d.get("shelved", []))
healthy = d.get("healthy", False)
# Backslashes are avoided inside this program on purpose: it is embedded in a
# single-quoted shell string, and an escaped quote here does not survive the
# shell. The first version used one inside an f-string, crashed on every run,
# and systemd reported a FAILED unit for a harvest that had actually shelved
# 15 papers and pushed them. A false failure destroys trust in the signal as
# surely as a false success.
print("library: %d candidates, %d already held, %d shelved, healthy=%s, pushed=%s, branch=%s" % (
d.get("candidates", 0), d.get("already_had", 0), shelved,
healthy, d.get("pushed"), d.get("branch")))
for f in d.get("failed", []):
print("library: FAILED %s" % f)
sys.exit(0 if healthy else 1)
'
+16
View File
@@ -0,0 +1,16 @@
[Unit]
Description=Clawmates paper library — weekly harvest
[Timer]
# Monday 07:00 local. Weekly rather than daily because arXiv moves at
# roughly that pace for a narrow topic set, and a run that almost always
# finds nothing trains you to ignore it.
OnCalendar=Mon *-*-* 07:00:00
# Fire on next boot if the machine was down at the scheduled time — a
# missed week is a silently empty library.
Persistent=true
AccuracySec=1min
Unit=clawmates-library.service
[Install]
WantedBy=timers.target
+69
View File
@@ -0,0 +1,69 @@
-- The seen-set for continuous missions.
--
-- Every "continuous X" mission has the same failure mode: it runs again and
-- redoes work it already did. Research resurfaces papers it already read; a
-- security scan re-reports findings already triaged. Orchestration does not
-- fix that — a record of what has already been covered does.
--
-- This repository already tried continuous research once. Migrations 0030-0044
-- built `research_topics`, `research_outcomes` and `loops`; 0053 dropped them
-- all. `research_topics` carried a status lifecycle but no seen-set, so it
-- could run forever and never know what it had covered. That is the gap this
-- table exists to close, and it is the reason it lands before any scheduling.
--
-- Authoritative here rather than in the runtime's memory: ZeroClaw memory is
-- scoped per agent, and mission agents are ephemeral `claw_<uuid>` aliases
-- minted per mission (measured: ~100 of them already). A seen-set that
-- disappears with the agent that wrote it is not a seen-set.
CREATE TABLE corpus_items (
id UUID PRIMARY KEY,
workspace_id UUID NOT NULL REFERENCES workspaces (id) ON DELETE CASCADE,
-- Which corpus this belongs to, e.g. 'valhalla-vault'. A workspace can
-- track several (a vault, a findings ledger, a paper collection).
corpus_id TEXT NOT NULL,
-- 'note' = something already in the corpus (a vault file). Establishes
-- coverage: what has this vault already got?
-- 'source' = an external thing a mission consumed (a paper, an advisory).
-- This is the dedupe key that stops re-reading.
--
-- Both are needed and they answer different questions. Measured against
-- the real vault: 416 notes, and ZERO carry an arxiv/doi/url key — so an
-- ingester keyed only on external identity would index nothing at all.
kind TEXT NOT NULL CHECK (kind IN ('note', 'source')),
-- Stable identity within the corpus. For notes, 'note:<vault-relative
-- path>'; for sources, a natural id like 'arxiv:2401.12345', 'doi:10...'
-- or 'url:<sha256>'. Uniqueness is on this, which is what makes
-- re-ingestion idempotent.
source_id TEXT NOT NULL,
title TEXT,
-- Vault-relative path for notes; NULL for external sources.
path TEXT,
url TEXT,
-- SHA-256 of the content at last sight. Lets a re-index distinguish
-- "unchanged" from "edited" without diffing, so an unchanged vault is a
-- genuine no-op rather than 416 pointless updates.
content_hash TEXT NOT NULL,
first_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(),
last_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- Which mission first recorded this. NULL for the initial vault index,
-- which is derived from files nobody's mission wrote.
mission_id UUID REFERENCES missions (id) ON DELETE SET NULL,
UNIQUE (workspace_id, corpus_id, source_id)
);
-- The hot query is "have I seen this?", which the UNIQUE index already covers.
-- This one serves "what does this corpus contain?" for briefing assembly.
CREATE INDEX corpus_items_corpus_idx
ON corpus_items (workspace_id, corpus_id, kind, last_seen_at DESC);
-- "What did this mission add?" — the verification predicate for a continuous
-- run is that it contributed at least one NEW source.
CREATE INDEX corpus_items_mission_idx
ON corpus_items (mission_id)
WHERE mission_id IS NOT NULL;
+38 -6
View File
@@ -32,19 +32,51 @@ TAG=${TAG:-latest}
SHA=$(git rev-parse --short HEAD 2>/dev/null || echo manual) SHA=$(git rev-parse --short HEAD 2>/dev/null || echo manual)
AGENT_IMAGES=(agent-base agent-browser agent-terminal) AGENT_IMAGES=(agent-base agent-browser agent-terminal)
load() { ssh "$BUILD_HOST" "docker save $1" | ssh "$2" "docker load"; } # Stream an image between two remote hosts. The tar crosses two SSH
# connections spliced through this workstation, so a stall on either side
# truncates it — that surfaces as `unexpected EOF` from `docker load`, which
# is a genuine failure and was previously indistinguishable from success
# because nothing checked afterwards. Compress (these images are mostly
# filesystem, and less bytes is less exposure to a stall) and set pipefail so
# a failed `save` cannot be masked by a `load` that exits 0 on a short stream.
load() {
( set -o pipefail
ssh "$BUILD_HOST" "docker save $1 | gzip -1" | ssh "$2" "gunzip | docker load" )
}
# Load only if the target lacks the exact image (skips re-transferring unchanged # Load only if the target lacks the exact image (skips re-transferring unchanged
# multi-hundred-MB agent images to every node on each code deploy). # multi-hundred-MB agent images to every node on each code deploy).
#
# Verifies by image ID afterwards rather than trusting the exit status: a
# truncated stream can still leave a partially-populated image, and shipping a
# corrupt agent image to every fleet node is worse than failing the deploy.
# One retry, because the observed failure is a transient stream stall.
#
# Identity is the image's `Created` stamp, NOT its `Id`. A BuildKit image on
# the build host carries attestation manifests that `docker save | docker load`
# does not reproduce, so the same build legitimately arrives with a different
# Id and a different reported Size — comparing Ids fails every transfer of a
# correctly-shipped image. `Created` comes from the config blob, survives the
# round trip, and is what actually answers "is the new build here".
load_if_changed() { load_if_changed() {
local lid rid local lts rts attempt
lid=$(ssh "$BUILD_HOST" "docker image inspect -f '{{.Id}}' $1 2>/dev/null" || true) lts=$(ssh "$BUILD_HOST" "docker image inspect -f '{{.Created}}' $1 2>/dev/null" || true)
rid=$(ssh "$2" "docker image inspect -f '{{.Id}}' $1 2>/dev/null" || true) rts=$(ssh "$2" "docker image inspect -f '{{.Created}}' $1 2>/dev/null" || true)
if [ -n "$lid" ] && [ "$lid" = "$rid" ]; then if [ -n "$lts" ] && [ "$lts" = "$rts" ]; then
echo " (unchanged — skip)" echo " (unchanged — skip)"
return 0 return 0
fi fi
load "$1" "$2" for attempt in 1 2; do
load "$1" "$2" || echo " (transfer attempt $attempt failed)"
rts=$(ssh "$2" "docker image inspect -f '{{.Created}}' $1 2>/dev/null" || true)
if [ -n "$lts" ] && [ "$lts" = "$rts" ]; then
[ "$attempt" -gt 1 ] && echo " (recovered on attempt $attempt)"
return 0
fi
echo " (build stamp mismatch after attempt $attempt — retrying)" >&2
done
echo "$1 did not land on $2 (wanted $lts, got ${rts:-nothing})" >&2
return 1
} }
echo "→ sync to $BUILD_HOST" echo "→ sync to $BUILD_HOST"