Omar SobhandClaude Opus 5
e4942ce985
Repairing the 55 broken skill bindings made the catalogue correct. This
makes it reachable, which it was not — for any skill, on any mission, since
the catalogue was built.
The skills had exactly ONE delivery channel: the `clawmates_skills` MCP
server. A mission claw could not reach it for three independent reasons:
1. `provision_claw` wrote the constant `["clawmates_door"]` and ignored
the template's mcp_bundles — which mission_orchestrator had already
resolved and stored on the team row.
2. The runtime config defines no `clawmates_skills` bundle. The live
local config defines no bundles at all, not even the door.
3. Mission claws run on `claude_cli`, which the runtime's own config
comments document as text-only: it cannot surface a tool call, so no
MCP server is reachable from a mission turn regardless of bundles.
And a mission turn's whole system context is two sentences synthesised from
the role slot in topology_exec::build_prompt. The template's role prose is
not used either — mission_orchestrator documents this, and it means the
role prompts describing which procedures to follow were never read.
Two doc comments in cm-runtime describe the mission path as already having
the summary-and-fetch contract. It never did. The belief was written down
twice and checked zero times, which is why nobody looked — and it is why
the Skill-Use measurement this review planned could only ever have returned
a trigger rate of zero. That would have read as a finding about the agents.
- provision_claw takes the bundles, with clawmates_door always added: a
template that forgets to list it must not get an ungated agent
- all 11 templates now request clawmates_skills; web_fetch removed, since
a list that is honoured must not name a bundle that does not exist
- the re-provision sweep re-asserts the team's own stored bundles rather
than a constant, which would have silently stripped a capability
mid-mission
- pinned skill BODIES are injected into the mission prompt, bounded and
with truncation stated. Bodies, not an index: there is no `skills.read`
tool on this path, so an index would advertise a capability that does
not exist — the exact failure this whole change is about
Three tests: the body reaches the prompt, an agent with no skills adds no
heading (an empty "Your skills" section announces skills the agent does not
have), and the composition is exercised separately from the lookup, because
`pinned_skills_text` working and `run_turn` calling it are different claims
and the second is the one that was false.
Also adds the three review documents: CAPABILITY-REVIEW (inventory, what
was repaired, what is deferred and why), PROVENANCE-ASSESSMENT (assess
only, per decision — what each store answers and the two candidate paths),
and RESEARCH-SWEEP (the fortnight's papers and what we did about each,
including the ones we deliberately did nothing about).
Full workspace suite green.
Co-Authored-By: Claude Opus 5 <[email protected]>
95 lines
4.1 KiB
Markdown
95 lines
4.1 KiB
Markdown
# Research sweep — papers from the fortnight to 2026-08-19
|
|
|
|
Scoped to what could change ClawMates: agent runtimes, sub-agent structure,
|
|
memory substrates, agent performance measurement, and provenance.
|
|
|
|
Each entry says what we did about it. "Nothing" is a legitimate outcome and is
|
|
recorded as such — a sweep where every paper is actionable is a sweep that
|
|
stopped judging.
|
|
|
|
---
|
|
|
|
## Skill-Use (2026-08-05) — *acted on*
|
|
|
|
Benchmarks whether an agent actually **uses** a skill under progressive
|
|
disclosure: it sees a name and description and must retrieve the procedure.
|
|
Three scores:
|
|
|
|
- **Trigger** — did it invoke the skill at all
|
|
- **Compliance** — did it follow the procedure
|
|
- **Boundary** — did it avoid what the skill forbids
|
|
|
|
That is exactly our skills shape, and reading it is what prompted asking
|
|
whether ours fire. The answer turned out to be structural rather than
|
|
behavioural: **they could not**. 55 of 85 bindings resolved to nothing, and the
|
|
catalogue had no delivery channel to a mission agent at all. Both are now fixed
|
|
(see [CAPABILITY-REVIEW.md](CAPABILITY-REVIEW.md)).
|
|
|
|
The measurement itself is now the obvious next piece of work, and it is worth
|
|
doing on the paper's three axes rather than as a pass/fail — the useful output
|
|
is *which* of our 53 skills are inert prose.
|
|
|
|
## MemoryLake on MemoryArena (2026-08-14) — *held as a finding*
|
|
|
|
A matched comparison: same framework, same model, same tasks, with the memory
|
|
backend the only changed component. Structured memory beat vector RAG and
|
|
long-context.
|
|
|
|
The numbers are the point. Success rates were **9/40, 12/20 and 4/20**, and
|
|
every system scored zero on some task category. A useful antidote to expecting
|
|
a memory swap to be transformative.
|
|
|
|
**What we did: nothing, deliberately.** It is the strongest argument against
|
|
adopting graph memory on enthusiasm, and the strongest argument for capturing a
|
|
baseline first.
|
|
|
|
## Harness the Memory (2026-08-15) — *held as a finding*
|
|
|
|
No memory substrate dominates. Broad retrieval helps factual QA, while
|
|
**excessive retrieval actively harms agent decision-making**.
|
|
|
|
Directly relevant to a decision we made this pass: mission prompts now carry
|
|
pinned skill *bodies*. This paper is the reason that is bounded
|
|
(`MAX_PINNED_SKILL_BYTES`) and restricted to pinned skills rather than "give the
|
|
agent everything it might need".
|
|
|
|
## D²ACCI (2026-08-18) — *the sharpest one for us*
|
|
|
|
> "End-to-end evaluation reveals that an error occurred, but not which stage
|
|
> caused it."
|
|
|
|
That sentence is this project's recurring defect class stated as a research
|
|
problem. The paper proposes **DCR**, a graded metric for whether failures stay
|
|
*localizable*.
|
|
|
|
This reframed the provenance assessment: the property to optimise is not
|
|
completeness of the record but **localizability of failure**. A system that
|
|
records everything and cannot tell you which stage broke has not solved the
|
|
problem. See [PROVENANCE-ASSESSMENT.md](PROVENANCE-ASSESSMENT.md).
|
|
|
|
---
|
|
|
|
## On clawhdf5 as the provenance layer
|
|
|
|
The specific question asked. `clawhdf5-agent` (~21k lines: typed entities,
|
|
relations including `RelationType::Causal`, BFS and spreading activation;
|
|
`provenance.rs` with `MemorySource` / `content_hash` / `session_id`) is
|
|
**already in the workspace manifest and used by no crate**. `cm-brain` exposes
|
|
`set_provenance` / `provenance` — a wired slot with **zero callers**.
|
|
|
|
So the adoption cost is lower than it looks. The assessment still recommends
|
|
against leading with it, for reasons that are about our system rather than its
|
|
quality: the `.brain` is reaped with the agent, which is the wrong lifetime for
|
|
an audit record, and it is file-local, so cross-agent questions need a second
|
|
mechanism anyway.
|
|
|
|
The honest sequencing is the one the memory papers argue for: make the record
|
|
complete and correctly retained on the path that matters, capture a baseline,
|
|
then measure whether the graph substrate earns its place.
|
|
|
|
## Not found
|
|
|
|
Nothing in the fortnight materially changes our **runtime** design (microVM
|
|
boot, vsock RPC, per-backend egress) or the sub-agent/topology model. Recording
|
|
that so the next sweep does not re-search the same ground.
|