feat(missions): Continuous Research is a mission type, not just a team checkbox
deploy / test (push) Successful in 4m6s
deploy / build (push) Successful in 5m32s

`templates/teams/continuous_research.toml` has existed with three well-written
roles since it was authored, but no workflow recipe pointed at it — every
recipe in templates/workflows/ defaults `default_team_template = "rust_sdlc"`.
So the only way to reach it was as a checkbox under Advanced. It is now a
Step-1 card: the registry loads it at boot and `GET /api/workflows` serves it,
with no frontend change (MissionWizard renders whatever the endpoint returns).

Both phases are kind `research`, deliberately, rather than new `read`/`script`
kinds. An unrecognised kind falls through `purposes_for`'s `_ => ["mission"]`
and is absent from `PRODUCING_KINDS`, so it would get the generic directive AND
be exempt from the empty-delivery rule — a phase that produces nothing and
still passes. That is the shape this codebase keeps paying for; two `research`
phases differentiated by `task` keep both guards.

`commit_policy = "always"`, not `on_green_tests`: the vault is prose with no
suite, so a test gate would find nothing to run and land every branch `-wip`.

The harvest is NOT an agent phase. `continuous_research.rs` calls the existing
`library::run_to_vault` — arXiv search, seen-set check, PDF shelf, vault note,
attributed by `mission_id` — because that path is deterministic, takes seconds,
and owns the `corpus_items` seen-set that is the whole reason a recurring
mission knows what it already covered. An agent redoing it would be slower and
would lose that.

The manifest path is not invented either: the team template has told
`signal_harvester` to write `ContinuousResearch/<date>/harvest.jsonl` all along.
This makes the code produce what the prompt already promised, and a test pins
the path and every documented key so the two cannot drift into an agent reading
a file nothing writes.

DEFAULT_CORPUS / DEFAULT_VAULT_URL exported rather than duplicated, so the
route and the launch hook cannot disagree about which vault.

344 tests pass.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-17 14:18:35 -07:00
co-authored by Claude Opus 5
parent f87853ecf9
commit e20b321055
4 changed files with 254 additions and 2 deletions
+158
View File
@@ -0,0 +1,158 @@
//! The harvest half of a Continuous Research mission.
//!
//! Finding papers is NOT agent work. `library::run_to_vault` already does arXiv
//! search → seen-set check → PDF fetch → blob shelf → vault note, deterministically
//! and in seconds, and it takes a `mission_id` so the run is attributed. Asking an
//! agent to redo it would be slower, non-repeatable, and would abandon the
//! `corpus_items` seen-set — which is the entire reason a recurring mission knows
//! what it already covered. `corpus.rs` puts it plainly: "A recurring mission's
//! hard problem is not running the agent — that is 23 seconds — it is knowing
//! what it already did last time."
//!
//! So the harvest runs here, at launch, and the agents start from its output.
//!
//! The manifest path (`ContinuousResearch/<date>/harvest.jsonl`) is not invented:
//! `templates/teams/continuous_research.toml` has told the `signal_harvester`
//! role to write exactly that file since the template was authored. This makes
//! the code produce what the prompt already promised, rather than leaving a role
//! to fabricate it.
use std::sync::Arc;
use serde_json::json;
use uuid::Uuid;
/// Template kind that triggers a harvest at launch.
pub const TEMPLATE_KIND: &str = "continuous_research";
/// Today's manifest, relative to the vault root.
pub fn manifest_path(date: &str) -> String {
format!("ContinuousResearch/{date}/harvest.jsonl")
}
/// UTC date stamp, the same key the vault folders use.
pub fn today() -> String {
let now = time::OffsetDateTime::now_utc();
format!(
"{:04}-{:02}-{:02}",
now.year(),
now.month() as u8,
now.day()
)
}
/// Run the harvest for a mission and leave a manifest the agents can read.
///
/// Non-fatal by contract: a launch whose harvest fails still starts its phases,
/// because a quiet day and a broken day must be distinguishable and the phase
/// itself is what reports which happened. What is NOT acceptable is failing
/// silently, so every outcome is logged with its counts.
pub async fn harvest_for_mission(
pool: &sqlx::PgPool,
blobs: &Arc<dyn cm_files::BlobStore>,
workspace_id: Uuid,
mission_id: Uuid,
topics: &[String],
per_topic: usize,
) -> Result<usize, String> {
let work_root = std::env::temp_dir().join("clawmates-library");
let run = crate::library::run_to_vault(
pool,
blobs,
workspace_id,
crate::routes::library::DEFAULT_CORPUS,
crate::routes::library::DEFAULT_VAULT_URL,
&work_root,
topics,
per_topic,
Some(mission_id),
)
.await?;
let shelved = run.harvest.shelved.len();
// A quiet day is not a failure. `Harvest::healthy()` (nothing errored) is a
// different question from `added_anything()` (something new arrived), and
// collapsing them is the defect class this codebase keeps paying for.
eprintln!(
"continuous_research: mission {mission_id} harvested {} candidate(s), {} already had, \
{} shelved, {} failed",
run.harvest.candidates,
run.harvest.already_had,
shelved,
run.harvest.failed.len()
);
for (source_id, why) in &run.harvest.failed {
eprintln!("continuous_research: {source_id} not shelved: {why}");
}
Ok(shelved)
}
/// The manifest lines for a set of freshly shelved papers.
///
/// Shape matches what `templates/teams/continuous_research.toml` documents:
/// `{ source, url, title, snippet, first_seen, topic_tags }`.
pub fn manifest_lines(papers: &[crate::papers::Paper], first_seen: &str) -> String {
papers
.iter()
.map(|p| {
json!({
"source": p.source_id(),
"url": format!("https://arxiv.org/abs/{}", p.arxiv_id),
"title": p.title,
"snippet": p.summary.chars().take(400).collect::<String>(),
"first_seen": first_seen,
"topic_tags": [],
})
.to_string()
})
.collect::<Vec<_>>()
.join("\n")
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_manifest_path_matches_what_the_team_template_promises() {
// templates/teams/continuous_research.toml tells signal_harvester to
// write ContinuousResearch/<date>/harvest.jsonl. If this drifts, the
// agents read a file nothing writes and silently review nothing.
assert_eq!(
manifest_path("2026-08-17"),
"ContinuousResearch/2026-08-17/harvest.jsonl"
);
}
#[test]
fn the_date_stamp_is_zero_padded() {
let d = today();
assert_eq!(d.len(), 10, "YYYY-MM-DD, got {d:?}");
assert_eq!(d.matches('-').count(), 2, "{d:?}");
}
/// One JSON object per line, and every key the template's prompt names —
/// an agent instructed to read `topic_tags` must not find it absent.
#[test]
fn manifest_lines_carry_every_documented_key() {
let p = crate::papers::Paper {
arxiv_id: "2401.12345".into(),
title: "A Paper".into(),
authors: vec!["A. Author".into()],
summary: "x".repeat(900),
published: "2026-08-17".into(),
pdf_url: "https://arxiv.org/pdf/2401.12345".into(),
};
let out = manifest_lines(std::slice::from_ref(&p), "2026-08-17");
assert_eq!(out.lines().count(), 1);
let v: serde_json::Value = serde_json::from_str(&out).expect("each line is JSON");
for key in ["source", "url", "title", "snippet", "first_seen", "topic_tags"] {
assert!(v.get(key).is_some(), "missing {key} in {v}");
}
assert_eq!(v["source"], "arxiv:2401.12345");
assert!(
v["snippet"].as_str().unwrap().chars().count() <= 400,
"snippet must be trimmed, not the whole abstract"
);
}
}
+1
View File
@@ -24,6 +24,7 @@ mod mcp_skills;
pub mod microvm_client; pub mod microvm_client;
pub mod microvm_executor; pub mod microvm_executor;
pub mod microvm_turn_executor; pub mod microvm_turn_executor;
pub mod continuous_research;
pub mod mission_delivery; pub mod mission_delivery;
pub mod mission_events; pub mod mission_events;
pub mod mission_fs; pub mod mission_fs;
+2 -2
View File
@@ -14,8 +14,8 @@ use crate::{ApiError, AppState, Authed};
/// Default corpus + repo. Single-operator deployment, so these are constants /// Default corpus + repo. Single-operator deployment, so these are constants
/// rather than another table to keep in sync; a second library becomes a /// rather than another table to keep in sync; a second library becomes a
/// request field the day one exists. /// request field the day one exists.
const DEFAULT_CORPUS: &str = "valhalla-vault"; pub const DEFAULT_CORPUS: &str = "valhalla-vault";
const DEFAULT_VAULT_URL: &str = "https://git.redclaw.dev/redclaw/valhalla-vault.git"; pub const DEFAULT_VAULT_URL: &str = "https://git.redclaw.dev/redclaw/valhalla-vault.git";
#[derive(Deserialize)] #[derive(Deserialize)]
pub struct RunRequest { pub struct RunRequest {
@@ -0,0 +1,93 @@
key = "continuous_research"
title = "Continuous Research"
blurb = "Harvest new arXiv papers, read them against your projects, and write a two-host podcast script for the morning."
requires_repo = true
# The vault, not `rust_sdlc`. Every other recipe defaults to the Rust SDLC team,
# which is why `continuous_research` has only ever been reachable as a checkbox
# in Advanced rather than as a mission type.
default_team_template = "continuous_research"
# ── How the papers arrive ────────────────────────────────────────────
#
# The harvest is NOT an agent phase. `library::run_to_vault` already does
# arXiv search → seen-set check → PDF fetch → blob shelf → vault note, and it
# takes a `mission_id` so the run is attributed. Asking an agent to redo that
# would be slower, non-deterministic, and would lose the `corpus_items`
# seen-set that is the entire reason a recurring mission knows what it already
# covered. `mission_orchestrator::on_launch` runs it before the phases start
# and drops a manifest at `ContinuousResearch/<date>/harvest.jsonl` — the path
# `templates/teams/continuous_research.toml` already tells the harvester role to
# write, now produced by the code that actually does the harvesting.
#
# The mission binds the VAULT repo, so `/mission/repo` is the card catalogue:
# the agents read the notes the harvest just wrote and commit their analysis and
# script back onto the run's own branch, never `main`.
[[phases]]
kind = "research"
order_idx = 0
[phases.config]
produces = ["md"]
default_topology = "hub_spoke"
# `research`, not a new `read` kind. An unrecognised kind falls through to
# `purposes_for`'s `_ => ["mission"]` and is absent from `PRODUCING_KINDS`
# (phase_runner.rs), so it would get a generic directive AND be exempt from the
# empty-delivery rule — a phase that could produce nothing and still pass. Two
# `research` phases differentiated by `task` keeps both guards.
task = """
Read today's harvested papers and judge them against the projects listed below.
Start from ContinuousResearch/<today>/harvest.jsonl — that is the list of papers \
that are NEW since the last run. Papers already covered are not in it, and you \
should not go looking for them.
For each paper write an entry in ContinuousResearch/<today>/analysis.md \
containing: what it actually does (not what its abstract claims), whether the \
evidence supports it, and — the part that matters — WHICH of the projects below \
it bears on and what concrete change it would imply. Name a file, a module or a \
roadmap item wherever you can.
Depth comes from the paper itself: the note carries the abstract, and `curl` on \
the arXiv abstract page gets you the rest. Do not review a paper from its title.
A paper with no bearing on any project is a real and useful finding — say so in \
one line and move on. Do not manufacture relevance.
PROJECTS THIS RESEARCH SERVES — replace this block when creating the mission:
(none configured yet)
"""
done_when = "ContinuousResearch/<today>/analysis.md exists and contains, for every paper in that day's harvest.jsonl, a judgement of the work and a statement of which project it bears on or that it bears on none"
max_iterations = 2
# Nothing here is compiled, so `on_green_tests` would gate on a suite that does
# not exist and land every branch `-wip`. The vault is prose.
commit_policy = "always"
[[phases]]
kind = "research"
order_idx = 1
[phases.config]
produces = ["md"]
default_topology = "hub_spoke"
task = """
Turn today's analysis into a podcast script for two hosts.
Read ContinuousResearch/<today>/analysis.md and write \
ContinuousResearch/<today>/script.md as a conversation between HOST and GUEST. \
Lead with what changed for our projects, not with a list of papers — the \
listener is on a treadmill, not at a desk.
Also write ContinuousResearch/<today>/episode.json:
{ "title": "<one line, under 80 chars>",
"highlights": ["<10-70 chars each, at most 5>"] }
Those bounds are the podcast API's, not a style preference — a highlight \
outside them is rejected.
Target seven minutes of speech, roughly 1,000 words. Say the specific thing: \
"this changes how we prune the HNSW graph in clawhdf5" beats "researchers \
propose a novel method". Skip a paper entirely rather than pad the episode \
with one that does not matter.
"""
done_when = "ContinuousResearch/<today>/script.md contains a two-host dialogue covering the analysis, and episode.json contains a title and a highlights array whose entries are each between 10 and 70 characters"
max_iterations = 2
commit_policy = "always"