feat(missions): Continuous Research is a mission type, not just a team checkbox
`templates/teams/continuous_research.toml` has existed with three well-written roles since it was authored, but no workflow recipe pointed at it — every recipe in templates/workflows/ defaults `default_team_template = "rust_sdlc"`. So the only way to reach it was as a checkbox under Advanced. It is now a Step-1 card: the registry loads it at boot and `GET /api/workflows` serves it, with no frontend change (MissionWizard renders whatever the endpoint returns). Both phases are kind `research`, deliberately, rather than new `read`/`script` kinds. An unrecognised kind falls through `purposes_for`'s `_ => ["mission"]` and is absent from `PRODUCING_KINDS`, so it would get the generic directive AND be exempt from the empty-delivery rule — a phase that produces nothing and still passes. That is the shape this codebase keeps paying for; two `research` phases differentiated by `task` keep both guards. `commit_policy = "always"`, not `on_green_tests`: the vault is prose with no suite, so a test gate would find nothing to run and land every branch `-wip`. The harvest is NOT an agent phase. `continuous_research.rs` calls the existing `library::run_to_vault` — arXiv search, seen-set check, PDF shelf, vault note, attributed by `mission_id` — because that path is deterministic, takes seconds, and owns the `corpus_items` seen-set that is the whole reason a recurring mission knows what it already covered. An agent redoing it would be slower and would lose that. The manifest path is not invented either: the team template has told `signal_harvester` to write `ContinuousResearch/<date>/harvest.jsonl` all along. This makes the code produce what the prompt already promised, and a test pins the path and every documented key so the two cannot drift into an agent reading a file nothing writes. DEFAULT_CORPUS / DEFAULT_VAULT_URL exported rather than duplicated, so the route and the launch hook cannot disagree about which vault. 344 tests pass. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
f87853ecf9
commit
e20b321055
@@ -0,0 +1,158 @@
|
|||||||
|
//! The harvest half of a Continuous Research mission.
|
||||||
|
//!
|
||||||
|
//! Finding papers is NOT agent work. `library::run_to_vault` already does arXiv
|
||||||
|
//! search → seen-set check → PDF fetch → blob shelf → vault note, deterministically
|
||||||
|
//! and in seconds, and it takes a `mission_id` so the run is attributed. Asking an
|
||||||
|
//! agent to redo it would be slower, non-repeatable, and would abandon the
|
||||||
|
//! `corpus_items` seen-set — which is the entire reason a recurring mission knows
|
||||||
|
//! what it already covered. `corpus.rs` puts it plainly: "A recurring mission's
|
||||||
|
//! hard problem is not running the agent — that is 23 seconds — it is knowing
|
||||||
|
//! what it already did last time."
|
||||||
|
//!
|
||||||
|
//! So the harvest runs here, at launch, and the agents start from its output.
|
||||||
|
//!
|
||||||
|
//! The manifest path (`ContinuousResearch/<date>/harvest.jsonl`) is not invented:
|
||||||
|
//! `templates/teams/continuous_research.toml` has told the `signal_harvester`
|
||||||
|
//! role to write exactly that file since the template was authored. This makes
|
||||||
|
//! the code produce what the prompt already promised, rather than leaving a role
|
||||||
|
//! to fabricate it.
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use serde_json::json;
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
/// Template kind that triggers a harvest at launch.
|
||||||
|
pub const TEMPLATE_KIND: &str = "continuous_research";
|
||||||
|
|
||||||
|
/// Today's manifest, relative to the vault root.
|
||||||
|
pub fn manifest_path(date: &str) -> String {
|
||||||
|
format!("ContinuousResearch/{date}/harvest.jsonl")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// UTC date stamp, the same key the vault folders use.
|
||||||
|
pub fn today() -> String {
|
||||||
|
let now = time::OffsetDateTime::now_utc();
|
||||||
|
format!(
|
||||||
|
"{:04}-{:02}-{:02}",
|
||||||
|
now.year(),
|
||||||
|
now.month() as u8,
|
||||||
|
now.day()
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Run the harvest for a mission and leave a manifest the agents can read.
|
||||||
|
///
|
||||||
|
/// Non-fatal by contract: a launch whose harvest fails still starts its phases,
|
||||||
|
/// because a quiet day and a broken day must be distinguishable and the phase
|
||||||
|
/// itself is what reports which happened. What is NOT acceptable is failing
|
||||||
|
/// silently, so every outcome is logged with its counts.
|
||||||
|
pub async fn harvest_for_mission(
|
||||||
|
pool: &sqlx::PgPool,
|
||||||
|
blobs: &Arc<dyn cm_files::BlobStore>,
|
||||||
|
workspace_id: Uuid,
|
||||||
|
mission_id: Uuid,
|
||||||
|
topics: &[String],
|
||||||
|
per_topic: usize,
|
||||||
|
) -> Result<usize, String> {
|
||||||
|
let work_root = std::env::temp_dir().join("clawmates-library");
|
||||||
|
let run = crate::library::run_to_vault(
|
||||||
|
pool,
|
||||||
|
blobs,
|
||||||
|
workspace_id,
|
||||||
|
crate::routes::library::DEFAULT_CORPUS,
|
||||||
|
crate::routes::library::DEFAULT_VAULT_URL,
|
||||||
|
&work_root,
|
||||||
|
topics,
|
||||||
|
per_topic,
|
||||||
|
Some(mission_id),
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
let shelved = run.harvest.shelved.len();
|
||||||
|
// A quiet day is not a failure. `Harvest::healthy()` (nothing errored) is a
|
||||||
|
// different question from `added_anything()` (something new arrived), and
|
||||||
|
// collapsing them is the defect class this codebase keeps paying for.
|
||||||
|
eprintln!(
|
||||||
|
"continuous_research: mission {mission_id} harvested {} candidate(s), {} already had, \
|
||||||
|
{} shelved, {} failed",
|
||||||
|
run.harvest.candidates,
|
||||||
|
run.harvest.already_had,
|
||||||
|
shelved,
|
||||||
|
run.harvest.failed.len()
|
||||||
|
);
|
||||||
|
for (source_id, why) in &run.harvest.failed {
|
||||||
|
eprintln!("continuous_research: {source_id} not shelved: {why}");
|
||||||
|
}
|
||||||
|
Ok(shelved)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The manifest lines for a set of freshly shelved papers.
|
||||||
|
///
|
||||||
|
/// Shape matches what `templates/teams/continuous_research.toml` documents:
|
||||||
|
/// `{ source, url, title, snippet, first_seen, topic_tags }`.
|
||||||
|
pub fn manifest_lines(papers: &[crate::papers::Paper], first_seen: &str) -> String {
|
||||||
|
papers
|
||||||
|
.iter()
|
||||||
|
.map(|p| {
|
||||||
|
json!({
|
||||||
|
"source": p.source_id(),
|
||||||
|
"url": format!("https://arxiv.org/abs/{}", p.arxiv_id),
|
||||||
|
"title": p.title,
|
||||||
|
"snippet": p.summary.chars().take(400).collect::<String>(),
|
||||||
|
"first_seen": first_seen,
|
||||||
|
"topic_tags": [],
|
||||||
|
})
|
||||||
|
.to_string()
|
||||||
|
})
|
||||||
|
.collect::<Vec<_>>()
|
||||||
|
.join("\n")
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_manifest_path_matches_what_the_team_template_promises() {
|
||||||
|
// templates/teams/continuous_research.toml tells signal_harvester to
|
||||||
|
// write ContinuousResearch/<date>/harvest.jsonl. If this drifts, the
|
||||||
|
// agents read a file nothing writes and silently review nothing.
|
||||||
|
assert_eq!(
|
||||||
|
manifest_path("2026-08-17"),
|
||||||
|
"ContinuousResearch/2026-08-17/harvest.jsonl"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_date_stamp_is_zero_padded() {
|
||||||
|
let d = today();
|
||||||
|
assert_eq!(d.len(), 10, "YYYY-MM-DD, got {d:?}");
|
||||||
|
assert_eq!(d.matches('-').count(), 2, "{d:?}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One JSON object per line, and every key the template's prompt names —
|
||||||
|
/// an agent instructed to read `topic_tags` must not find it absent.
|
||||||
|
#[test]
|
||||||
|
fn manifest_lines_carry_every_documented_key() {
|
||||||
|
let p = crate::papers::Paper {
|
||||||
|
arxiv_id: "2401.12345".into(),
|
||||||
|
title: "A Paper".into(),
|
||||||
|
authors: vec!["A. Author".into()],
|
||||||
|
summary: "x".repeat(900),
|
||||||
|
published: "2026-08-17".into(),
|
||||||
|
pdf_url: "https://arxiv.org/pdf/2401.12345".into(),
|
||||||
|
};
|
||||||
|
let out = manifest_lines(std::slice::from_ref(&p), "2026-08-17");
|
||||||
|
assert_eq!(out.lines().count(), 1);
|
||||||
|
let v: serde_json::Value = serde_json::from_str(&out).expect("each line is JSON");
|
||||||
|
for key in ["source", "url", "title", "snippet", "first_seen", "topic_tags"] {
|
||||||
|
assert!(v.get(key).is_some(), "missing {key} in {v}");
|
||||||
|
}
|
||||||
|
assert_eq!(v["source"], "arxiv:2401.12345");
|
||||||
|
assert!(
|
||||||
|
v["snippet"].as_str().unwrap().chars().count() <= 400,
|
||||||
|
"snippet must be trimmed, not the whole abstract"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -24,6 +24,7 @@ mod mcp_skills;
|
|||||||
pub mod microvm_client;
|
pub mod microvm_client;
|
||||||
pub mod microvm_executor;
|
pub mod microvm_executor;
|
||||||
pub mod microvm_turn_executor;
|
pub mod microvm_turn_executor;
|
||||||
|
pub mod continuous_research;
|
||||||
pub mod mission_delivery;
|
pub mod mission_delivery;
|
||||||
pub mod mission_events;
|
pub mod mission_events;
|
||||||
pub mod mission_fs;
|
pub mod mission_fs;
|
||||||
|
|||||||
@@ -14,8 +14,8 @@ use crate::{ApiError, AppState, Authed};
|
|||||||
/// Default corpus + repo. Single-operator deployment, so these are constants
|
/// Default corpus + repo. Single-operator deployment, so these are constants
|
||||||
/// rather than another table to keep in sync; a second library becomes a
|
/// rather than another table to keep in sync; a second library becomes a
|
||||||
/// request field the day one exists.
|
/// request field the day one exists.
|
||||||
const DEFAULT_CORPUS: &str = "valhalla-vault";
|
pub const DEFAULT_CORPUS: &str = "valhalla-vault";
|
||||||
const DEFAULT_VAULT_URL: &str = "https://git.redclaw.dev/redclaw/valhalla-vault.git";
|
pub const DEFAULT_VAULT_URL: &str = "https://git.redclaw.dev/redclaw/valhalla-vault.git";
|
||||||
|
|
||||||
#[derive(Deserialize)]
|
#[derive(Deserialize)]
|
||||||
pub struct RunRequest {
|
pub struct RunRequest {
|
||||||
|
|||||||
@@ -0,0 +1,93 @@
|
|||||||
|
key = "continuous_research"
|
||||||
|
title = "Continuous Research"
|
||||||
|
blurb = "Harvest new arXiv papers, read them against your projects, and write a two-host podcast script for the morning."
|
||||||
|
requires_repo = true
|
||||||
|
|
||||||
|
# The vault, not `rust_sdlc`. Every other recipe defaults to the Rust SDLC team,
|
||||||
|
# which is why `continuous_research` has only ever been reachable as a checkbox
|
||||||
|
# in Advanced rather than as a mission type.
|
||||||
|
default_team_template = "continuous_research"
|
||||||
|
|
||||||
|
# ── How the papers arrive ────────────────────────────────────────────
|
||||||
|
#
|
||||||
|
# The harvest is NOT an agent phase. `library::run_to_vault` already does
|
||||||
|
# arXiv search → seen-set check → PDF fetch → blob shelf → vault note, and it
|
||||||
|
# takes a `mission_id` so the run is attributed. Asking an agent to redo that
|
||||||
|
# would be slower, non-deterministic, and would lose the `corpus_items`
|
||||||
|
# seen-set that is the entire reason a recurring mission knows what it already
|
||||||
|
# covered. `mission_orchestrator::on_launch` runs it before the phases start
|
||||||
|
# and drops a manifest at `ContinuousResearch/<date>/harvest.jsonl` — the path
|
||||||
|
# `templates/teams/continuous_research.toml` already tells the harvester role to
|
||||||
|
# write, now produced by the code that actually does the harvesting.
|
||||||
|
#
|
||||||
|
# The mission binds the VAULT repo, so `/mission/repo` is the card catalogue:
|
||||||
|
# the agents read the notes the harvest just wrote and commit their analysis and
|
||||||
|
# script back onto the run's own branch, never `main`.
|
||||||
|
|
||||||
|
[[phases]]
|
||||||
|
kind = "research"
|
||||||
|
order_idx = 0
|
||||||
|
[phases.config]
|
||||||
|
produces = ["md"]
|
||||||
|
default_topology = "hub_spoke"
|
||||||
|
# `research`, not a new `read` kind. An unrecognised kind falls through to
|
||||||
|
# `purposes_for`'s `_ => ["mission"]` and is absent from `PRODUCING_KINDS`
|
||||||
|
# (phase_runner.rs), so it would get a generic directive AND be exempt from the
|
||||||
|
# empty-delivery rule — a phase that could produce nothing and still pass. Two
|
||||||
|
# `research` phases differentiated by `task` keeps both guards.
|
||||||
|
task = """
|
||||||
|
Read today's harvested papers and judge them against the projects listed below.
|
||||||
|
|
||||||
|
Start from ContinuousResearch/<today>/harvest.jsonl — that is the list of papers \
|
||||||
|
that are NEW since the last run. Papers already covered are not in it, and you \
|
||||||
|
should not go looking for them.
|
||||||
|
|
||||||
|
For each paper write an entry in ContinuousResearch/<today>/analysis.md \
|
||||||
|
containing: what it actually does (not what its abstract claims), whether the \
|
||||||
|
evidence supports it, and — the part that matters — WHICH of the projects below \
|
||||||
|
it bears on and what concrete change it would imply. Name a file, a module or a \
|
||||||
|
roadmap item wherever you can.
|
||||||
|
|
||||||
|
Depth comes from the paper itself: the note carries the abstract, and `curl` on \
|
||||||
|
the arXiv abstract page gets you the rest. Do not review a paper from its title.
|
||||||
|
|
||||||
|
A paper with no bearing on any project is a real and useful finding — say so in \
|
||||||
|
one line and move on. Do not manufacture relevance.
|
||||||
|
|
||||||
|
PROJECTS THIS RESEARCH SERVES — replace this block when creating the mission:
|
||||||
|
(none configured yet)
|
||||||
|
"""
|
||||||
|
done_when = "ContinuousResearch/<today>/analysis.md exists and contains, for every paper in that day's harvest.jsonl, a judgement of the work and a statement of which project it bears on or that it bears on none"
|
||||||
|
max_iterations = 2
|
||||||
|
# Nothing here is compiled, so `on_green_tests` would gate on a suite that does
|
||||||
|
# not exist and land every branch `-wip`. The vault is prose.
|
||||||
|
commit_policy = "always"
|
||||||
|
|
||||||
|
[[phases]]
|
||||||
|
kind = "research"
|
||||||
|
order_idx = 1
|
||||||
|
[phases.config]
|
||||||
|
produces = ["md"]
|
||||||
|
default_topology = "hub_spoke"
|
||||||
|
task = """
|
||||||
|
Turn today's analysis into a podcast script for two hosts.
|
||||||
|
|
||||||
|
Read ContinuousResearch/<today>/analysis.md and write \
|
||||||
|
ContinuousResearch/<today>/script.md as a conversation between HOST and GUEST. \
|
||||||
|
Lead with what changed for our projects, not with a list of papers — the \
|
||||||
|
listener is on a treadmill, not at a desk.
|
||||||
|
|
||||||
|
Also write ContinuousResearch/<today>/episode.json:
|
||||||
|
{ "title": "<one line, under 80 chars>",
|
||||||
|
"highlights": ["<10-70 chars each, at most 5>"] }
|
||||||
|
Those bounds are the podcast API's, not a style preference — a highlight \
|
||||||
|
outside them is rejected.
|
||||||
|
|
||||||
|
Target seven minutes of speech, roughly 1,000 words. Say the specific thing: \
|
||||||
|
"this changes how we prune the HNSW graph in clawhdf5" beats "researchers \
|
||||||
|
propose a novel method". Skip a paper entirely rather than pad the episode \
|
||||||
|
with one that does not matter.
|
||||||
|
"""
|
||||||
|
done_when = "ContinuousResearch/<today>/script.md contains a two-host dialogue covering the analysis, and episode.json contains a title and a highlights array whose entries are each between 10 and 70 characters"
|
||||||
|
max_iterations = 2
|
||||||
|
commit_policy = "always"
|
||||||
Reference in New Issue
Block a user