Omar SobhandClaude Opus 5
48606f2fe3
Mission 01a0c940 ran the paper triage live for the first time: 10 papers, 10 tagged, 10 scored — and the relevance scores were 2.93, 2.94, 2.97, 2.97, 2.98, 2.99, 2.99, 2.99, 3.00, 3.00. A spread of 0.07 across a 4-level scale, every answer confident, no ranking information at all. Of course: the harvest runs the operator's own arXiv topic queries, so every paper in the file is about agents by construction. Asked on the same ten abstracts, 'how actionable is it' saturated the same way (spread 0.20). What separated them was the strength of the evidence behind the claims: 1.36 (a benchmark paper) to 3.00 (measured on real systems with ablations), spread 1.64 — and what KIND of paper it is (method / benchmark / measurement / survey / position), with the confidence of that call beside it so an unplaceable paper reads as unplaceable. The manifest now carries those two and no relevance number, and arxiv-daily.md tells the reading agents what each means and why there is no relevance. The general rule, since this class of mistake is invisible — a saturated score looks exactly like a working feature: patterns::spread() with SATURATED_BELOW, and triage_papers warns when a live harvest's scores span less than that. A question that returns the same number for everything is a defect in the question, not a fact about the population. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
440 lines
18 KiB
Rust
440 lines
18 KiB
Rust
//! The harvest half of a Continuous Research mission.
|
||
//!
|
||
//! Finding papers is NOT agent work. `library::run_to_vault` already does arXiv
|
||
//! search → seen-set check → PDF fetch → blob shelf → vault note, deterministically
|
||
//! and in seconds, and it takes a `mission_id` so the run is attributed. Asking an
|
||
//! agent to redo it would be slower, non-repeatable, and would abandon the
|
||
//! `corpus_items` seen-set — which is the entire reason a recurring mission knows
|
||
//! what it already covered. `corpus.rs` puts it plainly: "A recurring mission's
|
||
//! hard problem is not running the agent — that is 23 seconds — it is knowing
|
||
//! what it already did last time."
|
||
//!
|
||
//! So the harvest runs here, at launch, and the agents start from its output.
|
||
//!
|
||
//! The manifest path (`ContinuousResearch/<date>/harvest.jsonl`) is not invented:
|
||
//! `templates/teams/continuous_research.toml` has told the `signal_harvester`
|
||
//! role to write exactly that file since the template was authored. This makes
|
||
//! the code produce what the prompt already promised, rather than leaving a role
|
||
//! to fabricate it.
|
||
|
||
use std::sync::Arc;
|
||
|
||
use serde_json::json;
|
||
use uuid::Uuid;
|
||
|
||
/// Template kind that triggers a harvest at launch.
|
||
pub const TEMPLATE_KIND: &str = "continuous_research";
|
||
|
||
/// Today's manifest, relative to the vault root.
|
||
pub fn manifest_path(date: &str) -> String {
|
||
format!("ContinuousResearch/{date}/harvest.jsonl")
|
||
}
|
||
|
||
/// UTC date stamp, the same key the vault folders use.
|
||
pub fn today() -> String {
|
||
let now = time::OffsetDateTime::now_utc();
|
||
format!(
|
||
"{:04}-{:02}-{:02}",
|
||
now.year(),
|
||
now.month() as u8,
|
||
now.day()
|
||
)
|
||
}
|
||
|
||
/// The arXiv queries this mission tracks.
|
||
///
|
||
/// `config.topics` on the mission when the operator set them, otherwise the
|
||
/// project-wide defaults. Read from config rather than a new column because the
|
||
/// wizard already round-trips `config` untouched, so a topic list needs no
|
||
/// schema change and no UI work to reach here.
|
||
pub fn topics_for(config: &serde_json::Value) -> Vec<String> {
|
||
config
|
||
.get("topics")
|
||
.and_then(|v| v.as_array())
|
||
.map(|a| {
|
||
a.iter()
|
||
.filter_map(|t| t.as_str())
|
||
.map(str::trim)
|
||
.filter(|t| !t.is_empty())
|
||
.map(str::to_string)
|
||
.collect::<Vec<_>>()
|
||
})
|
||
.filter(|t: &Vec<String>| !t.is_empty())
|
||
.unwrap_or_else(crate::library::default_topics)
|
||
}
|
||
|
||
/// Run the harvest for a mission and leave a manifest the agents can read.
|
||
///
|
||
/// Non-fatal by contract: a launch whose harvest fails still starts its phases,
|
||
/// because a quiet day and a broken day must be distinguishable and the phase
|
||
/// itself is what reports which happened. What is NOT acceptable is failing
|
||
/// silently, so every outcome is logged with its counts.
|
||
pub async fn harvest_for_mission(
|
||
pool: &sqlx::PgPool,
|
||
blobs: &Arc<dyn cm_files::BlobStore>,
|
||
workspace_id: Uuid,
|
||
mission_id: Uuid,
|
||
topics: &[String],
|
||
per_topic: usize,
|
||
) -> Result<Vec<crate::papers::Paper>, String> {
|
||
let work_root = std::env::temp_dir().join("clawmates-library");
|
||
let run = crate::library::run_to_vault(
|
||
pool,
|
||
blobs,
|
||
workspace_id,
|
||
crate::routes::library::DEFAULT_CORPUS,
|
||
crate::routes::library::DEFAULT_VAULT_URL,
|
||
&work_root,
|
||
topics,
|
||
per_topic,
|
||
Some(mission_id),
|
||
)
|
||
.await?;
|
||
|
||
let shelved = run.harvest.shelved.len();
|
||
// A quiet day is not a failure. `Harvest::healthy()` (nothing errored) is a
|
||
// different question from `added_anything()` (something new arrived), and
|
||
// collapsing them is the defect class this codebase keeps paying for.
|
||
eprintln!(
|
||
"continuous_research: mission {mission_id} harvested {} candidate(s), {} already had, \
|
||
{} shelved, {} failed",
|
||
run.harvest.candidates,
|
||
run.harvest.already_had,
|
||
shelved,
|
||
run.harvest.failed.len()
|
||
);
|
||
for (source_id, why) in &run.harvest.failed {
|
||
eprintln!("continuous_research: {source_id} not shelved: {why}");
|
||
}
|
||
Ok(run.harvest.papers)
|
||
}
|
||
|
||
/// Write the run manifest into the MISSION's checkout.
|
||
///
|
||
/// Not into the vault. The manifest is per-RUN input for one mission, and the
|
||
/// vault path is per-DATE and shared, so a second run on the same day rewrites
|
||
/// a file that already exists — which `auto_merge` correctly refuses, because
|
||
/// it only merges provably additive diffs:
|
||
///
|
||
/// "diff is not additive (1 non-add change(s), first:
|
||
/// M ContinuousResearch/2026-08-18/harvest.jsonl); left for a human"
|
||
///
|
||
/// The branch was then left unmerged, `main` kept the previous run's manifest,
|
||
/// and the next mission cloned STALE papers while every log line said the
|
||
/// harvest succeeded. Writing into the checkout keeps the vault additive and
|
||
/// gives each mission exactly its own papers. The agents commit it alongside
|
||
/// their analysis through the normal delivery path.
|
||
pub fn write_manifest(
|
||
checkout: &std::path::Path,
|
||
papers: &[crate::papers::Paper],
|
||
date: &str,
|
||
triage: &[PaperTriage],
|
||
) -> Result<std::path::PathBuf, String> {
|
||
let rel = manifest_path(date);
|
||
let abs = checkout.join(&rel);
|
||
if let Some(parent) = abs.parent() {
|
||
std::fs::create_dir_all(parent).map_err(|e| format!("create {}: {e}", parent.display()))?;
|
||
}
|
||
let body = manifest_lines(papers, date, triage);
|
||
std::fs::write(&abs, format!("{body}\n")).map_err(|e| format!("write {}: {e}", abs.display()))?;
|
||
Ok(abs)
|
||
}
|
||
|
||
/// What the decision model said about one harvested paper. `topic_tags`
|
||
/// was written as `[]` on every manifest line from the day the manifest
|
||
/// existed — a slot the agents were told to read and nothing filled.
|
||
///
|
||
/// **A score has to discriminate within the population it scores.** The
|
||
/// first version asked "how relevant is this to an agent platform" on a
|
||
/// four-level scale, and MEASURED on the ten papers of mission 01a0c940 it
|
||
/// answered 2.93–3.00 — a spread of 0.07, no ranking information at all.
|
||
/// Of course: the harvest runs the operator's own arXiv topic queries, so
|
||
/// every paper in it is about agents by construction. "How actionable is
|
||
/// it" saturated the same way (spread 0.20). What did discriminate on the
|
||
/// same ten abstracts was the strength of the evidence behind the claims
|
||
/// (1.36–3.00, spread 1.64) and what KIND of paper it is. So the manifest
|
||
/// carries those two and no relevance number: a digest phase ranking
|
||
/// already-relevant papers needs to know which ones measured something.
|
||
#[derive(Debug, Clone, serde::Serialize, Default)]
|
||
pub struct PaperTriage {
|
||
pub topic_tags: Vec<String>,
|
||
/// `benchmark` | `method` | `measurement` | `survey` | `position`, and
|
||
/// how peaked that choice was — a paper the model cannot place is one
|
||
/// the reader should look at rather than trust the label for.
|
||
pub kind: Option<String>,
|
||
pub kind_confidence: Option<f64>,
|
||
/// 0 = position piece, no experiments … 3 = measured on real systems
|
||
/// with ablations. `None` when no triage ran (no key, or a failed call).
|
||
pub evidence: Option<f64>,
|
||
pub evidence_confidence: Option<f64>,
|
||
}
|
||
|
||
const EVIDENCE_LEVELS: [&str; 4] = [
|
||
"Position, opinion, or framework description; no experiments.",
|
||
"Illustrative examples, a demo, or a single small case study.",
|
||
"Benchmarked with numbers, on a suite the authors assembled.",
|
||
"Measured on real systems or at scale, with ablations or failure analysis.",
|
||
];
|
||
|
||
const PAPER_KINDS: [(&str, &str); 5] = [
|
||
("benchmark", "Introduces a dataset or benchmark to measure something"),
|
||
("method", "Proposes a technique, architecture, or algorithm"),
|
||
("measurement", "Measures the behaviour of existing systems without proposing a new one"),
|
||
("survey", "Reviews or categorises a body of existing work"),
|
||
("position", "Argues a viewpoint or proposes an agenda"),
|
||
];
|
||
|
||
/// Triage every harvested paper in one call each. Best-effort: a missing key
|
||
/// or a failed call leaves that paper's tags empty and relevance `None`, the
|
||
/// state the manifest has always been in.
|
||
pub async fn triage_papers(
|
||
papers: &[crate::papers::Paper],
|
||
topics: &[String],
|
||
) -> Vec<PaperTriage> {
|
||
use cm_decide::{Answer, Decider as _, Question};
|
||
let Some(jev) = cm_decide::jev::Jev::from_env() else {
|
||
return vec![PaperTriage::default(); papers.len()];
|
||
};
|
||
// Topics are arXiv query strings; the option NAME the model sees is the
|
||
// readable form (`all:"agent memory" AND all:"long-term"` → `agent memory
|
||
// long-term`), the value the query itself for precision.
|
||
let mut criteria: std::collections::BTreeMap<String, Option<String>> = topics
|
||
.iter()
|
||
.map(|t| (readable_topic(t), Some(t.clone())))
|
||
.collect();
|
||
criteria.insert("none".into(), Some("Fits none of the listed topics".into()));
|
||
let questions: std::collections::BTreeMap<String, Question> = [
|
||
(
|
||
"topic".to_string(),
|
||
Question::Choice {
|
||
instructions: "Which of these research topics is this paper about?".into(),
|
||
criteria,
|
||
},
|
||
),
|
||
(
|
||
"kind".to_string(),
|
||
Question::choice(
|
||
"What kind of paper is this?",
|
||
PAPER_KINDS.map(|(k, d)| (k, Some(d))),
|
||
),
|
||
),
|
||
(
|
||
"evidence".to_string(),
|
||
Question::score(
|
||
"How strong is the evidence behind this paper's claims?",
|
||
EVIDENCE_LEVELS,
|
||
),
|
||
),
|
||
]
|
||
.into_iter()
|
||
.collect();
|
||
|
||
let mut out = Vec::with_capacity(papers.len());
|
||
for p in papers {
|
||
let state = format!("Title: {}\n\nAbstract: {}", p.title, p.summary);
|
||
let decided = tokio::time::timeout(
|
||
std::time::Duration::from_secs(10),
|
||
jev.decide(&state, &questions),
|
||
)
|
||
.await;
|
||
let mut t = PaperTriage::default();
|
||
match decided {
|
||
Ok(Ok(d)) => {
|
||
if let Some(Answer::Choice { probabilities, .. }) = d.answers.get("topic") {
|
||
let mut tags: Vec<(String, f64)> = probabilities
|
||
.iter()
|
||
.filter(|(k, p)| k.as_str() != "none" && **p >= 0.3)
|
||
.map(|(k, p)| (k.clone(), *p))
|
||
.collect();
|
||
tags.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
|
||
t.topic_tags = tags.into_iter().map(|(k, _)| k).collect();
|
||
}
|
||
if let Some(Answer::Choice { choice, confidence, .. }) = d.answers.get("kind") {
|
||
t.kind = Some(choice.clone());
|
||
t.kind_confidence = Some((*confidence * 100.0).round() / 100.0);
|
||
}
|
||
if let Some(Answer::Score { score, confidence, .. }) = d.answers.get("evidence") {
|
||
t.evidence = Some((*score * 100.0).round() / 100.0);
|
||
t.evidence_confidence = Some((*confidence * 100.0).round() / 100.0);
|
||
}
|
||
}
|
||
Ok(Err(e)) => eprintln!("continuous_research: triage of {} failed: {e}", p.arxiv_id),
|
||
Err(_) => eprintln!("continuous_research: triage of {} timed out", p.arxiv_id),
|
||
}
|
||
out.push(t);
|
||
}
|
||
let tagged = out.iter().filter(|t| !t.topic_tags.is_empty()).count();
|
||
let scores: Vec<f64> = out.iter().filter_map(|t| t.evidence).collect();
|
||
eprintln!(
|
||
"continuous_research: triaged {} paper(s) with {}: {tagged} tagged, {} with evidence scored",
|
||
papers.len(),
|
||
jev.name(),
|
||
scores.len()
|
||
);
|
||
// A score that came back the same for every paper ranked nothing. Said
|
||
// out loud because the first version of this question did exactly that
|
||
// and looked like a working feature — ten confident numbers, no
|
||
// information. See `PaperTriage`.
|
||
let spread = cm_decide::patterns::spread(&scores);
|
||
if scores.len() > 2 && spread < cm_decide::patterns::SATURATED_BELOW {
|
||
eprintln!(
|
||
"continuous_research: WARNING — evidence scores span only {spread:.2} across \
|
||
{} papers. The question is not separating this harvest; the ranking phase \
|
||
gets no signal from it.",
|
||
scores.len()
|
||
);
|
||
}
|
||
out
|
||
}
|
||
|
||
/// `all:"agent memory" AND all:"long-term"` → `agent memory long-term`.
|
||
fn readable_topic(query: &str) -> String {
|
||
let words: Vec<&str> = query
|
||
.split(|c: char| c == '"' || c.is_whitespace() || c == '(' || c == ')')
|
||
.filter(|w| !w.is_empty())
|
||
.filter(|w| !matches!(*w, "AND" | "OR" | "NOT"))
|
||
.map(|w| w.strip_prefix("all:").unwrap_or(w))
|
||
.map(|w| w.strip_prefix("ti:").unwrap_or(w))
|
||
.map(|w| w.strip_prefix("abs:").unwrap_or(w))
|
||
.filter(|w| !w.is_empty())
|
||
.collect();
|
||
words.join(" ")
|
||
}
|
||
|
||
/// The manifest lines for a set of freshly shelved papers.
|
||
///
|
||
/// Shape matches what `skills/research/arxiv-daily.md` documents:
|
||
/// `{ source, url, title, snippet, first_seen, topic_tags }`, plus `kind`
|
||
/// and `evidence` since 2026-09-22 (see [`PaperTriage`]). `triage` is
|
||
/// positional with `papers`; shorter means the rest are untriaged.
|
||
pub fn manifest_lines(
|
||
papers: &[crate::papers::Paper],
|
||
first_seen: &str,
|
||
triage: &[PaperTriage],
|
||
) -> String {
|
||
papers
|
||
.iter()
|
||
.enumerate()
|
||
.map(|(i, p)| {
|
||
let t = triage.get(i).cloned().unwrap_or_default();
|
||
json!({
|
||
"source": p.source_id(),
|
||
"url": format!("https://arxiv.org/abs/{}", p.arxiv_id),
|
||
"title": p.title,
|
||
"snippet": p.summary.chars().take(400).collect::<String>(),
|
||
"first_seen": first_seen,
|
||
"topic_tags": t.topic_tags,
|
||
"kind": t.kind.as_ref().map(|k| json!({
|
||
"is": k,
|
||
"confidence": t.kind_confidence,
|
||
})),
|
||
"evidence": t.evidence.map(|e| json!({
|
||
"score": e,
|
||
"confidence": t.evidence_confidence,
|
||
"scale": "0 position piece, no experiments … 3 measured on real systems with ablations",
|
||
})),
|
||
})
|
||
.to_string()
|
||
})
|
||
.collect::<Vec<_>>()
|
||
.join("\n")
|
||
}
|
||
|
||
#[cfg(test)]
|
||
mod tests {
|
||
use super::*;
|
||
|
||
#[test]
|
||
fn the_manifest_path_matches_what_the_team_template_promises() {
|
||
// templates/teams/continuous_research.toml tells signal_harvester to
|
||
// write ContinuousResearch/<date>/harvest.jsonl. If this drifts, the
|
||
// agents read a file nothing writes and silently review nothing.
|
||
assert_eq!(
|
||
manifest_path("2026-08-17"),
|
||
"ContinuousResearch/2026-08-17/harvest.jsonl"
|
||
);
|
||
}
|
||
|
||
/// An operator's topic list must win over the defaults, and a blank or
|
||
/// missing list must fall back rather than harvesting nothing.
|
||
#[test]
|
||
fn topics_come_from_config_and_fall_back_when_absent() {
|
||
assert_eq!(
|
||
topics_for(&serde_json::json!({"topics": ["world models", " robots "]})),
|
||
vec!["world models".to_string(), "robots".to_string()],
|
||
"operator topics win, and are trimmed"
|
||
);
|
||
for empty in [
|
||
serde_json::json!({}),
|
||
serde_json::json!({"topics": []}),
|
||
serde_json::json!({"topics": [" "]}),
|
||
] {
|
||
assert_eq!(
|
||
topics_for(&empty),
|
||
crate::library::default_topics(),
|
||
"an absent or blank list must fall back, not harvest nothing: {empty}"
|
||
);
|
||
}
|
||
}
|
||
|
||
#[test]
|
||
fn arxiv_queries_become_readable_option_names() {
|
||
assert_eq!(
|
||
readable_topic(r#"all:"agent memory" AND all:"long-term""#),
|
||
"agent memory long-term"
|
||
);
|
||
assert_eq!(
|
||
readable_topic(r#"all:"agentic topology" OR all:"multi-agent topology""#),
|
||
"agentic topology multi-agent topology"
|
||
);
|
||
}
|
||
|
||
#[test]
|
||
fn the_date_stamp_is_zero_padded() {
|
||
let d = today();
|
||
assert_eq!(d.len(), 10, "YYYY-MM-DD, got {d:?}");
|
||
assert_eq!(d.matches('-').count(), 2, "{d:?}");
|
||
}
|
||
|
||
/// One JSON object per line, and every key the template's prompt names —
|
||
/// an agent instructed to read `topic_tags` must not find it absent.
|
||
#[test]
|
||
fn manifest_lines_carry_every_documented_key() {
|
||
let p = crate::papers::Paper {
|
||
arxiv_id: "2401.12345".into(),
|
||
title: "A Paper".into(),
|
||
authors: vec!["A. Author".into()],
|
||
summary: "x".repeat(900),
|
||
published: "2026-08-17".into(),
|
||
pdf_url: "https://arxiv.org/pdf/2401.12345".into(),
|
||
};
|
||
let out = manifest_lines(std::slice::from_ref(&p), "2026-08-17", &[]);
|
||
assert_eq!(out.lines().count(), 1);
|
||
let v: serde_json::Value = serde_json::from_str(&out).expect("each line is JSON");
|
||
for key in ["source", "url", "title", "snippet", "first_seen", "topic_tags", "kind", "evidence"] {
|
||
assert!(v.get(key).is_some(), "missing {key} in {v}");
|
||
}
|
||
// Untriaged: the slots are there and empty, as they always were.
|
||
assert_eq!(v["topic_tags"], serde_json::json!([]));
|
||
assert!(v["kind"].is_null() && v["evidence"].is_null());
|
||
// Triaged: the tags, the kind and the evidence score land on the line.
|
||
let t = PaperTriage {
|
||
topic_tags: vec!["agent memory long-term".into()],
|
||
kind: Some("benchmark".into()),
|
||
kind_confidence: Some(0.91),
|
||
evidence: Some(1.36),
|
||
evidence_confidence: Some(0.62),
|
||
};
|
||
let out = manifest_lines(std::slice::from_ref(&p), "2026-08-17", std::slice::from_ref(&t));
|
||
let v: serde_json::Value = serde_json::from_str(&out).unwrap();
|
||
assert_eq!(v["topic_tags"][0], "agent memory long-term");
|
||
assert_eq!(v["kind"]["is"], "benchmark");
|
||
assert_eq!(v["evidence"]["score"], 1.36);
|
||
assert_eq!(v["source"], "arxiv:2401.12345");
|
||
assert!(
|
||
v["snippet"].as_str().unwrap().chars().count() <= 400,
|
||
"snippet must be trimmed, not the whole abstract"
|
||
);
|
||
}
|
||
}
|