Files
clawmates/skills/research/duplicate-detection.md
T
Omar SobhandClaude Opus 5 a2d7e3ea92
deploy / test (push) Successful in 4m4s
deploy / build (push) Successful in 1m45s
feat(skills): author the Continuous Research skills, and make its roles honest
The boot log has said `continuous_research — 4 role skills bound, 8 unresolved`
on every start. Those eight roles ran without the instructions their template
promised them, silently: `skills_loader` reports the miss and carries on.

Worse than the missing files was what the prompts described. v1 told the
harvester to sweep "RSS feeds, GitHub trending, HN front page, YouTube /
podcast RSS" — none of which exist. `harvest.rs` searches arXiv and nothing
else. A role prompt describing a machine that was never built is the failure
this codebase keeps paying for, so v2 describes the machine that exists.

Roles now match the pipeline: paper_reader (the harvest already ran; read the
manifest and the papers), signal_ranker (unchanged in spirit), script_writer
(the podcast half, which had no role at all).

Seven skills authored under skills/research/, kebab-case to match the loader —
team_template_loader.rs:177-181 documents the snake_case/kebab-case trap that
already unbinds skills elsewhere:

  arxiv-daily                  what the harvest guarantees, so an agent does
                               NOT re-search arXiv and corrupt the seen-set
  paper-to-project-relevance   name a file or roadmap item, or say "no bearing"
  duplicate-detection          the seen-set catches identity; this catches the
                               same work under a different id
  signal-to-noise-ranking      novelty/relevance/depth, and the two biases to
                               resist (recency up, inconvenience down)
  executive-summary-writing    what it is / why it matters / what to do — decide,
                               do not hedge
  obsidian-vault-conventions   the vault is a human's live workspace; never
                               main, never reorganise, hash the body not the file
  podcast-dialogue-writing     write for someone on a treadmill; the 10-70 char
                               highlight bound is the API's, not a style rule

`web_fetch` dropped from mcp_bundles: runtime_provision.rs binds every mission
claw to `["clawmates_door"]` and never reads that field, so declaring it
instructed roles to use a tool that never arrived. The prompts say `curl` via
Bash, which is what they actually have.

Boot now reports `continuous_research — 11 role skills bound`, with no
unresolved clause. 344 tests pass.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-17 14:44:03 -07:00

49 lines
2.1 KiB
Markdown

---
name: duplicate-detection
description: Recognising that two harvested items are the same work, and why the seen-set is not enough on its own.
when_to_use: You are processing a harvest manifest and must avoid covering the same work twice.
tags: [research, judgement]
---
# The same work wearing different clothes
The harvest already dedupes by identity: `corpus_items` keys papers by
`arxiv:NNNN.NNNNN` with the version suffix stripped, so `2401.12345v3` and
`2401.12345v1` are one paper and a paper already shelved never reappears. That
is exact-identity dedup, and it is done for you.
What it cannot catch is the same *work* arriving under a different identity.
## The cases the seen-set misses
- **A paper and its blog post.** Different URLs, same result. Cover the paper;
mention the post only if it adds an author's clarification.
- **Preprint then conference version.** Different arXiv id, often a different
title, sometimes months apart. If the contribution is the same, it is one
item — and the interesting fact is what changed in review.
- **The same group's incremental follow-up.** "X for Video" three months after
"X". Genuinely new, but the listener needs the earlier one as context, so
cover them together rather than as two unrelated items.
- **Two groups, one idea, same season.** Not duplicates. Say so explicitly —
independent convergence is itself a signal that the idea is real.
## How to decide
Compare on **contribution**, not on title or authors:
1. What does it claim that nobody claimed before?
2. What experiment supports that specific claim?
If two items give the same answer to both, they are one item. If they differ on
either, they are two — even from the same lab, even with near-identical titles.
## When you merge, say so
Never silently drop the second one. A merged entry states it:
> Also appeared as a NeurIPS submission with an extra ablation; same core
> result, so covered once here.
Silent dedup is indistinguishable from a harvest that missed something, and the
whole point of a standing sweep is knowing which of the two happened.