---
name: duplicate-detection
description: Recognising that two harvested items are the same work, and why the seen-set is not enough on its own.
when_to_use: You are processing a harvest manifest and must avoid covering the same work twice.
tags: [research, judgement]
---

# The same work wearing different clothes

The harvest already dedupes by identity: `corpus_items` keys papers by
`arxiv:NNNN.NNNNN` with the version suffix stripped, so `2401.12345v3` and
`2401.12345v1` are one paper and a paper already shelved never reappears. That
is exact-identity dedup, and it is done for you.

What it cannot catch is the same *work* arriving under a different identity.

## The cases the seen-set misses

- **A paper and its blog post.** Different URLs, same result. Cover the paper;
  mention the post only if it adds an author's clarification.
- **Preprint then conference version.** Different arXiv id, often a different
  title, sometimes months apart. If the contribution is the same, it is one
  item — and the interesting fact is what changed in review.
- **The same group's incremental follow-up.** "X for Video" three months after
  "X". Genuinely new, but the listener needs the earlier one as context, so
  cover them together rather than as two unrelated items.
- **Two groups, one idea, same season.** Not duplicates. Say so explicitly —
  independent convergence is itself a signal that the idea is real.

## How to decide

Compare on **contribution**, not on title or authors:

1. What does it claim that nobody claimed before?
2. What experiment supports that specific claim?

If two items give the same answer to both, they are one item. If they differ on
either, they are two — even from the same lab, even with near-identical titles.

## When you merge, say so

Never silently drop the second one. A merged entry states it:

> Also appeared as a NeurIPS submission with an extra ablation; same core
> result, so covered once here.

Silent dedup is indistinguishable from a harvest that missed something, and the
whole point of a standing sweep is knowing which of the two happened.