blob: size-based LRU eviction + auto-cap in the GC ticker

Orphan-chunk GC alone doesn't stop unbounded growth: as long as
fingerprint→blob refs keep getting PutRef'd, the manifest set keeps
growing and no chunk is ever an orphan.

* `BlobStore::evict_to_size_cap(max_bytes)` — walks manifests oldest
  first by mtime, deletes them, refcount-decrements each chunk they
  used, unlinks + reclaims size for any chunk whose refcount hits
  zero. Shared chunks stay put until the last blob referencing them
  is evicted.
* `ManifestSummary` internal type keeps the diff-set bookkeeping
  cheap (one HashMap<ChunkHash, u32>, no repeated tree walks).
* `claw-store cluster-gc --evict-to-gb <N>` extends the CLI: still
  runs the orphan sweep first, then optionally caps the store.
* Config: `cluster.blob_max_gb: Option<u64>`. The auto-GC ticker
  runs eviction after every orphan sweep when this is set. Silent
  when the store is already under cap; INFO log when it evicts.

+3 tests:
- evict_to_size_cap_reclaims_oldest_blobs_first: 3 blobs with
  distinct mtimes, cap below combined size → oldest evicted,
  newer blobs survive
- evict_keeps_shared_chunks_when_still_referenced: guards the
  refcount decrement path (content-addressed dedup keeps identical
  content as one blob → chunk survives until manifest deleted)
- evict_on_empty_store_is_a_noop: sanity

257 tests pass (baseline +3). Pre-existing macOS failure unchanged.
This commit is contained in:
Omar Sobh
2026-07-12 06:35:17 -07:00
parent 84aa758fd4
commit 2f3055a3aa
5 changed files with 319 additions and 8 deletions
+9
View File
@@ -161,6 +161,13 @@ pub struct ClusterConfig {
/// Typical value: `6` hours on a runner cache.
#[serde(default)]
pub gc_interval_hours: Option<u64>,
/// Field finding 2026-07-12: total blob-store size cap in GiB.
/// When set, the auto-GC ticker runs `evict_to_size_cap` after
/// its orphan sweep, deleting oldest manifests until the
/// live-referenced footprint sits at or below this bound.
/// Absent means "grow unbounded".
#[serde(default)]
pub blob_max_gb: Option<u64>,
}
/// Compute the default RPC address for a gossip address: same IP, port + 1.
@@ -420,6 +427,7 @@ tailscale_addr = "100.64.1.5:7701"
blob_store_root: None,
prom_bind: None,
gc_interval_hours: None,
blob_max_gb: None,
};
let err = cluster.validate().unwrap_err().to_string();
assert!(err.contains("no bind address"), "unexpected error: {err}");
@@ -451,6 +459,7 @@ tailscale_addr = "100.64.1.5:7701"
blob_store_root: None,
prom_bind: None,
gc_interval_hours: None,
blob_max_gb: None,
};
let err = cluster.validate().unwrap_err().to_string();
assert!(