4ea1cbed2eb719d5b04daaed70b4c3902ca0768d
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar Sobh
|
e564b0ce89 |
Phase 7d: snapshot primitives + CLI
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
A snapshot is a named, immutable point-in-time record of every blob
live in the store. It's NOT a data copy — blobs are content-addressed
and already live under blobs/. A snapshot is a JSON reference set at
<root>/snapshots/<name>.json.
Why:
* Rollback anchor before risky migrations.
* Retention pin: combined with the Phase 4a pin-aware LRU eviction,
operators can guarantee "these blobs stay on disk N days".
* Audit: "which blobs existed at release time?"
New module cluster::snapshot:
* SnapshotStore::create(name, blob_store, created_at)
* SnapshotStore::get(name) / list() / delete(name)
* SnapshotManifest { name, created_at_unix, blob_ids }
* SnapshotSummary for cheap list rendering (no blob-list slurp).
BlobStore gains list_blob_ids() — walks blobs/**/*.manifest.json
and returns the blob id set. Manifests only, no chunk reads.
New CLI commands:
* claw-store cluster-snapshot-create --name <>
* claw-store cluster-snapshot-list
* claw-store cluster-snapshot-show --name <>
* claw-store cluster-snapshot-delete --name <>
Semantics:
* Snapshots are immutable: create with existing name errors, does
not clobber. Delete-then-create if you really want to overwrite.
* delete() removes only the reference file. Never touches blob
data — protects against operators nuking live data by pruning
snapshots.
* list() sorts by created_at_unix ascending — oldest first so
triage picks pruning candidates quickly.
* blob_ids are sorted at write time so the same content on two
nodes yields byte-identical snapshot files.
* Names validated: no /, \\, NUL, control chars; max 512 bytes.
+8 tests covering create+capture, immutability, get-missing,
list-ordering, delete truth-values, delete-doesn't-touch-blobs,
name-validation, and sorted round-trip.
353 tests pass (+8). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
|
||
|
Omar Sobh
|
cf0a07099d |
Phase 7b: chunk-level repair library
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Failing after 3s
New primitive: BlobStore::repair_chunks(chunks, fetch) → RepairReport. Consumer flow: cluster-scrub returns a list of (blob, chunk) bad pairs. cluster-repair (next slice) will hand the chunk hashes here with a fetcher that walks peers via HasChunk/GetChunk. This PR is the library-only half — no peer wiring — so it's testable in isolation and reusable by callers who already have a chunk source. Fetcher contract: * Ok(Some(bytes)) → put locally, count repaired * Ok(None) → nobody has it, record as unrecoverable * Err(e) → per-chunk error, batch continues Guardrails: * Bytes are re-hashed by put_chunk before writing. A peer that returns wrong bytes for a hash cannot corrupt us further. * Duplicate chunk hashes in the input dedupe → fetcher called exactly once per unique chunk. Matters because scrub reports shared chunks once per owning manifest. * Errors on one chunk never abort the batch — the remaining chunks still get their shot. * Repair overwrites a corrupt file: unlink-then-put_chunk, since put_chunk itself is write-if-absent. NotFound on unlink is fine (missing-chunk case). +4 tests: - repair_writes_fetched_bytes_and_marks_repaired (happy: corrupt → repair → post-scrub clean) - repair_records_unrecoverable_when_fetcher_returns_none - repair_records_error_and_continues_batch (batch survives one chunk's error) - repair_dedups_duplicate_chunks_in_input (fetcher called exactly once for 3 identical hashes) 345 tests pass (+4). Pre-existing macOS hot::tests::test_project_target_size_bytes failure unchanged. |
||
|
Omar Sobh
|
701861787f |
Phase 7a: read-only fsck for the blob store
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 12s
New primitive: BlobStore::scrub_all() → ScrubReport. Walks every .manifest.json under blobs/, for each referenced chunk reads the file from disk and recomputes BLAKE3. Verdict per chunk: * file absent → missing * hash mismatch → corrupt * match → ok Design points: * Read-only. Never touches disk state. Safe against a live daemon — worst case a chunk lands mid-scrub and is skipped this pass. * Per-reference counting: a bad chunk that N manifests depend on shows up as N corrupt entries so operators see the full blast radius. But each unique chunk is hashed exactly once via an in-memory verdict cache. * Report holds explicit (blob_id, chunk_hash) pairs for every bad chunk so the fix path (repair in Phase 7b) has enough info to act. CLI: `claw-store cluster-scrub [--verbose]`. Non-zero exit when integrity issues exist so cron / CI notice. +4 tests: - scrub_reports_all_ok_when_store_is_healthy - scrub_detects_corrupt_chunk (owner blob id preserved) - scrub_detects_missing_chunk (owner blob id preserved) - scrub_dedups_shared_chunk_hashing_once (shared chunk, 2 owners reported, single disk read) 341 tests pass (+4). Pre-existing macOS hot::tests::test_project_target_size_bytes failure unchanged. |
||
|
Omar Sobh
|
5be11a11b0 |
Phase 4a: pin-aware LRU eviction
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
A `claw-cargo pin` used to be silently vulnerable to the size-cap
eviction ticker — the tag existed but the underlying blob could get
LRU'd out, leaving a dangling reference. Now tags act as
retention markers: any blob referenced by any tag (stamped or
legacy) is protected from `evict_to_size_cap`.
* `BlobStore::evict_to_size_cap_with_pins(max_bytes, pinned_set)` —
same LRU-by-mtime pass, but pinned blob IDs skip the eviction
loop. Existing `evict_to_size_cap` is now a thin wrapper with an
empty pin set (100% backward compat).
* `TagStore::pinned_blob_values()` — unions every 32-byte value
referenced by any tag across `tags/` (legacy) and `tags-v2/`
(Phase 3c stamped). Dedupes naturally.
* Auto-GC ticker in `ClusterServices` now collects the pin set on
every eviction pass and passes it in. Log fields include
`pinned_blobs = N` so operators can see the retention set size.
* `claw-store cluster-gc --evict-to-gb N` CLI opens the tag store
the same way, prints `pinned blobs: N` in the report.
+3 tests:
- evict_with_pins_protects_pinned_blobs_from_eviction — 3 blobs
ordered oldest→newest, pin the oldest; without pins LRU would
evict it; with pins the next-oldest goes instead. Guards the
main semantic.
- evict_with_pins_stops_when_pinned_footprint_dominates —
everything pinned + cap = 0 → no-op. Guards the "operator asked
for the impossible" case.
- pinned_blob_values_unions_both_stores — legacy tag with value V1,
stamped tag with value V2, second stamped tag also referencing
V1 → set contains {V1, V2}. Dedupe check.
283 tests pass (baseline +3). Pre-existing macOS failure unchanged.
|
||
|
Omar Sobh
|
2f3055a3aa |
blob: size-based LRU eviction + auto-cap in the GC ticker
Orphan-chunk GC alone doesn't stop unbounded growth: as long as fingerprint→blob refs keep getting PutRef'd, the manifest set keeps growing and no chunk is ever an orphan. * `BlobStore::evict_to_size_cap(max_bytes)` — walks manifests oldest first by mtime, deletes them, refcount-decrements each chunk they used, unlinks + reclaims size for any chunk whose refcount hits zero. Shared chunks stay put until the last blob referencing them is evicted. * `ManifestSummary` internal type keeps the diff-set bookkeeping cheap (one HashMap<ChunkHash, u32>, no repeated tree walks). * `claw-store cluster-gc --evict-to-gb <N>` extends the CLI: still runs the orphan sweep first, then optionally caps the store. * Config: `cluster.blob_max_gb: Option<u64>`. The auto-GC ticker runs eviction after every orphan sweep when this is set. Silent when the store is already under cap; INFO log when it evicts. +3 tests: - evict_to_size_cap_reclaims_oldest_blobs_first: 3 blobs with distinct mtimes, cap below combined size → oldest evicted, newer blobs survive - evict_keeps_shared_chunks_when_still_referenced: guards the refcount decrement path (content-addressed dedup keeps identical content as one blob → chunk survives until manifest deleted) - evict_on_empty_store_is_a_noop: sanity 257 tests pass (baseline +3). Pre-existing macOS failure unchanged. |
||
|
Omar Sobh
|
2e984b924d |
Phase 2d: chunk-level RPC (HasChunk / PutChunk / GetChunk / PutManifest)
Unlocks partial-sync replication — a peer that already has some
chunks of a blob (typical when two nodes share overlapping cargo
build caches) only receives the chunks it's missing.
## New methods
| Byte | Method | Payload | Reply |
|---|---|---|---|
| 0x09 | HasChunk | 32-byte ChunkHash | STREAM_STATUS_OK / NotFound |
| 0x0a | PutChunk | ChunkHash \|\| bytes | STREAM_STATUS_OK / error |
| 0x0b | GetChunk | ChunkHash | STREAM_STATUS_OK \|\| bytes / NotFound |
| 0x0c | PutManifest | JSON BlobManifest | JSON PutManifestReply |
`PutManifestReply { blob_id, missing: Vec<ChunkHash> }`: empty
`missing` means the manifest was written; non-empty tells the
client which chunks to upload before retrying.
Server verifies bytes hash to claimed hash on PutChunk; a
mismatch surfaces as InvalidRequest and the store is untouched.
## BlobStore additions
- `has_chunk(&ChunkHash) → bool`
- `read_chunk(&ChunkHash) → Option<Vec<u8>>` — verifies hash on read
- `put_chunk(&ChunkHash, bytes) → Result<()>` — verifies bytes-vs-hash
- `put_manifest_verified(&manifest) → Result<Vec<ChunkHash>>` —
returns the list of chunks missing on disk (empty on success)
- `chunk_path` promoted to `pub` for advanced callers
## Client helpers
- `call_has_chunk` / `call_put_chunk` / `call_get_chunk` / `call_put_manifest`
- `push_blob_missing_chunks(conn, local_store, blob_id) →
Result<(uploaded, total)>` — high-level partial-sync helper
`push_blob_missing_chunks` loads the local manifest, calls HasChunk
for each chunk, uploads only the missing ones via PutChunk, then
commits via PutManifest. On a fully-overlapping cache the uploaded
count is 0 and only the ~small manifest crosses the wire.
## Tests (17 new, all real filesystem + real QUIC — no mocks)
Blob store (6):
- has_chunk_is_false_before_put_and_true_after
- read_chunk_returns_bytes_and_none_when_missing
- put_chunk_rejects_hash_mismatch (nothing written)
- read_chunk_detects_corruption (bit-flip → mismatch error)
- put_manifest_verified_reports_missing_chunks
- put_manifest_verified_writes_when_all_chunks_present
Router dispatch (7):
- phase_2d_method_byte_encoding
- method_reports_streaming_variants — extended for 4 new methods
- has_chunk_returns_ok_for_present_and_not_found_for_missing
- put_chunk_stores_and_returns_status_ok
- put_chunk_rejects_hash_mismatch_over_wire
- get_chunk_returns_content_prefixed_with_status_ok
- get_chunk_returns_not_found_for_missing
- put_manifest_reports_missing_chunks_when_incomplete
- put_manifest_writes_when_chunks_present
- chunk_rpcs_return_not_configured_without_store
End-to-end (2):
- **end_to_end_push_blob_missing_chunks_replicates_only_needed_bytes**:
Peer A pre-seeded with chunk 0 of a 2-chunk (8 MiB) blob;
`push_blob_missing_chunks` reports `(uploaded=1, total=2)`,
only chunk 1 crosses the wire, A's store then contains the
complete blob and `get_bytes` returns byte-equal content.
- **call_get_chunk_verifies_returned_hash**: real 2-node fetch,
client hashes received bytes and compares to requested hash.
156 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 1053
- cluster/rpc/tests.rs: 940
- cluster/blob.rs: 1186
## Where this fits
With Phase 2c whole-blob streaming + Phase 2d partial-chunk sync,
the storage substrate is now genuinely bandwidth-efficient in the
distributed setting:
- First-ever push of a blob: `push_blob_missing_chunks` uploads
everything (all chunks missing).
- Second push of a similar blob (95% chunk overlap with prior
contents): only the 5% new chunks cross the wire, plus a tiny
manifest.
- Whole-blob download: BlobGetStream, bounded by network bandwidth.
## Follow-on
- Phase 3: CRDT metadata for human-readable namespaces on top of
content hashes.
- Phase 5: the killer feature. Fingerprint cargo target dir → tar
→ hash → PutBlobStream (or push_blob_missing_chunks if a similar
build already lives on the peer). Same fingerprint on the next
node → BlobGetStream. This is the whole cargo-cache design in
one line and it now sits on a substrate that handles all the
hard cases (dedup, verification, resumability, partial sync).
|
||
|
Omar Sobh
|
1fd1027da4 |
Phase 2c: streaming Blob RPC (BlobPutStream / BlobGetStream)
Removes the 16 MiB message cap for blob transfers. The bounded Blob* methods from Phase 2b still exist; the streaming variants let a peer push or pull a many-GB blob without either side holding it in memory. ## Wire format Streaming methods use a slightly different reply shape so the client can route on the first byte alone: Reply : status:u8 || payload:bytes... Where `status` is either `STREAM_STATUS_OK` (0x00, content follows) or a single-byte ErrorCode. `serve_connection` now peeks at the method tag byte via read_exact and hands streaming methods the raw send/recv streams; bounded methods still use the old read_to_end path. ## Method additions - BlobPutStream (0x07): client streams bytes → server pipes into BlobStore::put_stream → reply is 0x00 || 32-byte BlobId - BlobGetStream (0x08): client sends 32-byte BlobId → server verifies existence, writes 0x00 status, then streams chunks from disk into the send stream Method::is_streaming() introspection so callers can decide which wire variant to use. ## BlobStore additions - put_stream<R: AsyncRead + Unpin>(reader) -> BlobId Memory ceiling: one CHUNK_SIZE (4 MiB) buffer regardless of blob size. Handles short-reads correctly (loops until CHUNK_SIZE bytes are available or EOF), including the empty-reader case (produces the empty-blob BlobId, zero chunks). - stream_to<W: AsyncWrite + Unpin>(id, writer) -> bool Ok(false) on NotFound (writer untouched). Verifies each chunk hash before emitting; corruption halts mid-stream with Err. ## Client helpers - call_blob_put_stream(conn, reader) -> Result<BlobId> Uses tokio::io::copy directly onto quinn's SendStream. - call_blob_get_stream(conn, id, writer) -> Result<bool> Ok(false) on NotFound; other errors surface as Err. ## Tests (11 new, all real — no mocks) Blob store (6): - put_stream_produces_same_hash_as_put_bytes (3-chunk blob via Cursor) - put_stream_handles_empty_reader (produces empty-blob BlobId) - put_stream_handles_short_reads (custom Trickle reader that only serves 100 bytes per read call — must still assemble full chunks) - stream_to_writes_full_blob (2-chunk write to Vec<u8>) - stream_to_returns_false_when_missing (writer untouched) - stream_to_detects_chunk_corruption (bit-flip a chunk → Err with "chunk hash mismatch") RPC (5): - method_reports_streaming_variants - end_to_end_stream_put_and_get_over_real_quic — 12 MiB + 777 bytes → 4 chunks, real 2-node QUIC + mTLS + stream round-trip - stream_get_returns_false_for_missing_blob - stream_methods_return_not_configured_without_store - stream_put_deduplicates_with_prior_put_bytes — verify streaming put produces the same BlobId as a prior bounded put on identical content, and the manifest chunk count didn't fork ## Housekeeping rpc.rs was tipping over the 1300-line ceiling with the streaming handlers + helpers + tests. Tests split into `cluster/rpc/tests.rs` via `#[path = "rpc/tests.rs"] mod tests;`. Result: - rpc.rs: 748 lines - rpc/tests.rs: 694 lines - blob.rs: 1002 lines - All under ceiling. 139 tests pass. Pre-existing macOS-only failure unchanged. ## What's next - Phase 2d: chunk-level RPC (BlobPutChunk / BlobGetChunk) so a receiver can `LoadManifest` then request only the chunks it's missing — big bandwidth win on partially-overlapping caches. - Phase 3: CRDT metadata for human-readable namespaces on top of content hashes. - Phase 5: the killer feature — fingerprint the cargo target dir, BlobPutStream it, next node BlobGetStream by the same fingerprint. Now buildable directly on Phase 2c since target dirs run 100 MB to a few GB and the previous 16 MiB cap would have blocked us. |
||
|
Omar Sobh
|
eab10005fd |
Phase 2: content-addressed blob store
The storage substrate everything after Phase 1 sits on. Every blob is
identified by its BLAKE3 whole-content hash (BlobId); on disk it lives
as an ordered sequence of BLAKE3-hashed 4 MB chunks, so blobs that
share a prefix (two cargo target dirs with 95% of the same deps) share
storage at chunk granularity with no special detection logic.
New: cluster/blob.rs (802 lines).
Types:
- BlobId — 32-byte BLAKE3 output, hex-serialised (serde ↔ string)
- ChunkHash — same shape as BlobId but a distinct type so blob and
chunk lookups can't accidentally swap
- BlobStat — { total_size, chunk_count }
- BlobManifest — { blob_id, total_size, chunks: Vec<ChunkHash> },
public because Phase 2b RPC serves it directly so a receiver can
request only the chunks it's missing
- GcReport — { chunks_scanned, chunks_removed, bytes_reclaimed }
- BlobStore — root-directory-based store
Public API:
- BlobStore::open(root)
- put_bytes(&[u8]) → BlobId
- get_bytes(&BlobId) → Option<Vec<u8>> (verifies hash + size on read)
- contains(&BlobId) → bool
- stat(&BlobId) → Option<BlobStat>
- load_manifest(&BlobId) → Option<BlobManifest>
- delete_manifest(&BlobId) → bool (chunks stay; orphan by GC)
- gc_orphan_chunks() → GcReport
On-disk layout:
<root>/
blobs/<bb>/<blob_hash>.manifest.json
chunks/<cc>/<chunk_hash>
.tmp/
Two-char bucket prefixes cap fan-out at 256 entries per level — safe
on a warm-tier ZFS dataset with tens of thousands of blobs.
Every write is atomic (tmp file + rename on same filesystem).
Every chunk write is a no-op if the file already exists — same
content across two put()s stores exactly one physical copy.
Correctness:
- Reads verify each chunk against its hash + recompute the whole-blob
hash before returning; a bit-flipped chunk raises "chunk hash
mismatch" instead of silently corrupting the answer.
- delete_manifest is the only deletion primitive; chunks are only
ever removed by gc_orphan_chunks after a full manifest scan proves
they're unreferenced.
Dep: blake3 = "1".
Tests (21 new, all real filesystem, no mocks):
- BlobId/ChunkHash hex round-trip + serde JSON
- BlobId::from_hex rejects wrong-length + non-hex input
- open creates blobs/, chunks/, .tmp/
- put + get round trip: small, empty, 10 MB (3 chunks)
- put is deterministic (same bytes → same BlobId every time)
- put is idempotent (writing twice → exactly one manifest file)
- Different content → different BlobId
- shared_chunks_are_stored_only_once: two blobs sharing a 4 MB prefix
produce exactly 3 chunk files, not 4
- get_returns_none_when_missing / contains false / stat None
- delete_manifest keeps chunks (proven by counting chunk files)
- delete_manifest on missing returns false
- gc_reclaims_orphan_chunks_but_keeps_referenced: put 2 blobs, delete
one manifest, GC removes exactly the orphaned chunk, keeps
live A readable
- gc_on_empty_store_reports_zero
- corrupted_chunk_detected_on_read: rewrite a chunk with garbage →
get_bytes errors with "chunk hash mismatch"
- load_manifest round-trips the chunk list
116 tests pass. Pre-existing macOS-only failure unchanged.
File size: cluster/blob.rs = 802 lines (ceiling 1300).
Follow-on Phase 2 cuts:
- 2b: RPC methods BlobStat / BlobGet / BlobPut, wired into RpcRouter
and served over the QUIC transport built in Phase 1c-1e.
- 2c: streaming put/get (AsyncRead / AsyncWrite variants) for
many-GB build artifacts.
Phase 3 (metadata + CRDTs) can start independently — this store is the
substrate the fingerprint cache in Phase 5 layers onto.
|