Omar Sobh
eab10005fd
The storage substrate everything after Phase 1 sits on. Every blob is
identified by its BLAKE3 whole-content hash (BlobId); on disk it lives
as an ordered sequence of BLAKE3-hashed 4 MB chunks, so blobs that
share a prefix (two cargo target dirs with 95% of the same deps) share
storage at chunk granularity with no special detection logic.
New: cluster/blob.rs (802 lines).
Types:
- BlobId — 32-byte BLAKE3 output, hex-serialised (serde ↔ string)
- ChunkHash — same shape as BlobId but a distinct type so blob and
chunk lookups can't accidentally swap
- BlobStat — { total_size, chunk_count }
- BlobManifest — { blob_id, total_size, chunks: Vec<ChunkHash> },
public because Phase 2b RPC serves it directly so a receiver can
request only the chunks it's missing
- GcReport — { chunks_scanned, chunks_removed, bytes_reclaimed }
- BlobStore — root-directory-based store
Public API:
- BlobStore::open(root)
- put_bytes(&[u8]) → BlobId
- get_bytes(&BlobId) → Option<Vec<u8>> (verifies hash + size on read)
- contains(&BlobId) → bool
- stat(&BlobId) → Option<BlobStat>
- load_manifest(&BlobId) → Option<BlobManifest>
- delete_manifest(&BlobId) → bool (chunks stay; orphan by GC)
- gc_orphan_chunks() → GcReport
On-disk layout:
<root>/
blobs/<bb>/<blob_hash>.manifest.json
chunks/<cc>/<chunk_hash>
.tmp/
Two-char bucket prefixes cap fan-out at 256 entries per level — safe
on a warm-tier ZFS dataset with tens of thousands of blobs.
Every write is atomic (tmp file + rename on same filesystem).
Every chunk write is a no-op if the file already exists — same
content across two put()s stores exactly one physical copy.
Correctness:
- Reads verify each chunk against its hash + recompute the whole-blob
hash before returning; a bit-flipped chunk raises "chunk hash
mismatch" instead of silently corrupting the answer.
- delete_manifest is the only deletion primitive; chunks are only
ever removed by gc_orphan_chunks after a full manifest scan proves
they're unreferenced.
Dep: blake3 = "1".
Tests (21 new, all real filesystem, no mocks):
- BlobId/ChunkHash hex round-trip + serde JSON
- BlobId::from_hex rejects wrong-length + non-hex input
- open creates blobs/, chunks/, .tmp/
- put + get round trip: small, empty, 10 MB (3 chunks)
- put is deterministic (same bytes → same BlobId every time)
- put is idempotent (writing twice → exactly one manifest file)
- Different content → different BlobId
- shared_chunks_are_stored_only_once: two blobs sharing a 4 MB prefix
produce exactly 3 chunk files, not 4
- get_returns_none_when_missing / contains false / stat None
- delete_manifest keeps chunks (proven by counting chunk files)
- delete_manifest on missing returns false
- gc_reclaims_orphan_chunks_but_keeps_referenced: put 2 blobs, delete
one manifest, GC removes exactly the orphaned chunk, keeps
live A readable
- gc_on_empty_store_reports_zero
- corrupted_chunk_detected_on_read: rewrite a chunk with garbage →
get_bytes errors with "chunk hash mismatch"
- load_manifest round-trips the chunk list
116 tests pass. Pre-existing macOS-only failure unchanged.
File size: cluster/blob.rs = 802 lines (ceiling 1300).
Follow-on Phase 2 cuts:
- 2b: RPC methods BlobStat / BlobGet / BlobPut, wired into RpcRouter
and served over the QUIC transport built in Phase 1c-1e.
- 2c: streaming put/get (AsyncRead / AsyncWrite variants) for
many-GB build artifacts.
Phase 3 (metadata + CRDTs) can start independently — this store is the
substrate the fingerprint cache in Phase 5 layers onto.
60 lines
2.4 KiB
TOML
60 lines
2.4 KiB
TOML
[package]
|
|
name = "claw-store"
|
|
version = "0.3.0"
|
|
edition = "2021"
|
|
|
|
[[bin]]
|
|
name = "claw-store"
|
|
path = "src/main.rs"
|
|
|
|
[dependencies]
|
|
clap = { version = "4", features = ["derive"] }
|
|
serde = { version = "1", features = ["derive"] }
|
|
toml = "0.8"
|
|
tokio = { version = "1", features = ["full"] }
|
|
sysinfo = "0.30"
|
|
anyhow = "1"
|
|
tracing = "0.1"
|
|
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
|
chrono = { version = "0.4", features = ["serde"] }
|
|
axum = { version = "0.7", features = ["macros"] }
|
|
tower-http = { version = "0.5", features = ["cors", "fs"] }
|
|
tokio-stream = "0.1"
|
|
serde_json = "1"
|
|
# v0.2.0 — flock(2) wrapper for atomic+locked manifest writes
|
|
# (manifest.rs). Already a transitive dep; declaring it directly
|
|
# makes the call site obvious.
|
|
libc = "0.2"
|
|
# v0.2.0 — sibling-tempfile + rename for atomic manifest persistence.
|
|
# Was previously dev-dep only; promoted to main.
|
|
tempfile = "3"
|
|
# v0.11.1 — cluster membership via scuttlebutt gossip + phi-accrual failure
|
|
# detection. Wraps in cluster/gossip.rs. UDP transport, keyed KV state per
|
|
# node, seed_nodes bootstrap from [[cluster.peers]] config.
|
|
chitchat = "0.11"
|
|
# v0.11 — QUIC transport for peer RPC (Phase 1c). UDP-based; runs on a
|
|
# separate port from chitchat gossip. TLS 1.3 by default; we wire mTLS
|
|
# against a fleet root CA in cluster/transport.rs.
|
|
quinn = { version = "0.11", default-features = false, features = ["runtime-tokio", "rustls-ring"] }
|
|
# v0.23 — TLS 1.3 provider driving quinn's crypto. Pinned to the ring
|
|
# provider so a single CryptoProvider is installed process-wide.
|
|
rustls = { version = "0.23", default-features = false, features = ["ring"] }
|
|
# v0.13 — X.509 cert generation for the fleet CA + per-node leaf certs.
|
|
# Used both by production bootstrap (writes PEM to /etc/claw-store/tls/)
|
|
# and by tests (in-memory ephemeral CA).
|
|
rcgen = { version = "0.13", features = ["pem", "x509-parser"] }
|
|
# v2 — parse PEM files into DER for rustls consumption. Used in
|
|
# NodeIdentity::from_pem_files (Phase 1d) so persistent identity round-trips
|
|
# through the file system on daemon restart.
|
|
rustls-pemfile = "2"
|
|
# v1 — content-addressed hashing for the Phase 2 blob store. BLAKE3 is
|
|
# used both for whole-blob addressing (BlobId) and for per-chunk
|
|
# addressing (dedup). Fast enough that a full-blob rehash on read
|
|
# verification stays cheap.
|
|
blake3 = "1"
|
|
|
|
[dev-dependencies]
|
|
tempfile = "3"
|
|
tower = { version = "0.5", features = ["util"] }
|
|
http-body-util = "0.1"
|