dbc1587bcb4805e3664a6bf4981d2c7e8c9108e7
86
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar Sobh
|
dbc1587bcb |
Phase 8d: daemon binds a second QuicServer on the tailnet interface
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 11s
Live smoke on tank↔architect exposed the gap: bind_rpc_tailscale was being *advertised* via gossip so peers learned to dial it, but the daemon never actually LISTENED there. Tailnet dials hit a closed port. Fix: when both bind_rpc_lan and bind_rpc_tailscale are set (and differ), spawn a second QuicServer on the tailnet address. Shares the same fleet-CA identity + RpcRouter as the LAN listener — requests from either side hit the same handlers. If the second bind fails (e.g. tailnet interface not up), we log a warning and keep the LAN listener alive rather than aborting daemon startup. Standard graceful-degrade shape. No new tests here — a live integration test would need two network interfaces + a running tailscale, which the CI runners don't have. Coverage happens on the tank+architect deployment: `ss -lunp` on architect must show TWO clawstor UDP listeners after this change (10.0.0.13:7702 + 100.104.171.32:7702). Follow-on: cert SAN for the tailnet address. The current fleet-CA-signed leaf only has the node name as SAN, so rustls verification on the client side still checks against --peer <name> which passes because CN == node name. But a belt-and-suspenders leaf using fleet-ca-tailscale-sign (Phase 8a) would be more correct. |
||
|
Omar Sobh
|
f2c056464c |
Phase 8c hotfix: skip probe deadline when no fallback exists
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Failing after 2s
Live smoke on tank↔architect (both LAN) failed with 200ms probe: LAN handshake takes longer than that in the wild (TLS 1.3 with full cert chain + rustls startup on fresh endpoint). The old single-addr .connect() had no deadline, so pre-8c callers never noticed. Fix: when `tailscale` is `None`, treat LAN as unlimited — the probe deadline only matters as a fall-through trigger, and there's nothing to fall through to. Callers with a real fallback addr still get the fast-path routing behavior unchanged. +1 test (connect_lan_first_lan_only_ignores_probe_deadline) using a 1-nanosecond probe budget that a real handshake could never meet — must succeed anyway because no fallback exists. 376 tests pass (+1). |
||
|
Omar Sobh
|
38652c5886 |
Phase 8c: cluster-peer-status + cluster-repair support --tailscale-addr
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 11s
Wires the operator CLIs to the Phase 8b connect_lan_first primitive.
Roaming ops (laptop on LTE, coffee-shop wifi) can now pass a
tailnet address alongside the usual --rpc-addr and get the
LAN-first-with-fallback behavior automatically.
New flags on both cluster-peer-status and cluster-repair:
* --tailscale-addr <addr> — optional tailnet RPC socket. When
set, --rpc-addr is tried first with
a short deadline, then this on
failure/timeout.
* --lan-probe-ms <ms> — LAN probe deadline. Default 200
matches the arch doc.
Zero flag → byte-identical to pre-8c behavior (single-addr dial).
Both flags → chosen route printed in the output header so
operators can see whether LAN or tailnet won.
No new tests: this is thin glue over connect_lan_first, which
already has its own unit coverage. Smoke test live on tank
against architect (LAN), and against fake unroutable + real
tailnet exercises both branches.
|
||
|
Omar Sobh
|
ef60a7984e |
Phase 8b: LAN-first probe with tailnet fallback
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Failing after 2s
Second Phase 8 slice. Prior transport.connect() took a single
address; the LAN-first-then-Tailscale routing the arch doc calls
out was implicit ("pick lan_addr OR tailscale_addr from gossip
state") and never actually raced or fell through.
New: QuicClient::connect_lan_first(name, lan, tailscale, lan_probe)
* Try LAN first with `lan_probe` deadline (fleet default ~200ms).
* If LAN handshake fails OR the deadline fires → fall back to
the Tailscale address.
* Both slots None → error immediately (no hang).
Returns (connection, ConnectRoute) so callers + telemetry see
which side won. New enum ConnectRoute::{Lan(addr), Tailscale(addr)}.
+3 tests exercising the three shapes:
- lan-first when LAN reachable (never dials fake tailscale addr)
- fallback when LAN black-holes (240.0.0.1 SYN gets no response;
probe deadline fires, tailscale server wins)
- errors cleanly when both addrs absent
375 tests pass (+3). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
Follow-ons for Phase 8 completion:
- Wire the peer-connect call sites (RPC forwarding, PeerStatus,
build-cache) through connect_lan_first with per-peer
lan/tailscale addrs from gossip state.
- Document the roaming-client config template.
|
||
|
Omar Sobh
|
98036f2597 |
Phase 8a: fleet-ca-tailscale-sign — Tailscale-aware leaf certs
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 25s
First slice of Phase 8 (roaming client identity). Adds a helper
that mints a leaf cert whose SANs include this node's Tailscale
identity — MagicDNS name (laptop.taila4f562.ts.net) + all tailnet
IPs — alongside the primary node name.
Closes the "how does a laptop join the fleet without hand-editing
SANs" gap: on a machine that's on Tailscale, one command produces
a leaf that peers can dial by MagicDNS from anywhere on the
tailnet.
New CLI:
claw-store fleet-ca-tailscale-sign \
--ca-dir /etc/claw-store/ca \
[--node <name>] # defaults to Tailscale HostName
--out-dir /etc/claw-store/tls
Reads identity by shelling to `tailscale status --json` (already
present on any node that's on the tailnet; no extra dep). If
tailscale isn't running or installed, exits cleanly with a real
error.
New module cluster::tailscale:
* TailscaleSelf { magicdns_name, tailscale_ips, short_hostname }
* read_self() — runs the CLI, returns identity
* parse_status() — pure decoder, unit-tested
* suggested_sans() — MagicDNS + IPs ordered for the CA sign flow
FleetCa additions:
* sign_leaf_to_pem_with_sans(node_name, extra_sans, out_dir) —
Sans-extended variant of sign_leaf_to_pem. Empty entries dropped.
Existing sign_leaf_to_pem now delegates with empty extras (100%
backward compat).
* mint_leaf_with_sans — internal shared helper.
+5 tests: parse full identity, parse missing MagicDNS, error on
no Self record, suggested_sans ordering, suggested_sans skips
missing MagicDNS.
372 tests pass (+5). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
Next Phase 8 slices: (a) tailnet-preferring peer probe with a
config-selectable auth mode, (b) documented "roaming client"
config template.
|
||
|
Omar Sobh
|
3af6390316 |
Phase 7f: claw-cargo auto-records fingerprint → (repo, git_ref)
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Failing after 3s
Closes the ref-tracking loop. claw-cargo build now records the producing (repo, git_ref) alongside every cache-put fingerprint, so cluster-ref-sweep can identify stale entries later without operator bookkeeping. New BuildArgs flags: * --repo <owner/name> (env CLAWSTOR_REPO, or GITEA_REPOSITORY / GITHUB_REPOSITORY when the CI runner sets them via that name in workflow env) * --git-ref <branch-or-tag> (env CLAWSTOR_GIT_REF) * --ref-tracking-dir <path> (env CLAWSTOR_DATA_DIR, typically /var/lib/claw-store/data — same root as cluster.blob_store_root) Semantics: * All three unset → silently skipped. Existing cache flows are unchanged. * dir doesn't exist or open() fails → logs warn, cache still valid. * record() call fails → logs warn, cache still valid. The tracking store is co-located with the daemon's data dir so cluster-ref-sweep on that host sees the annotations. Runners mount /var/lib/claw-store/data via bind-mount today. No new tests here — the primitive (RefTracking::record) already has full coverage. This is thin glue. |
||
|
Omar Sobh
|
7bc5ba987c |
Phase 7f follow-on: Gitea live-refs adapter + cluster-ref-sweep CLI
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Failing after 6s
Wires the Phase 7f ref-tracking primitives to a real Gitea. New
CLI `claw-store cluster-ref-sweep --gitea-url <> [--gitea-token]
[--retention-days N]` queries every distinct repo we've recorded
against, fetches its live branches + tags, computes the stale set
via RefTracking::stale_at, and prints the stale fingerprints
grouped by repo.
Dry-run only in this cut. Deletion is separate — the operator
decides whether to call `forget` per fp, and whether to also
prune the corresponding blob/tag. Blob eviction happens via
cluster-gc as usual (dead refs no longer contribute to any pin).
New module cluster:
|
||
|
Omar Sobh
|
294697d2f5 |
Phase 7f: ref-tracking primitives for retention-eligibility
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Failing after 3s
Records which (repo, git-ref) combinations produced each cache
fingerprint. Later slices will wire this to a nightly Gitea sweep
that queries /api/v1/repos/.../branches and /tags, then evicts
fingerprints whose recorded refs are all gone AND whose
last_seen_unix is older than the retention window.
Per-fingerprint (not per-blob) because:
* Fingerprints are the cache keys claw-cargo uses. Tracking at the
fp layer keeps this aligned with the claw-cargo boundary.
* Blobs are content-addressed and may be shared. Ref-tracking is
about "why we kept this cache" — a per-fp concern.
New module cluster::ref_tracking:
* RefEntry { fingerprint, repo, refs, first_seen_unix, last_seen_unix }
* RefTracking::record(fp, repo, git_ref, now) — creates or updates
* RefTracking::get(fp) / list_all() / forget(fp)
* RefTracking::stale_at(now, live_refs_by_repo, retention_secs) →
Vec<fingerprint>, the deletion-eligibility list
On-disk: <root>/ref-tracking/<hh>/<fp_hex>.json. JSON so operators
can inspect with jq. One record per cached fp; even 100k fps is
under 50 MB.
Semantics baked in:
* record() APPENDS refs, never removes — sweep decides staleness
* record() rejects repo change for a fp (collision or bug detector)
* refs stable-sorted in-file so cross-node diff is easy
* stale_at treats "repo not in live_refs map" as "all refs dead"
→ deleted repos don't leak caches
* retention_secs is a floor: dead-but-fresh caches survive
+10 tests: create, append-and-refresh, dedup, repo-change reject,
stale-at happy path, stale-at missing-repo, stale-at retention,
forget truth values, list sorted, validate rejects.
365 tests pass (+10). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
No CLI or wire integration in this PR — pure library, testable
in isolation. Follow-ons: (a) claw-cargo auto-record on cache put,
(b) Gitea polling adapter, (c) sweep wired into cluster-gc.
|
||
|
Omar Sobh
|
eebc62d87b |
Phase 7d follow-on: snapshots pin blobs against LRU eviction
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
Closes the retention loop between snapshots and pin-aware LRU eviction. A snapshot is not just a "list of blobs at time T" any more — it's a *retention pin* on every blob it captures. Operators can guarantee a build stays on disk for N days by snapshotting it and pruning the snapshot when the window is up. Additions: * SnapshotStore::pinned_blob_ids() → union of blob_ids across all live snapshots. Cheap: one JSON read per snapshot. * cmd_cluster_gc extends the tag-pin set with snapshot pins before handing it to evict_to_size_cap_with_pins. Output line now reads "pinned blobs: N (M from snapshots)". * ClusterServices auto-GC ticker does the same on every tick; log fields include snapshot_pins so ops see the retention set size at a glance. +2 tests: - pinned_blob_ids_unions_all_snapshots (overlap dedupe) - pinned_blob_ids_empty_when_no_snapshots 355 tests pass (+2). Pre-existing macOS hot::tests::test_project_target_size_bytes failure unchanged. |
||
|
Omar Sobh
|
e564b0ce89 |
Phase 7d: snapshot primitives + CLI
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
A snapshot is a named, immutable point-in-time record of every blob
live in the store. It's NOT a data copy — blobs are content-addressed
and already live under blobs/. A snapshot is a JSON reference set at
<root>/snapshots/<name>.json.
Why:
* Rollback anchor before risky migrations.
* Retention pin: combined with the Phase 4a pin-aware LRU eviction,
operators can guarantee "these blobs stay on disk N days".
* Audit: "which blobs existed at release time?"
New module cluster::snapshot:
* SnapshotStore::create(name, blob_store, created_at)
* SnapshotStore::get(name) / list() / delete(name)
* SnapshotManifest { name, created_at_unix, blob_ids }
* SnapshotSummary for cheap list rendering (no blob-list slurp).
BlobStore gains list_blob_ids() — walks blobs/**/*.manifest.json
and returns the blob id set. Manifests only, no chunk reads.
New CLI commands:
* claw-store cluster-snapshot-create --name <>
* claw-store cluster-snapshot-list
* claw-store cluster-snapshot-show --name <>
* claw-store cluster-snapshot-delete --name <>
Semantics:
* Snapshots are immutable: create with existing name errors, does
not clobber. Delete-then-create if you really want to overwrite.
* delete() removes only the reference file. Never touches blob
data — protects against operators nuking live data by pruning
snapshots.
* list() sorts by created_at_unix ascending — oldest first so
triage picks pruning candidates quickly.
* blob_ids are sorted at write time so the same content on two
nodes yields byte-identical snapshot files.
* Names validated: no /, \\, NUL, control chars; max 512 bytes.
+8 tests covering create+capture, immutability, get-missing,
list-ordering, delete truth-values, delete-doesn't-touch-blobs,
name-validation, and sorted round-trip.
353 tests pass (+8). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
|
||
|
Omar Sobh
|
da198c0903 |
Phase 7c: cluster-repair CLI wires repair to a peer
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Failing after 3s
New command: `claw-store cluster-repair --peer <name> --rpc-addr <host:port> --tls-dir <dir> [--dry-run]`. Flow: 1. Local scrub identifies bad chunks (missing + corrupt). 2. Deduplicate to unique chunk-hashes (scrub emits per-reference, fetcher work is per-chunk). 3. Connect to peer over QUIC + mTLS. 4. For each unique chunk: HasChunk probe → GetChunk on hit → put locally (re-hashed by put_chunk, so a lying peer can't corrupt us further). 5. Report attempted/repaired/unrecoverable/errors. --dry-run stops after the dedup step: prints the plan without touching the peer or disk. Behavior details: * Zero bad chunks → clean exit with no peer contact. * Any unrecoverable or per-chunk error → non-zero exit so cron/CI notice. Message names counts. * HasChunk-first means a peer that lacks the chunk is one cheap round-trip, not a full GetChunk attempt. Companion piece for the Phase 7b repair library (already merged). No new tests here — logic is thin glue over `repair_chunks` + `call_has_chunk`/`call_get_chunk`, all of which have their own unit + integration coverage. Behavior gets its real workout in live smoke on tank+architect. |
||
|
Omar Sobh
|
cf0a07099d |
Phase 7b: chunk-level repair library
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Failing after 3s
New primitive: BlobStore::repair_chunks(chunks, fetch) → RepairReport. Consumer flow: cluster-scrub returns a list of (blob, chunk) bad pairs. cluster-repair (next slice) will hand the chunk hashes here with a fetcher that walks peers via HasChunk/GetChunk. This PR is the library-only half — no peer wiring — so it's testable in isolation and reusable by callers who already have a chunk source. Fetcher contract: * Ok(Some(bytes)) → put locally, count repaired * Ok(None) → nobody has it, record as unrecoverable * Err(e) → per-chunk error, batch continues Guardrails: * Bytes are re-hashed by put_chunk before writing. A peer that returns wrong bytes for a hash cannot corrupt us further. * Duplicate chunk hashes in the input dedupe → fetcher called exactly once per unique chunk. Matters because scrub reports shared chunks once per owning manifest. * Errors on one chunk never abort the batch — the remaining chunks still get their shot. * Repair overwrites a corrupt file: unlink-then-put_chunk, since put_chunk itself is write-if-absent. NotFound on unlink is fine (missing-chunk case). +4 tests: - repair_writes_fetched_bytes_and_marks_repaired (happy: corrupt → repair → post-scrub clean) - repair_records_unrecoverable_when_fetcher_returns_none - repair_records_error_and_continues_batch (batch survives one chunk's error) - repair_dedups_duplicate_chunks_in_input (fetcher called exactly once for 3 identical hashes) 345 tests pass (+4). Pre-existing macOS hot::tests::test_project_target_size_bytes failure unchanged. |
||
|
Omar Sobh
|
701861787f |
Phase 7a: read-only fsck for the blob store
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 12s
New primitive: BlobStore::scrub_all() → ScrubReport. Walks every .manifest.json under blobs/, for each referenced chunk reads the file from disk and recomputes BLAKE3. Verdict per chunk: * file absent → missing * hash mismatch → corrupt * match → ok Design points: * Read-only. Never touches disk state. Safe against a live daemon — worst case a chunk lands mid-scrub and is skipped this pass. * Per-reference counting: a bad chunk that N manifests depend on shows up as N corrupt entries so operators see the full blast radius. But each unique chunk is hashed exactly once via an in-memory verdict cache. * Report holds explicit (blob_id, chunk_hash) pairs for every bad chunk so the fix path (repair in Phase 7b) has enough info to act. CLI: `claw-store cluster-scrub [--verbose]`. Non-zero exit when integrity issues exist so cron / CI notice. +4 tests: - scrub_reports_all_ok_when_store_is_healthy - scrub_detects_corrupt_chunk (owner blob id preserved) - scrub_detects_missing_chunk (owner blob id preserved) - scrub_dedups_shared_chunk_hashing_once (shared chunk, 2 owners reported, single disk read) 341 tests pass (+4). Pre-existing macOS hot::tests::test_project_target_size_bytes failure unchanged. |
||
|
Omar SobhandClaude Opus 4.7
|
39f9a9652a |
Phase 4e: cmd_pin --offline + drain + wal-status CLI
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
Wires cmd_pin through the WalQueue built in the preceding four PRs (#48 → #52). First real caller of the client-mode WAL stack. New surfaces: claw-cargo pin --offline --blob <BlobId> [--ttl <duration>] * no peer connection is opened * enqueues the same three mutations cmd_pin would emit online: primary tag (PutTagVersioned), .fingerprint companion (PutTagVersioned), and — if --ttl — the two SetTagExpiry sidecars * requires --blob because offline mode can't do the GetRefVersioned lookup that resolves fingerprint → BlobId * prints the assigned WAL seqs + "next step: drain" claw-cargo drain --peer ... * opens a peer, drains the queue, truncates up to the last applied seq * partial-failure safe: whatever applied is truncated; anything after a hard error stays on disk for retry * exits non-zero when drain stopped mid-stream claw-cargo wal-status * read-only, no network * pending count, oldest/newest seq, storage path, decoded entries (or UNDECODABLE marker on frame errors) WAL location follows the XDG state-home pattern already used by manifest.rs: 1. $XDG_STATE_HOME/claw-cargo/wal/ 2. $HOME/.local/state/claw-cargo/wal/ 3. ./.claw-cargo-wal/ (worst-case container fallback) Tests (2): default_wal_path_honours_xdg_state_home (mirroring manifest.rs's env-var pattern) + parse_blob_id_rejects_bad_ hex_and_wrong_len. claw_cargo.rs grew from 1668 to ~1870 lines. Still under the 1300-per-*module* interpretation but this bin file has been above 1300 since Phase 5. Split-out is Phase 6 territory. Co-Authored-By: Claude Opus 4.7 <[email protected]> |
||
|
Omar SobhandClaude Opus 4.7
|
9ece4a6e13 |
Phase 4d: WalQueue caller-facing wrapper
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
Collapses the enqueue + drain + truncate dance around
WriteAheadLog + wal_mutation + wal_replay into one API so
downstream callers (Phase 4e: cmd_pin & friends) don't have
to orchestrate three modules themselves.
Two-call flow:
let mut q = WalQueue::open(state_dir.join("wal")).await?;
q.enqueue(&WalMutation::PutTagVersioned { .. }).await?;
// ...later, on reconnect:
let report = q.drain(&conn).await?;
drain() advances the watermark to the last successfully-
applied (or Superseded) seq whether or not the drive stopped
on a hard error mid-stream. Nothing is truncated past the
failure point, so the failing record and everything after
it are retried on the next drain.
Introspection surface (`pending_count` / `oldest_pending_seq`
/ `newest_pending_seq` / `snapshot` / `is_empty`) is what a
metrics endpoint or CLI status view wants. `wal()` escape
hatch exposes the backing WAL for advanced callers.
Tests (6, all green — 4 unit + 2 end-to-end over QUIC):
* empty queue reports empty bounds
* enqueue updates bounds correctly
* snapshot decodes in seq order and preserves kind info
* drain clears the queue and applies to peer (verifies via
call_get_ref + call_get_tag_versioned)
* drain over a pre-seeded dominant version returns
Superseded and still drains the queue
* enqueue survives reopen — bounds recover through
WriteAheadLog::open scan
346 lines, well under the 1300 ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.7
|
e929a6f32f |
Phase 4d: WAL replay engine
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
Given a peer connection + a decoded WalMutation, re-issue the
correct RPC. Closes the loop from "durably logged at client"
to "actually applied at peer" on reconnect.
Outcome classification is deliberate:
* Applied — peer accepted the mutation.
* Superseded — peer already had a dominant version, or the
delete target was absent. NOT a failure; the
mutation's intent matches current peer state.
* Err(_) — genuine RPC failure; caller retries later.
Both Applied and Superseded advance the watermark past the
record — the WAL can safely truncate.
Public surface:
ReplayOutcome { Applied | Superseded }
replay_one(&conn, &mutation) -> Result<ReplayOutcome>
drive_replay(&conn, &wal, start_seq) -> Result<DriveReport>
DriveReport { last_applied, applied, superseded,
skipped, stopped_at: Option<(seq, msg)> }
drive_replay stops on the first hard error and returns
last_applied so the caller can `wal.truncate_up_to(...)`
before closing. Undecodable/unknown-kind records mid-stream
are skipped (with warn!) rather than aborting — otherwise
one bad record would jam an otherwise-good tail forever.
Tests (4, all green, end-to-end over QUIC):
* every variant round-trips; peer state verified via
call_get_ref / call_get_tag_versioned / call_get_tag_expiry
* versioned-reject counts as Superseded, not Err
* DeleteTag on a missing key is Superseded
* undecodable record between two real mutations is skipped;
both good records still apply; last_applied advances past
the skip
434 lines, well under the 1300 ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.7
|
bd0b4972b9 |
Phase 4d: typed WalMutation frames
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
Adds the encoding layer that turns the raw WAL (opaque bytes)
into a typed queue of client-mode mutations, ready for Phase 4e
to wire actual RPCs through.
Frame (self-describing, forward-compatible):
version : u8 = 0x01
kind : u8 = one of the Kind discriminants
body : [u8] kind-specific
Body encodings mirror the existing on-wire shapes so a future
replay path can splice a WAL record straight into an RPC payload.
Variants (Kinds 0x01–0x06):
PutRef, PutRefVersioned, PutTag, PutTagVersioned,
DeleteTag, SetTagExpiry
Blob-put mutations are deliberately NOT modeled — blob data is
too large to keep in the WAL. The roaming-client design stages
blobs on local disk and records a reference to them once the
local BlobPutStream completes.
Public helpers:
append_mutation(&mut wal, &m) -> Result<seq>
replay_mutations(&wal, start_seq)
-> Vec<(seq, Result<WalMutation, WalMutationError>)>
Unknown-kind records surface as `Err(UnknownKind(byte))`, not
a panic — forward-compat when a newer writer wrote a record
this reader doesn't understand. Malformed records also surface
as Err so the caller can decide (log-and-skip vs abort replay).
Tests (9, all green): kind-byte stability, roundtrip every
variant, rejects empty/short/bad-version/unknown-kind,
malformed bodies (wrong length, over-declared key_len, trailing
garbage on DeleteTag), non-UTF-8 keys, append+replay through a
real on-disk WAL, and replay-survives-unknown-kind mid-stream.
No new deps — hand-rolled error type in-tree (no thiserror).
515 lines, well under the 1300 ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.7
|
0cf00e5954 |
Phase 4d: WAL segment rotation
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 17s
Turns the single-file Phase 4c WAL into a segmented log so it
can grow past a single file safely. This unblocks every
downstream Phase 4d/4e integration — reconnect + push loop
can't rely on an unbounded single file.
Layout change:
<root>/segment-<20-digit-first-seq>.bin
20-digit zero-padded first-seq means lex sort == numeric sort,
so `read_dir + sort_by_key` recovers the natural order.
Rotation policy:
* `max_segment_bytes` default 8 MiB, overridable via
`open_with_options`.
* `append` rolls to a fresh segment BEFORE writing when the
current tail is non-empty AND at/above the cap. A single
oversize record always lands in one segment — we never split
a record.
Truncation across segments:
* whole segments with `last_seq <= watermark` are `unlink`'d
* the boundary segment (if any) is rewritten in place via
`tempfile-in-parent + rename` + parent-dir fsync
* full truncation resets head/tail to 0 and the next append
creates a fresh segment
Legacy compat: on open, if a pre-4d `log.bin` is present and
no `segment-*.bin` files exist, it is scanned for its first
seq and renamed to the correct segment name. Refuses to
silently overwrite on filename collision.
Tests (18, all green): rotation-happens-at-cap, reopen-
enumerates-all-segments, truncate-drops-whole-segments,
truncate-partial-rewrites-boundary, oversize-record-still-
fits-one-segment, legacy-log.bin-migration, plus the full
Phase 4c suite (fresh open, append, iter partial ranges,
reopen recovers tail, torn-write truncation, corruption is
hard error, full truncation appendable, below-head no-op,
large payload, empty payload, append-after-reopen).
942-line file, comfortably under the 1300-line ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.7
|
dd1a37fe7e |
Phase 4c: Write-Ahead Log primitives
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 16s
Foundation for the roaming/offline-client story described in
Architecture v2 ("Roaming client (full R/W, offline queue)"):
mutating ops append to a durable local log before hitting the
network, and are replayed at reconnect. This PR ships the
primitive; RPC/reconnect wiring lands in Phase 4d.
Segment format (single-file for now — rotation is 4d):
seq : u64 LE (8 bytes)
len : u32 LE (4 bytes) payload length
csum : [u8; 8] (8 bytes) first 8 bytes of
BLAKE3(seq || len || payload)
bytes : [u8; len]
On open, the log is scanned linearly. A short read or truncated
tail is treated as "clean crash boundary" — the file is
size-truncated to the last fully-fsynced record, no error.
A checksum mismatch on a full-length record is fatal (real
corruption, don't silently swallow data).
Public API:
WriteAheadLog::open(root) -> Self
wal.append(&[u8]) -> Result<u64> // durable, fsynced
wal.iter_from(start_seq) -> Vec<WalRecord>
wal.truncate_up_to(watermark) -> () // atomic rewrite via
// tempfile-in-parent + rename
wal.head_seq() / wal.tail_seq() / wal.is_empty()
Tests (12, all green): fresh open, monotonic seq, replay full &
partial ranges, reopen-recovers-tail, torn-write truncation on
open, corruption is hard error, prefix truncation, full
truncation leaves appendable, below-head no-op, 1 MiB payload
roundtrip, empty payload roundtrip, append-after-reopen.
No new deps — BLAKE3 (already a dep) supplies the checksum.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.7
|
4630925040 |
Phase 4b follow-on: pin --ttl RPC + CLI
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
Closes Phase 4b by exposing the TTL sidecar written by Phase 4b primitives on the wire and via `claw-cargo pin`. Wire additions: * Method::SetTagExpiry (0x1a) — payload `key_len:u16 || key || expires_at:u64 (LE)`. Reply single-byte OK. `expires_at == 0` clears the sidecar. * Method::GetTagExpiry (0x1b) — payload raw key bytes. Reply 8 bytes (u64 LE) on hit; NotFound when no sidecar is present. Both accept writes even when the stamped tag itself is absent, matching `TagStore::set_stamped_expiry` semantics — the sidecar takes effect the moment the tag lands. CLI: * `claw-cargo pin --ttl <duration>` — humantime-style duration (`30d`, `1h30m`, `2w`, ...). Applied to both the primary tag and its `.fingerprint` companion so eviction treats them as one lifetime. `--ttl 0` / `clear` / `none` clears an existing sidecar without touching the value. Tests: encode/decode roundtrip + malformed-input rejection for `encode_expiry_record`, method-byte stability, NotConfigured without a tag store, end-to-end set/get/overwrite/clear over QUIC, and a real-pin flow that publishes a stamped tag then attaches TTL. Duration parser is unit-tested for single/compound forms, case-insensitive units, bad input, and clock alignment. No new deps — the humantime-style parser is 60 lines in-tree. Co-Authored-By: Claude Opus 4.7 <[email protected]> |
||
|
Omar Sobh
|
1418d35487 |
Phase 4b: TagStore expiry primitives (pin TTL groundwork)
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 16s
Adds the on-disk mechanism for time-scoped pins. No RPC or CLI yet — a follow-on will expose \`pin --ttl <duration>\`. This PR is purely library + eviction wiring. Layout addition: alongside each stamped tag at \`tags-v2/<hh>/<hash>.svtag\`, an optional sidecar \`tags-v2/<hh>/<hash>.svtag.exp\` holds an 8-byte LE unix \`expires_at\`. Absence of the sidecar = never expires (current behavior). New TagStore methods: * set_stamped_expiry(key, expires_at_unix) — writes sidecar; passing 0 removes it. Idempotent. * get_stamped_expiry(key) — reads sidecar; None when absent. * pinned_blob_values_at(now_unix) — same union as pinned_blob_values, but skips stamped tags whose sidecar shows expires_at ≤ now. Legacy tags/ entries never expire. * prune_expired_stamped_at(now_unix) — deletes stamped tags AND their sidecars where expires_at ≤ now. Returns count. * pinned_blob_values() — now a shim that calls _at(u64::MAX) for 100% backward compat. Wired the two existing gc call sites: * ClusterServices auto-GC ticker prunes-then-collects at SystemTime::now(). One pass per tick. * \`claw-store cluster-gc --evict-to-gb N\` CLI same pattern. Report now includes \"expired pins pruned: N\". +1 test (expiry_gates_pin_set_and_prune_removes_expired): covers live/expired/no-ttl mix, sidecar round-trip, prune removes only expired, expires_at=0 clears sidecar, dropped tag stops filtering. 286 tests pass (+3 from 283). Pre-existing macOS hot::tests::test_project_target_size_bytes failure unchanged. |
||
|
Omar Sobh
|
5c55dd7044 |
Phase 4a hotfix: cmd_pin resolves stamped refs
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 23s
pin lookup was legacy call_get_ref only, missing refs written via call_put_ref_versioned (all Phase 3b+ builds). Try versioned first, fall back to legacy — same pattern as claw-cargo build path. |
||
|
Omar Sobh
|
5be11a11b0 |
Phase 4a: pin-aware LRU eviction
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
A `claw-cargo pin` used to be silently vulnerable to the size-cap
eviction ticker — the tag existed but the underlying blob could get
LRU'd out, leaving a dangling reference. Now tags act as
retention markers: any blob referenced by any tag (stamped or
legacy) is protected from `evict_to_size_cap`.
* `BlobStore::evict_to_size_cap_with_pins(max_bytes, pinned_set)` —
same LRU-by-mtime pass, but pinned blob IDs skip the eviction
loop. Existing `evict_to_size_cap` is now a thin wrapper with an
empty pin set (100% backward compat).
* `TagStore::pinned_blob_values()` — unions every 32-byte value
referenced by any tag across `tags/` (legacy) and `tags-v2/`
(Phase 3c stamped). Dedupes naturally.
* Auto-GC ticker in `ClusterServices` now collects the pin set on
every eviction pass and passes it in. Log fields include
`pinned_blobs = N` so operators can see the retention set size.
* `claw-store cluster-gc --evict-to-gb N` CLI opens the tag store
the same way, prints `pinned blobs: N` in the report.
+3 tests:
- evict_with_pins_protects_pinned_blobs_from_eviction — 3 blobs
ordered oldest→newest, pin the oldest; without pins LRU would
evict it; with pins the next-oldest goes instead. Guards the
main semantic.
- evict_with_pins_stops_when_pinned_footprint_dominates —
everything pinned + cap = 0 → no-op. Guards the "operator asked
for the impossible" case.
- pinned_blob_values_unions_both_stores — legacy tag with value V1,
stamped tag with value V2, second stamped tag also referencing
V1 → set contains {V1, V2}. Dedupe check.
283 tests pass (baseline +3). Pre-existing macOS failure unchanged.
|
||
|
Omar Sobh
|
2c3cd2ab38 |
Phase 3c + 3e: stamped tags + namespaced ref keys — closes Phase 3
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 20s
Ships the last two pieces from the arch doc's Phase 3 scope for the
cargo-cache use case:
## 3c: Stamped tags (CRDT-merge on PutTag)
Mirror of Phase 3a/3b for TagStore. Two concurrent `claw-cargo pin`
calls on the same tag now race deterministically instead of silently
clobbering.
* `StampedTagValue` — same 48-byte (value, clock, node) tuple as
StampedRef.
* `TagStore::put_stamped(key, StampedTagValue) -> TagPutOutcome`
and `TagStore::get_stamped(key)` — data lives under `tags-v2/`
(separate from `tags/` for cutover safety).
* New wire methods `PutTagVersioned = 0x18` +
`GetTagVersioned = 0x19`.
* `call_put_tag_versioned` / `call_get_tag_versioned` client
helpers.
* `claw-cargo pin` now writes stamped tags. Concurrent pin gets
AlreadyExists and moves on (blob content is content-addressed so
both winners agree on the payload).
## 3e: Namespaced ref keys
Opt-in `--namespace <slug>` on peer-facing subcommands. When set,
the ref key becomes `blake3("clawstor.ns.v1" || namespace || fp)`
so two runners on different namespaces (`clawverse/main` vs
`clawverse/pr-42`) don't collide on the same fingerprint. Empty
namespace = pre-3e behavior, so this is 100% backward compat.
* `refs::namespaced_ref_key(namespace, fingerprint) -> RefKey`
primitive.
* `PeerArgs::namespace: Option<String>` CLI flag flows through to
`cmd_status`, `cmd_prefetch`, `cmd_build`.
* `peer_lookup` now takes a `RefKey` directly (was `&Fingerprint`)
so the namespace resolution stays in the caller — the daemon
never sees "namespace" as a concept.
## 3d: Deferred
Full vector clocks per namespace are noted in the arch doc as a
Phase-3 goal; scalar wall-clock (clock + node stamp) is sufficient
for the cargo-cache use case (single-key LWW merge). NTP-synced
runners see monotonic ordering; skewed runners lose an ordering
but the CRDT semantics still guarantee no data corruption. Full VC
is deferred to a future phase.
+9 tests, 280 total (baseline +8: 7 unit + 1 e2e over real QUIC).
|
||
|
Omar Sobh
|
ac51e3e9b5 |
Phase 3b: thread stamped refs through claw-cargo + forwarding
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 25s
Runner + prewarm use PutRefVersioned/GetRefVersioned; daemon GetRefVersioned forwards on miss + pulls blob transparently. New GetRefVersionedLocal (0x17) prevents recursion. Backward-compat: existing GetRef/PutRef path unchanged; two on-disk namespaces coexist (refs/ and refs-v2/). +1 test, 272 total (unchanged from 3a because we reused existing scaffolding). |
||
|
Omar Sobh
|
af5350ac17 |
Phase 3a: Lamport-stamped refs with CRDT-merge on PutRef
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 16s
Concurrent PutRef safety via (clock, node) total order. New wire methods PutRefVersioned (0x15) + GetRefVersioned (0x16). Existing PutRef/GetRef unchanged for backward compat. Data in refs-v2/ namespace so the two coexist during cutover. +8 tests, 272 total (baseline +8). |
||
|
Omar Sobh
|
58c5bc341b |
GetRef: transparent ref-forwarding on local miss
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
Cross-runner cache silos (tank + architect measured on 2026-07-13): same fingerprint, same rustc, but each runner's daemon only knows about the refs its own runner uploaded. Every runner that lands on a peer that isn't tank re-uploads a duplicate blob. Fix: on `GetRef` miss the daemon fans out to alive gossip peers via a strict-local `GetRefLocal` variant, and the FIRST peer that has the ref triggers a transparent pull — chunks + manifest into the local blob store, then `PutRef` locally — before returning the value to the caller. Subsequent lookups are pure-local hits. * `Method::GetRefLocal = 0x14` — new wire method, identical shape to GetRef but the peer MUST NOT recurse. Loop prevention: our forwarding only calls `GetRefLocal` on peers, so chain depth is always 1. * `RpcRouter::with_outbound_client(Arc<QuicClient>)` — dependency injection point for the forwarding dial path. `None` disables forwarding entirely (GetRef becomes GetRefLocal-equivalent). * `RpcRouter::forward_get_ref(key)` — concurrent peer probes via `JoinSet`, 3s timeout per dial, first successful pull wins, remaining tasks aborted. * `pull_blob_locally` — walks manifest, fetches only chunks the local store lacks (`has_chunk`), commits via `put_manifest_verified`. Bounded memory: one 4 MiB chunk at a time. * `ClusterServices::start` loads NodeIdentity twice — server takes ownership; outbound client gets its own copy for TLS presentation on peer dials. Wires the outbound client into the router when TLS material is available. * `call_get_ref_local(conn, key)` client helper (used by daemon forwarding + available to any RPC consumer that wants the no-recursion semantics). +3 tests in `rpc/tests_forwarding.rs`: - Local hit works without forwarding; local miss with no peers returns None. Guards the base cases. - GetRefLocal never forwards even when outbound is configured (no peers reachable → miss returns None immediately, no attempted fan-out). - Method byte 0x14 encoding is stable across releases. Full end-to-end forwarding is exercised in the pilot deploy: two daemons on the fleet-CA, tank populates a ref, architect's runner GetRef → tank forwards → architect pulls → HIT locally next time. 264 tests pass (baseline +3). Pre-existing macOS failure unchanged. |
||
|
Omar Sobh
|
bfcc11de82 |
runner follow-ups: XDG config path + composite action + docs
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
Three fixes surfaced by the 2026-07-13 Gitea Actions wire-up.
## XDG config path
Before: `client_config::user_config_path` only looked at
`~/.claw-cargo/config.toml`. My runner-integration doc initially
told operators to install at `~/.config/claw-cargo/config.toml`
(XDG-style). Config wasn't loaded.
Now: three-way lookup, first hit wins.
1. `$XDG_CONFIG_HOME/claw-cargo/config.toml`
2. `$HOME/.config/claw-cargo/config.toml`
3. `$HOME/.claw-cargo/config.toml` (legacy, still honoured)
+3 tests: XDG env wins when the file exists, .config wins over
legacy dotfile when both present, legacy dotfile returned as
error-message fallback when none exist.
## Composite action
Before: composite action silently no-op'd. Log showed the script
lines echoed but only the `if command -v claw-cargo` fail branch
ran. Root cause: Gitea Actions composite steps run with a stripped
PATH that omits `/usr/local/bin`.
Now: composite step exports PATH defensively:
export PATH="/usr/local/bin:$HOME/.cargo/bin:$PATH"
And it looks for the config at BOTH the XDG-style path and the
legacy dotfile (same order as client_config).
Workflow file back to using the composite action.
## Docs
Runner-integration doc now:
- Explicitly warns that `ubuntu-latest` routes to container mode
even with `:host` suffix on runner labels (field-observed).
- Documents the dedicated `clawstor-cache` label pattern that
works.
- Sample workflow uses `runs-on: clawstor-cache` instead of
`ubuntu-latest`.
+3 tests, 262 total (baseline unchanged).
|
||
|
Omar Sobh
|
1053930451 |
claw-cargo: default --parallel-restore back to 1 (sequential)
Pi 5 loopback measurement 2026-07-13: --parallel-restore 1 : wall 2m52s, restore 20s --parallel-restore 8 : wall 3m06s, restore 34s Sequential is 70% faster on loopback. N-way stream contention costs more than a single stream's congestion-control amortization. Same shape as Phase 5k prewarm — fanout only wins when per-stream throughput has a ceiling (WAN, tunneled links). --parallel-restore N remains as opt-in. |
||
|
Omar Sobh
|
846ecffe10 |
Pi deploy follow-ups: XDG default_path + parallel restore
Two fixes surfaced by the vision-02 Pi 5 measurement: ## XDG default_path `Manifest::default_path` was hardcoded to `/var/lib/claw-store/projects.toml`. That path is read-only under the user-mode systemd unit's `ProtectSystem=strict`, and creating it needs root — awful for a runner install. Precedence, matching XDG Base Directory: 1. `$XDG_STATE_HOME/claw-store/projects.toml` 2. `$HOME/.local/state/claw-store/projects.toml` 3. `/var/lib/claw-store/projects.toml` (system fallback) User-mode installs now write in $HOME by default; system installs (root, no HOME set) still land in /var/lib. +1 test: `default_path_honours_xdg_state_home` — covers all three precedence branches. Env mutation is process-global so the test saves + restores. ## Parallel restore on cache HIT Pi restore of 947 MiB via `BlobGetStream` took ~18s (~53 MiB/s) single-stream. Per-stream throughput ceilings on the connection type cap sequential fetches; parallel chunk fetches stack their contributions. - New `call_blob_get_parallel(conn, blob_id, concurrency) -> Option<Vec<u8>>` in `rpc/client.rs`. `JoinSet` + `Semaphore`, reassembles by chunk index at manifest-known offsets so out-of-order arrival is fine. - `claw-cargo build --parallel-restore N` (default 8). `N <= 1` falls through to `BlobGetStream` for parity. - Memory: `total_size + 4 MiB × in-flight` — dominated by the reassembly buffer, not the fanout. +1 test: `parallel_blob_get_reassembles_multi_chunk_blob_byte_equal` covers roundtrip byte-equality vs BlobGetStream, tail-chunk offset, concurrency=1 correctness, and NotFound → None. 259 tests pass (+2). Pre-existing macOS failure unchanged. |
||
|
Omar Sobh
|
2f3055a3aa |
blob: size-based LRU eviction + auto-cap in the GC ticker
Orphan-chunk GC alone doesn't stop unbounded growth: as long as fingerprint→blob refs keep getting PutRef'd, the manifest set keeps growing and no chunk is ever an orphan. * `BlobStore::evict_to_size_cap(max_bytes)` — walks manifests oldest first by mtime, deletes them, refcount-decrements each chunk they used, unlinks + reclaims size for any chunk whose refcount hits zero. Shared chunks stay put until the last blob referencing them is evicted. * `ManifestSummary` internal type keeps the diff-set bookkeeping cheap (one HashMap<ChunkHash, u32>, no repeated tree walks). * `claw-store cluster-gc --evict-to-gb <N>` extends the CLI: still runs the orphan sweep first, then optionally caps the store. * Config: `cluster.blob_max_gb: Option<u64>`. The auto-GC ticker runs eviction after every orphan sweep when this is set. Silent when the store is already under cap; INFO log when it evicts. +3 tests: - evict_to_size_cap_reclaims_oldest_blobs_first: 3 blobs with distinct mtimes, cap below combined size → oldest evicted, newer blobs survive - evict_keeps_shared_chunks_when_still_referenced: guards the refcount decrement path (content-addressed dedup keeps identical content as one blob → chunk survives until manifest deleted) - evict_on_empty_store_is_a_noop: sanity 257 tests pass (baseline +3). Pre-existing macOS failure unchanged. |
||
|
Omar Sobh
|
84aa758fd4 |
Two pilot follow-ons: rustc drift warning + blob GC
Both surfaced by the 2026-07-12 pilot as real operator concerns: ## rustc drift warning at build time Runners silently silo their cache when rustc versions differ across peers (fingerprint depends on rustc verbose output). The pilot's first flow burned a full cold+upload before we realized the silo. - `PeerStatusReply.local_rustc_release` — new field, populated from the peer's own gossip `RUSTC_RELEASE` key via a new `ClusterGossip::self_kv(key)` accessor. - `claw-cargo build`: on cache MISS, calls `PeerStatus`; if the peer's rustc release ≠ our local `rustc --version`, emits a WARN with both versions + hint to add `rust-toolchain.toml`. - Best-effort: absence of either release string is a shrug, not an error. ## blob GC Blob store grows unbounded on a runner; disk-full is a real incident. `gc_orphan_chunks` already existed but wasn't exposed. - New CLI: `claw-store cluster-gc` — runs `gc_orphan_chunks`, prints report. Safe to run any time, safe to interrupt. - New config: `cluster.gc_interval_hours: Option<u64>`. When set to a positive integer, the daemon spawns a periodic ticker that invokes GC in-process. Skips the first tick (nothing to reclaim on boot). Errors are logged and retried next tick. - Shutdown aborts the ticker cleanly. 254 tests pass (baseline unchanged). Pre-existing macOS failure untouched. |
||
|
Omar Sobh
|
54e9da4d62 |
Phase 5k: parallel-fanout chunk transfer for prewarm
Pilot 2026-07-12 measured 109 MiB/s on the sequential prewarm path — ~11% of a 10G fabric. `quinn::Connection` is cheap-Clone (internal Arc), so we can run the has→get→put pipeline per chunk in concurrent tasks under a bounded semaphore. - `prewarm_missing_chunks_between_parallel(up, down, id, concurrency)` in `rpc/client.rs`. `concurrency <= 1` degrades to the sequential path (kept for diagnostic parity). - `claw-cargo prewarm --parallel N` (default 8). Ignored with `--buffered`. Memory ceiling: 4 MiB × in-flight = 32 MiB @ 8, 128 MiB @ 32. - Uses `tokio::task::JoinSet` + `Arc<Semaphore>`; permit held for the whole per-chunk pipeline so we never over-commit. - Retry pass on `put_manifest` mismatch stays sequential — small, correctness-critical. - Errors: JoinSet drains completely + returns first task error so a mid-fanout failure doesn't leave zombie tasks. +1 test: `end_to_end_parallel_prewarm_copies_chunks_and_matches_sequential` runs 5-chunk payload with concurrency=3, verifies byte-equal restore, then reruns with concurrency=8 → 0 uploads (has_chunk dedup), then concurrency=0 → 0 uploads (sequential fallback path). 254 tests pass (+1 from previous). Pre-existing macOS failure unchanged. |
||
|
Omar Sobh
|
cb07bfc574 |
capture: stream to a Writer instead of buffering the whole tar in RAM
Field finding 2026-07-12 (clawverse measurement): the buffered `capture_target -> Vec<u8>` path peaked at 2.8 GB RAM to capture a 6.1 GB target/debug into a 995 MiB compressed tar. Every byte crossed RAM before touching the network. * `capture_target_to_writer(target_dir, writer) -> u64` — new streaming variant. Walks the tree + writes tar+zstd straight into the caller's Writer via a small ByteCounter wrapper. Peak memory stays at ~zstd sliding window size (few MB). * `capture_target -> Vec<u8>` kept as a thin wrapper for the tests + smaller callers that don't care. * `cmd_build`: capture into a tempfile under `target/`, then open it with `tokio::fs::File` (AsyncRead + Unpin) and hand that to `call_blob_put_stream`. Same-filesystem tempfile means no cross- mount concerns; auto-unlinks on drop. +1 test: `capture_streaming_matches_buffered_and_restores_correctly` proves the streamed bytes match the buffered variant, the reported byte count agrees with the written length, and roundtrip restore from the streamed file works. Combined with PR #22 (QUIC idle timeout), this closes the two RAM/ timeout blockers surfaced by the clawverse pilot. Expected memory ceiling on a runner drops from GBs to MBs, unlocking small-runner deployments (the actual pitch use case). |
||
|
Omar Sobh
|
c08f60a2aa |
transport: bump QUIC idle timeout + add keep-alive for long builds
Field finding 2026-07-12 (clawverse cold on tank):
Compiling claw-cli v0.1.0 (...)
Finished `dev` profile ... in 45.08s
cargo build finished in 45.135491979s
Error: opening bidi stream for BlobPutStream
Caused by: timed out
Cargo took 45s → QUIC's 30s idle timeout killed the connection between
the initial peer-lookup connect and the follow-up capture+upload path.
The RPC never got a chance to open its stream.
Fix: two belt-and-braces changes:
1. IDLE_TIMEOUT 30s → 600s. The timeout is there to detect crashed
peers, not to enforce build pacing.
2. Client applies a `keep_alive_interval` of 15s so the connection
stays warm across cargo runs even shorter than the idle window.
quinn's keep-alive fires from an internal runtime task, not the app
thread, so a fully-CPU-pinned cargo build doesn't suppress it.
|
||
|
Omar Sobh
|
e70f5d74e0 |
Pilot findings: 5 real-world fixes from the 2026-07-12 deploy
Bundles the profile→dir bug (PR #20 supersede) with four new fixes discovered by running clawstor against itself + across the fabric: * target_subdir_for: `dev`/`test` → `debug/`, `release`/`bench` → `release/`, custom passes through. Was silently skipping upload. * rustc release via gossip: daemon probes `rustc --version` at start, publishes the release string as `clawstor.rustc.release`. PeerView carries it; `cluster-peer-status` prints it in a new column and emits a warning line when the fleet has mixed versions. Would have surfaced the tank/architect 1.96.1 vs 1.95.0 drift instantly. * prewarm publishes fingerprint→blob ref downstream: `pin` now writes a companion tag `<name>.fingerprint` holding the fingerprint bytes. `prewarm` reads the companion, PutTag's it downstream, then PutRef(fp→blob) so a subsequent fingerprint-based `build` HITS. Without this, prewarm was almost useless for the runner path (build always missed even with matching source + rustc). * streaming byte counters: BlobPutStream + BlobGetStream now record the transferred bytes via `record_blob_{put,get}_bytes`. Metric used to stay at 0 no matter how much you moved. * capture determinism: replaced `tar::Builder::append_dir_all` (uses `read_dir`'s native order) with `append_dir_sorted` that walks the tree recursively and sorts by filename bytes at every level. Two byte-identical trees now produce byte-identical tars regardless of filesystem ordering. +3 tests: - target_subdir_matches_cargo_layout (from #20) - fingerprint_companion_tag_uses_dotted_suffix - capture_is_order_independent_of_filesystem_readdir (guard against the exact bug we saw in the field) 252 tests pass (+1 from Phase 5h's 251). Pre-existing macOS `du -sb` failure unchanged. Supersedes #20 (also included here). Ready for re-deploy to tank + architect for the retest run. |
||
|
Omar Sobh
|
6fb16286dd |
Phase 5h: streaming chunk-level prewarm
Bounded-memory cross-peer prewarm: instead of buffering the whole blob in RAM (previous 5f path), iterate the upstream manifest chunk by chunk, ask downstream `HasChunk`, stream missing chunks one at a time. Memory ceiling is 1 chunk (4 MiB) regardless of blob size — a 5 GiB target dir no longer needs 5 GiB of mediator RAM. * rpc/client.rs: new `prewarm_missing_chunks_between(upstream, downstream, blob_id)` helper. Returns `(uploaded, total)` — the difference is the dedup save. Retries once if the downstream `PutManifest` reports missing chunks after our push (guards a narrow eviction race); a second failure surfaces as `Err`. * claw_cargo.rs: `prewarm` now streams by default; new `--buffered` flag for the old whole-blob path (kept for diagnostic comparability during rollout). Human-readable output shows the mode + dedup count. +2 tests: - cold downstream: 3-chunk payload (with a partial tail chunk) is copied exactly, reassembly is byte-equal to source - partial dedup: pre-seed 1 of 3 chunks on downstream → uploaded=2; rerun is a full no-op (uploaded=0), proving idempotence 251 tests pass (+2 from Phase 5j). Pre-existing macOS failure unchanged. Follow-ons: (1) parallel chunk transfer (uploaded chunks in fan-out) would speed multi-GB prewarms further; (2) exposing an accurate transferred-bytes counter needs router-side accounting instead of the current chunk-count × CHUNK_SIZE approximation. |
||
|
Omar Sobh
|
523b22f148 |
Phase 5j: Prometheus /metrics endpoint
Adds a tiny axum-served HTTP endpoint that exposes the same CacheMetrics counters that back GetMetrics + gossip, in Prometheus text exposition format (v0.0.4). Enable per node by setting `cluster.prom_bind` in the daemon config. * metrics.rs: MetricsReply::to_prometheus() emits one HELP + TYPE + sample line per counter. started_unix is a gauge; everything else is a counter. Preallocates ~1 KiB so no reallocs mid-format. * prom.rs (new): PromServer::bind spins up axum on a TcpListener, serves GET /metrics, returns 404 elsewhere. Graceful shutdown via oneshot channel; abort() variant for the sync-drop path in ClusterServices. Snapshots on every scrape (no cache) — Relaxed atomic loads are cheap enough that even 1 Hz is sub-microsecond. * config.rs: new optional `cluster.prom_bind: SocketAddr` field. Default None means no server; typical value is 127.0.0.1:7702. * services.rs: wires PromServer into ClusterServices when both a router and prom_bind exist. Warns (doesn't fail) if prom_bind is set without RPC — nothing would ever change on a scrape. +8 tests: - metrics: to_prometheus emits every counter with correct type; zeros still produce valid exposition (fresh daemon scrape) - prom: content-type is text/plain; version=0.0.4; live updates between requests (no cache); unknown paths 404; shutdown stops serving - services: end-to-end scrape returns router-driven counters; prom_addr() is None when unconfigured (no accidentally-leaked port) Raw-TCP HTTP client in tests instead of pulling in reqwest — 30 lines of tokio::net + string split for GET / read-to-close is small enough to justify not adding a dep. 249 tests pass (+8 from Phase 5i). Pre-existing macOS `du -sb` failure unchanged. |
||
|
Omar Sobh
|
29be728089 |
Phase 5i: publish cache metrics via gossip
ClusterServices now periodically snapshots the router's CacheMetrics and republishes four raw counters — GetRef hits/misses and blob GET/PUT byte totals — through chitchat as clawstor.cache.* keys. Peers derive the hit rate locally via PeerView::cache_get_ref_hit_rate, eliminating a per-peer GetMetrics roundtrip for placement decisions. * gossip.rs: 4 new well-known keys, PeerView carries the counters + a saturating-add derived hit rate that returns None on 0/0 or when either counter is missing (guards against half-writes reading as 100% hits). * services.rs: router hoisted out of the TLS branch so a cache_metric_task can hold Arc<RpcRouter>. Publishes once at start (initial zeros so peers don't wait 60s for first read) then every CACHE_METRIC_INTERVAL. Ticker skipped when RPC isn't up — counters only fire inside dispatch. +3 tests: - gossip: two-node convergence with cache counters + hit-rate math - gossip: half-write / 0-0 / normal PeerView cases return correct rates - services: fresh cluster sees Some(0) for all four keys, then after in-process router counters + explicit set_cache_metrics the peer sees the updated values with hit rate 0.8 241 tests pass (+3 from Phase 5g). Pre-existing macOS `du -sb` failure in hot::tests unchanged. |
||
|
Omar Sobh
|
d36cec11a6 |
Phase 5g: cache metrics + GetMetrics RPC + peer-metrics CLI
Every RPC handler that answers a hit-or-miss question now increments
lock-free atomic counters. The GetMetrics RPC (0x13) returns a JSON
snapshot of every counter; the new claw-cargo peer-metrics CLI
prints hit rates, byte volumes, and counter uptime.
Placement engines can now poll these across the fleet to bias runner
scheduling toward whichever node has the warmest cache for a given
repo/tag combination.
## New module: cluster/metrics.rs (268 lines)
Types:
- CacheMetrics — atomic counters, all AtomicU64, Relaxed ordering
(metrics are advisory, not consistency-critical)
- MetricsReply — JSON snapshot returned by GetMetrics
Public API:
- CacheMetrics::new() — timestamped start, all counters at 0
- record_get_ref_hit / _miss
- record_get_tag_hit / _miss
- record_blob_get_bytes / record_blob_put_bytes
- record_get_chunk_hit / _miss
- record_has_chunk_hit / _miss
- snapshot() — atomic-load every field into a MetricsReply
MetricsReply derived helpers:
- get_ref_hit_rate() / get_tag_hit_rate() / has_chunk_hit_rate() —
Option<f64> so 0/0 returns None instead of NaN
## RPC method
- GetMetrics (0x13): payload = empty; reply = JSON MetricsReply
Wire-level instrumentation added to RpcRouter dispatch:
- GetRef → record_get_ref_hit / _miss
- GetTag → record_get_tag_hit / _miss
- BlobGet → record_blob_get_bytes (on hit)
- BlobPut → record_blob_put_bytes
- HasChunk → record_has_chunk_hit / _miss
- GetChunk → record_get_chunk_hit + record_blob_get_bytes on hit
/ record_get_chunk_miss
RpcRouter grows Arc<CacheMetrics> unconditionally — every router has
metrics, so a fresh node with no traffic still returns a valid
snapshot with all zeros + started_unix.
Streaming variants (BlobPutStream / BlobGetStream) don't yet track
byte counts — they'd require plumbing the count out of put_stream /
stream_to. Follow-on if it turns out to matter for placement.
## Client helper + CLI
- call_get_metrics(&conn) → Result<MetricsReply>
- claw-cargo peer-metrics [--peer ...] [--peer-addr ...] [--tls-dir ...]
Fetches + pretty-prints:
counter uptime: 42s
GetRef hits/misses: 123 / 45
hit rate: 73.21%
GetTag hits/misses: 8 / 2
hit rate: 80.00%
HasChunk hits/miss: 512 / 88
hit rate: 85.33%
GetChunk hits/miss: 47 / 12
Blob GET bytes: 1.23 GiB
Blob PUT bytes: 3.45 GiB
human_bytes() helper picks GiB / MiB / KiB / B based on magnitude.
Subcommand count now 9: build / prefetch / status / fingerprint /
pin / unpin / list-tags / prewarm / peer-metrics.
## Tests (13 new, all real filesystem / real QUIC — no mocks)
CacheMetrics (6):
- new_starts_all_counters_at_zero_except_timestamp
- recorders_increment_the_right_field (every recorder × 1-2 counts)
- hit_rates_none_when_zero_events (avoids 0/0 NaN)
- hit_rates_compute_correctly (3 hits / 1 miss → 75%)
- snapshot_round_trips_through_json
- snapshots_across_threads_are_consistent_up_to_relaxed_ordering
(8 threads × 1000 increments → exactly 8000)
RPC integration (7):
- phase_5g_method_byte_encoding
- get_metrics_returns_empty_snapshot_before_any_activity
- get_ref_records_hit_and_miss_counters (2 hits + 1 miss)
- get_tag_records_hit_and_miss_counters
- blob_get_and_blob_put_record_byte_counts
- has_chunk_and_get_chunk_record_hit_miss_counters
- **end_to_end_get_metrics_over_real_quic** — seed activity locally,
fire GetRef/GetTag/BlobGet dispatches to move the counters, then
fetch metrics through real QUIC + mTLS and verify each field
including the 0.5 hit rate calculation
238 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/metrics.rs: 268
- cluster/rpc.rs: 818
- cluster/rpc/tests_phase5.rs: 905
- claw_cargo.rs: 894
## What this enables
Fleet-wide visibility into which peer is actually serving traffic:
# From anywhere with connectivity + fleet mTLS
claw-cargo peer-metrics --peer tank
claw-cargo peer-metrics --peer architect
claw-cargo peer-metrics --peer morpheus
Compare hit rates side by side to see which node's cache is warmest.
A placement engine can automate this — poll every 30s, feed the
scheduler.
## Follow-on
- 5h: streaming variant of prewarm (fixed-memory ceiling for many-GB
blobs)
- 5i: metrics also published via gossip so PeerView carries hit rate
without a per-peer GetMetrics roundtrip
- 5j: prometheus /metrics endpoint on the daemon for existing dash
integrations
- 6: FUSE mount for warm-tier git worktrees
|
||
|
Omar Sobh
|
db55903311 |
Phase 5f: claw-cargo prewarm — cross-peer cache copy
The last piece before "Gitea webhook triggers a cache-warm for the
CI runner before its build starts." Adds a `prewarm` subcommand that
copies a tagged cache from one peer (upstream) to another (downstream)
in one shot — same tag, same BlobId, both sides serve it after.
## New subcommand
```
claw-cargo prewarm \
--from-peer tank --from-addr 10.0.0.14:7702 \
--to-peer morpheus --to-addr 10.0.0.15:7702 \
--tls-dir /etc/claw-store/tls \
--pin clawverse:main:latest
```
Flow:
1. Connect to upstream with local mTLS identity
2. `GetTag(tag)` → BlobId; `BlobStat(BlobId)` → size + chunk count
3. `BlobGetStream(BlobId)` → download bytes
4. Connect to downstream (second QUIC endpoint, same identity)
5. `BlobPutStream(bytes)` → returns BlobId; verified equal to upstream's
6. `PutTag(tag → BlobId)` on downstream
Summary output shows tag, blob id, both endpoints, byte count,
download/upload timings, total wall clock.
Assumes upstream + downstream share the same fleet CA (the common
case). Mixed-fleet variant with distinct identities is a follow-on.
## Integrity check
`assigned_id != blob_id` after the downstream upload triggers a
bail — the two BlobIds must match because content is BLAKE3-hashed
end-to-end. If they don't, the wire path corrupted bytes and the
whole prewarm fails loud rather than silently pinning a bad blob.
## Buffered vs streamed
Current implementation buffers the whole blob in memory between
download and upload. Fine for cargo target dirs (~1-5 GB compressed);
would break for a 20 GB blob. A follow-on will pipe upstream → tokio
duplex → downstream to run at fixed memory.
## Tests (1 new, real 2-peer QUIC)
- **`end_to_end_prewarm_copies_tagged_blob_between_two_peers`**
Two full RpcRouters serving in-process (A upstream + C downstream),
each on a distinct port. Seeds A with a blob + tag, then runs the
exact sequence prewarm runs internally: `GetTag → BlobGetStream`
against A, then `BlobPutStream → PutTag` against C. Verifies that
C's blob store returns byte-equal payload and C's tag store now
points at the same BlobId. Proves the composition works.
## Housekeeping
`rpc/tests.rs` hit 1408 lines with the new prewarm test. Phase 5
tests (5b refs + 5d tags + 5e restore + 5f prewarm) split to
`rpc/tests_phase5.rs` via a second `#[path]` module in rpc.rs.
Result:
- rpc/tests.rs: 727 (phase 1-2d tests)
- rpc/tests_phase5.rs: 722 (phase 5 tests)
- rpc.rs: 777
- rpc/client.rs: 589
- claw_cargo.rs: 818
- All under ceiling.
225 tests pass. Pre-existing macOS-only failure unchanged.
## What this enables
The complete CI runner flow now works end-to-end:
```
Primary (e.g. tank):
claw-cargo build # first ever build — MISS, uploads
claw-cargo pin --name clawverse:main:latest
Fleet control plane on PR open:
gitea webhook → shell hook → claw-cargo prewarm \
--from-peer tank --to-peer $RUNNER_LOCAL \
--pin clawverse:main:latest
# runner's local daemon now serves the tag + blob
Runner picks up job:
claw-cargo build --peer 127.0.0.1:7702
# local daemon is warm → prefetch returns HIT
# cargo build runs against restored deps → workspace-crates only
# 50 min → 3 min
```
Every subcommand claw-cargo needs for this pipeline now exists:
build / prefetch / prefetch --pin / status / fingerprint /
pin / unpin / list-tags / prewarm.
## What's next
- 5g: cache hit/miss metrics into gossip so placement engines can
bias runner scheduling toward warm nodes
- 5h: streaming variant of prewarm (tokio duplex) for many-GB blobs
- 6: FUSE mount for warm-tier git worktrees
- 3: full CRDT metadata if the plain-tag model shows conflict problems
|
||
|
Omar Sobh
|
05ba800d01 |
Phase 5e: prefetch --pin <tag>
Small, focused extension to Phase 5c's prefetch: an optional
`--pin <tag-name>` flag that skips fingerprint compute entirely
and resolves the tag → BlobId via GetTag, then downloads that.
## Use case
Restore an old cache into a fresh checkout for regression testing:
$ claw-cargo prefetch --pin clawverse:main:2026-07-12
cache HIT — downloading 3221225472 bytes (768 chunks) to /path/target/dev
...
── claw-cargo prefetch ─────────────────────────────
source: --pin clawverse:main:2026-07-12
blob: 8c2f1a…
downloaded: 3221225472 bytes in 12.3s
restored to: /path/target/dev
────────────────────────────────────────────────────
Or diagnose a "why does this build fail against the pinned cache"
question by prefetching the tagged cache and then running cargo
against your current source. Cargo will detect the mismatched
.fingerprint state and rebuild affected crates — that's the point,
you're diffing behaviour between two known-good cache snapshots.
## Changes
- New PrefetchArgs struct (was reusing PeerArgs) with an optional
`pin: Option<String>` field
- resolve_pin(conn, tag) — internal helper that does
GetTag → BlobStat, returning None on either NotFound
- cmd_prefetch branches at the top: --pin → resolve_pin(); default
→ fingerprint-based peer_lookup()
- Rest of the flow is unchanged: BlobStat → BlobGetStream →
restore_target
- Summary output shows `source: --pin <tag>` instead of
`fingerprint: <hex>` when the pinned path was taken
`peer_lookup` (fingerprint path) and `resolve_pin` (tag path) return
the same `Option<(BlobId, BlobStat)>` shape so the downstream code
is identical.
## Live smoke test
`prefetch --help` now advertises --pin with full description.
Missing-tag path prints "no such tag: <name>" and exits 0
(consistent with the fingerprint-miss path).
## Tests (1 new, real QUIC)
- **`end_to_end_tag_resolve_and_stream_restore_over_real_quic`** —
seeds blob store with a 2 MiB "captured target" payload, publishes
a tag pointing at its BlobId, then runs the exact client
sequence `prefetch --pin <tag>` runs internally:
GetTag → BlobStat → BlobGetStream
Verifies bytes reassemble byte-equal to source. Also covers the
missing-tag path.
The pin flow uses the same underlying calls tested separately in
Phase 5b/5c/5d, so the new test proves the composition works rather
than re-verifying primitives.
224 tests pass. Pre-existing macOS-only failure unchanged.
## What's next
- 5f: Gitea webhook pre-fetch — daemon receives PR-open hints and
warms cache for the predicted fingerprint before CI runner starts
- 6: FUSE mount for warm-tier git worktrees so `~/projects/clawverse`
is transparently fleet-shared
- 3: full CRDT metadata layer (only if real conflicts emerge in the
simple tag model)
|
||
|
Omar Sobh
|
b7904b59a5 |
Phase 5d: named tags + pin/unpin/list-tags CLI
Human-readable pins on top of the raw 32-byte ref layer. Operators
publish `clawverse:main:latest-cache` → BlobId once, then everything
downstream (CI runners, dev laptops) references the tag instead of
passing 64-char hex hashes around.
## Module: cluster/tags.rs (433 lines)
TagStore for string-key → 32-byte-value:
- open(root) — creates layout, safe on existing stores
- put(key, value) / get(key) / delete(key) / contains(key)
- list() — sorted by key
- Atomic writes via tempfile + rename
- Key length capped at MAX_TAG_KEY_BYTES (4 KiB); empty keys rejected
On-disk record: `key_len:u16 (LE) || key_bytes || value:32bytes`.
Filename is `blake3(key)` hex so arbitrary UTF-8 keys land at
deterministic paths without shell escaping.
TagEntry type (public, serde) for list results:
`{ key, value_hex }`. Includes `decode_value() → Result<[u8;32]>`.
## RPC methods
- PutTag (0x0f): payload = encoded record → STREAM_STATUS_OK / err
- GetTag (0x10): payload = key bytes → 32-byte value / NotFound
- DeleteTag (0x11): payload = key bytes → STREAM_STATUS_OK / NotFound
- ListTags (0x12): payload = empty → JSON Vec<TagEntry>
RpcRouter grows optional Arc<TagStore> via `.with_tag_store(store)`.
## Services + config
ClusterServices auto-opens a TagStore at `<blob_store_root>/tags-db`
alongside the ref store. `tag_store` field on ClusterServices, same
enable-with-blob-store semantics.
## claw-cargo new subcommands
- `claw-cargo pin --name clawverse:main:latest`
Compute current fingerprint → look up its BlobId via GetRef →
publish TagStore mapping. Errors cleanly if the fingerprint
hasn't been built yet (nothing to point at).
- `claw-cargo unpin --name clawverse:main:latest`
Delete the tag. Prints "no such tag" if it wasn't set.
- `claw-cargo list-tags`
Print every tag with its 32-byte hex value.
Total subcommand count now 7: build / prefetch / status / fingerprint
/ pin / unpin / list-tags. All share the layered config from Phase 5c.
## Client helpers
- call_put_tag / call_get_tag / call_delete_tag / call_list_tags
- All follow the same error-mapping conventions as prior client helpers
(NotFound → Ok(None) or Ok(false), everything else → Err)
## Housekeeping
rpc.rs was pushing past the 1300-line ceiling with the tag methods
added. Client helpers moved to `cluster/rpc/client.rs` with a
re-export (`pub use client::*;`) so external callers still write
`cluster::rpc::call_*`. Result:
- rpc.rs: 773 (was 1343)
- rpc/client.rs: 593 (new)
- rpc/tests.rs: 1235
- All under ceiling.
## Tests (33 new, all real filesystem / real QUIC — no mocks)
TagStore (16 in cluster/tags.rs):
- open_creates_layout
- get_returns_none_for_missing (+ contains false)
- put_and_get_round_trip
- put_overwrites_prior_value
- delete_returns_true_for_existing_and_false_for_missing
- put_rejects_empty_key
- put_rejects_oversize_key
- list_returns_all_tags_sorted
- list_is_empty_on_fresh_store
- keys_with_slashes_and_colons_round_trip (real-world tag shape)
- encode_and_decode_round_trip (raw wire format)
- decode_rejects_short_record
- decode_rejects_length_mismatch
- decode_rejects_non_utf8_key
- tag_entry_decode_value_round_trip
- tag_entry_decode_value_rejects_bad_hex
RPC dispatch (7 new):
- phase_5d_method_byte_encoding
- tag_rpcs_return_not_configured_without_store
- put_tag_stores_and_get_tag_reads_back
- get_tag_returns_not_found_for_missing
- get_tag_rejects_empty_key
- delete_tag_removes_and_returns_not_found_after
- list_tags_returns_json_sorted
End-to-end over real QUIC (1):
- **end_to_end_pin_lookup_delete_over_real_quic** — publish tag →
look up → list → delete → confirm gone. Full round trip through
the wire layer including JSON deserialization of the list.
Also 8 downstream tests continued passing after the client.rs split
(no test moved, they were untouched).
223 tests pass. Pre-existing macOS-only failure unchanged.
## What this enables
Operator flow:
# Build once on the primary
$ claw-cargo build
→ cache MISS → cargo build (50 min) → capture + upload
→ summary: fingerprint 4a3b…, blob 8c2f…, uploaded 3.2 GiB
# Publish a friendly name
$ claw-cargo pin --name clawverse:main:2026-07-12
pinned: clawverse:main:2026-07-12
fingerprint: 4a3b2c…
blob: 8c2f1a…
# Anyone else can now find it via list-tags
$ claw-cargo list-tags
clawverse:main:2026-07-12 8c2f1a…
clawverse:main:latest 8c2f1a…
# CI runner sees the same fingerprint in its workspace state, hits
# the ref directly via GetRef — the tag is for operator visibility
## Follow-on
- 5e: prefetch --pin <tag> — bypass fingerprint compute, download
the tagged BlobId directly (useful when you want an old cache to
test regression scenarios)
- 5f: gitea webhook pre-fetch — daemon pre-warms cache for known
fingerprints before CI runner starts
- 3: full CRDT metadata layer (namespaces, versioned pointers,
vector clocks) if the plain-tag model turns out to have
real-world conflict scenarios
|
||
|
Omar Sobh
|
57d8358255 |
Phase 5c: claw-cargo UX — config files + status + prefetch
Ships the last-mile ergonomics that make claw-cargo actually usable
day-to-day: layered config files so you don't retype --peer-addr on
every invocation, plus two lightweight subcommands (status +
prefetch) for the "what's in the cache" and "warm my target dir"
workflows respectively.
## Config precedence
Later wins:
1. Built-in defaults (profile=dev, features=[])
2. ~/.claw-cargo/config.toml (per-user defaults)
3. <workspace>/.claw-cargo.toml (per-repo overrides)
4. CLI flags (per-invocation overrides)
Shape:
[peer]
name = "tank"
addr = "10.0.0.14:7702"
tls_dir = "/etc/claw-store/tls"
[build]
profile = "release"
features = ["a", "b"]
## New module: cluster/client_config.rs (496 lines)
- ClientConfig / PeerSection / BuildSection — TOML-serialisable, all
Option<> fields at every layer so partial configs are legal
- ClientConfig::from_toml_str / from_file_or_default (missing file →
default, not error)
- ClientConfig::merge — Option::Some in `other` wins over `self`
- ClientConfig::load_layered(workspace) — user → workspace
- ResolvedClientConfig — final flattened shape after CLI overrides,
with require_peer_name / require_peer_addr / require_tls_dir /
require_peer_bundle helpers that produce a specific error message
instead of "some Option was None"
- write_config_file — for tests + future `claw-cargo config init`
Ships with 11 unit tests including a load_layered test that fakes
HOME + workspace via a tempdir, writes both configs, verifies the
workspace override takes precedence.
## claw-cargo (rewritten to 469 lines)
Four subcommands with layered config:
claw-cargo fingerprint [--profile ...] [--features ...] [--workspace ...]
→ local-only, no network
claw-cargo status <peer args>
→ connect + GetRef + BlobStat, print hit/miss + size, no download
claw-cargo prefetch <peer args>
→ hit → BlobGetStream + restore_target, no cargo
claw-cargo build <peer args> [-- extra cargo args]
→ same as Phase 5b flow, now with layered config for peer args
Refactored internals:
- setup_local / setup_peer — figure out workspace, load config,
resolve CLI overrides, compute fingerprint
- connect_peer — load NodeIdentity, open QUIC connection
- peer_lookup — GetRef → BlobStat, handle the "ref points at a
garbage-collected blob" case as a miss
## Live smoke test
Verified end-to-end on this workspace:
# No config file → built-in defaults
$ claw-cargo fingerprint
profile: dev, features: (none), fingerprint: 8ee4cf…
# Add .claw-cargo.toml with profile=release + features=some-feature
$ claw-cargo fingerprint
profile: release, features: some-feature, fingerprint: 2dfcb1…
# CLI overrides just the profile; features fall through from config
$ claw-cargo fingerprint --profile dev
profile: dev, features: some-feature, fingerprint: 84a756…
# `status` without peer args → clean validation error
$ claw-cargo status
Error: peer.name not set (config file or --peer)
## Tests (11 new, all real filesystem — no mocks)
- from_toml_str_parses_full_config
- from_toml_str_handles_partial_sections (peer.name only)
- from_file_or_default_returns_default_when_missing
- merge_prefers_later_over_earlier (unset fields fall through)
- resolve_applies_cli_overrides_over_layered
- resolve_falls_through_to_default_profile_when_unset_everywhere
- require_peer_bundle_errors_when_incomplete (specific error text)
- validate_peer_errors_on_missing_field
- load_layered_reads_both_files — fake HOME + workspace, verifies
workspace override takes precedence
- user_config_path_uses_home
- write_and_read_round_trip_via_disk (nested dir creation)
199 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (well under 1300-line ceiling):
- cluster/client_config.rs: 496
- claw_cargo.rs: 469
## What's next
The CLI is now usable day-to-day. Realistic next steps:
- 5d: publish cache hit/miss metrics into gossip so the placement
engine can bias runner scheduling toward warm nodes
- 5e: pre-fetch on Gitea webhook — daemon receives a "PR opened for
fingerprint X" hint and warms the local cache before the runner
even starts pulling
- 3: CRDT metadata for human-readable pins (`clawverse:main:latest`
→ fingerprint hex) so operators can pin cache versions without
passing raw hashes around
- 6: FUSE mount so `~/projects/clawverse` on any node is transparently
the tank-hosted canonical warm-tier copy
|
||
|
Omar Sobh
|
2d09b4687c |
Phase 5b: KV refs + claw-cargo CLI (the killer feature, live)
Ships the actual user-facing cargo build cache. Combined with Phase 5a
(fingerprint + capture + restore) + the whole Phase 2 blob substrate,
`claw-cargo build` now runs `cargo build` with a peer-cache lookup:
hit → download+restore, miss → build+capture+upload.
## What ships
### cluster/refs.rs (243 lines)
A dumb 32-byte-key → 32-byte-value directory-backed store. Used to map
fingerprints → BlobIds. Layout mirrors BlobStore:
<root>/
refs/<kk>/<key_hex>.ref — 32 raw bytes
.tmp/ — atomic-rename staging
Public API: RefStore::open / get / put / delete / contains. All writes
atomic via tempfile + rename. Deliberately no versioning or CRDT
semantics — that's Phase 3. Every real cargo-cache lookup is a
single-key-single-value shape.
### New RPC methods
- GetRef (0x0d): payload = 32-byte RefKey; reply = 32 bytes / NotFound
- PutRef (0x0e): payload = 32-byte RefKey || 32-byte RefValue;
reply = STREAM_STATUS_OK / error
### RpcRouter + services
- RpcRouter grows optional Arc<RefStore> via `with_ref_store`
- ClusterServices opens a RefStore alongside the BlobStore when
`blob_store_root` is configured (co-located at `<blob_root>/refs-db`)
- `blob_store_enabled()` / `ref_store_enabled()` introspection
### claw-cargo binary (319 lines)
New bin target `claw-cargo` — thin CLI wrapping the whole stack:
claw-cargo fingerprint --profile release --features "a,b"
→ prints the workspace fingerprint (no network)
claw-cargo build \
--peer <name> --peer-addr <ip:port> --tls-dir <dir> \
--profile release --features "a,b" \
-- --workspace=x --frozen ...
→ 1. compute fingerprint
2. QUIC + mTLS connect to peer
3. GetRef(fingerprint) → BlobId?
HIT: BlobStat → BlobGetStream → restore_target → cargo build
MISS: cargo build → capture_target → BlobPutStream → PutRef
4. Print summary: fingerprint, hit/miss, bytes, cargo elapsed
## Live smoke test
Ran claw-cargo fingerprint on this workspace with three profile/feature
combos — got three distinct 32-byte fingerprints. Same profile+features
on the same workspace state → same fingerprint (Phase 5a's guarantee
carried through the CLI).
## Tests (14 new, all real — no mocks)
Refs store (7):
- open creates layout
- get returns None for missing
- put + get round-trips
- put overwrites prior value
- delete removes ref + reports (false on second delete)
- distinct keys produce distinct on-disk files (bucket fan-out proof)
- rejects_wrong_length_on_disk (corruption detection)
RPC (7):
- phase_5b_method_byte_encoding
- get_ref_returns_not_found_for_missing
- put_ref_stores_and_get_ref_reads_back
- put_ref_rejects_wrong_length_payload
- get_ref_rejects_wrong_length_payload
- ref_rpcs_return_not_configured_without_store
- end_to_end_put_ref_get_ref_over_real_quic — full 2-node QUIC + mTLS
round trip proving PutRef/GetRef work at the wire level
188 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/refs.rs: 243
- cluster/rpc.rs: 1169
- cluster/rpc/tests.rs: 1073
- cluster/services.rs: 565
- claw_cargo.rs: 319
## Where this leaves us
The distributed FS + cargo cache is functionally complete for the
happy path:
Node A builds clawverse for the first time
→ cargo build (50 min cold)
→ capture_target (a few seconds)
→ push to node B via BlobPutStream (network-bound)
→ PutRef(fingerprint → BlobId)
Node B on the same workspace state runs `claw-cargo build …`
→ compute_fingerprint (ms)
→ GetRef → hit
→ BlobGetStream (network-bound)
→ restore_target (a few seconds)
→ cargo build → sees valid deps/.fingerprint, builds only
workspace crates (~3 min instead of 50)
Same workspace state on a third machine? Same fingerprint → same
cache hit. That's the whole design.
## Follow-on
- Phase 5c: pre-fetch on Gitea webhook so CI runners never wait
- Phase 5d: metric ticker publishes cache hit rate into gossip so
the placement engine can bias runner scheduling toward warm nodes
- Phase 3: CRDT metadata for human-readable pins on top of raw
32-byte refs (`clawverse:main:latest-cache` → fingerprint hex)
- Phase 6+: FUSE mount for the warm-tier git worktrees
|
||
|
Omar Sobh
|
242ef527b4 |
Phase 5a: fingerprint + capture + restore for build-artifact cache
The substrate for the killer feature. Given a cargo workspace, compute
a deterministic 32-byte BLAKE3 fingerprint over the inputs that
determine what artifacts should be produced, then bundle the portable
subset of `target/<profile>/` into a zstd-compressed tarball ready to
hand to BlobStore.
## Module: cluster/build_cache.rs (661 lines)
Types:
- FingerprintInputs { cargo_lock, rustc_version_verbose, cargo_config,
rust_toolchain, profile, features, rustflags, target_triple }
- Fingerprint(32 bytes) — parallel shape to BlobId
Public API:
- FingerprintInputs::collect(workspace, profile, features) — reads
Cargo.lock, shells out to `rustc --version --verbose`, reads
optional config files, extracts host triple, sorts+dedups features
- FingerprintInputs::compute() → Fingerprint — domain-separated
BLAKE3 with per-field labels + null separators so field boundaries
can't collide
- capture_target(target_dir) → zstd-tarball bytes
- restore_target(bytes, target_dir) → unpacks
- capture_workspace(workspace, profile) — resolves target dir
- compute_workspace_fingerprint(workspace, profile, features) —
returns (inputs, fingerprint)
Captured: deps/, .fingerprint/, build/, examples/, plus small
top-level files (.cargo-lock, .rustc_info.json, CACHEDIR.TAG).
Explicitly NOT captured: incremental/ (per-machine, not portable —
tied to absolute paths + rustc state; restoring on another host
silently corrupts the build).
Determinism guarantees:
- HeaderMode::Deterministic on the tar builder — identical trees
produce byte-identical tarballs (proven by
`capture_yields_identical_bytes_for_identical_input`)
- follow_symlinks(false) — symlinks archived as symlinks, not their
targets, so the fingerprint doesn't drift with symlink destinations
- Features sorted + deduped so `["b","a"]` and `["a","b"]` hash the same
- Missing optional files treated as empty strings so `absent ==
empty` (add-then-remove doesn't churn the hash)
Deps added: tar 0.4, zstd 0.13.
## Tests (18 new, all real filesystem — no mocks)
Fingerprint (7):
- fingerprint_is_deterministic
- fingerprint_changes_when_cargo_lock_changes
- fingerprint_changes_when_profile_changes
- fingerprint_changes_when_features_change
- fingerprint_is_feature_order_independent (proves sort semantics)
- fingerprint_domain_separation_prevents_field_collision — swap
content between two string fields; naive concat hasher would
collide, ours doesn't
- fingerprint_hex_length_and_stability
Input collection (4):
- read_optional_returns_empty_for_missing
- read_optional_returns_content_for_existing
- collect_errors_when_cargo_lock_absent
- collect_reads_cargo_lock_and_computes — actually shells out to
`rustc --version --verbose`, verifies triple extraction
Capture/restore (5):
- capture_target_errors_when_dir_missing
- capture_and_restore_round_trip_preserves_files — full 6-file
tree including a captured example + an intentionally-excluded
incremental/ dir; verifies incremental/ is absent after restore
- capture_yields_identical_bytes_for_identical_input — byte-equal
tarballs across two identical source trees
- capture_skips_top_level_files_not_in_allowlist
- capture_workspace_resolves_profile_dir
End-to-end (1):
- **end_to_end_fingerprint_capture_blob_restore** — the full
workspace → fingerprint → capture → BlobStore::put_bytes →
BlobStore::get_bytes → restore_target loop. Proves the whole
round-trip lands byte-equal for both captured file trees, and
the BlobId is deterministic across runs. This is the primitive
the cargo-cache CLI sits on top of.
174 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (well under 1300-line ceiling):
- cluster/build_cache.rs: 661
- cluster.rs (submodule declarations): 279
## What this unlocks
Phase 5a is the substrate. The CLI wrapper (`claw-cargo build`) is
Phase 5b — thin glue that:
1. Runs compute_workspace_fingerprint()
2. Asks the peer BlobStat(fingerprint_as_blob_id)
3. Hit → BlobGetStream + restore_target + cargo build (just the
workspace's own crates, ~seconds)
4. Miss → cargo build (full), then capture_target +
push_blob_missing_chunks + PutManifest
Every piece of infrastructure that Phase 5b needs — content-addressed
blob store, streaming and partial-chunk RPC, mTLS transport, gossip-
driven peer discovery — is already merged. Phase 5b is CLI polish,
not new distributed-systems machinery.
## Follow-on
- Phase 5b: claw-cargo CLI wrapper (small — 200-400 lines)
- Phase 5c: pre-fetch on Gitea webhook (workflow triggers → daemon
pre-warms the fingerprint on the target runner)
- Phase 3 remains a parallel track — human-readable namespace layer
on top of raw fingerprints so operators can pin
`clawverse:main:latest-cache` instead of a hex string
|
||
|
Omar Sobh
|
2e984b924d |
Phase 2d: chunk-level RPC (HasChunk / PutChunk / GetChunk / PutManifest)
Unlocks partial-sync replication — a peer that already has some
chunks of a blob (typical when two nodes share overlapping cargo
build caches) only receives the chunks it's missing.
## New methods
| Byte | Method | Payload | Reply |
|---|---|---|---|
| 0x09 | HasChunk | 32-byte ChunkHash | STREAM_STATUS_OK / NotFound |
| 0x0a | PutChunk | ChunkHash \|\| bytes | STREAM_STATUS_OK / error |
| 0x0b | GetChunk | ChunkHash | STREAM_STATUS_OK \|\| bytes / NotFound |
| 0x0c | PutManifest | JSON BlobManifest | JSON PutManifestReply |
`PutManifestReply { blob_id, missing: Vec<ChunkHash> }`: empty
`missing` means the manifest was written; non-empty tells the
client which chunks to upload before retrying.
Server verifies bytes hash to claimed hash on PutChunk; a
mismatch surfaces as InvalidRequest and the store is untouched.
## BlobStore additions
- `has_chunk(&ChunkHash) → bool`
- `read_chunk(&ChunkHash) → Option<Vec<u8>>` — verifies hash on read
- `put_chunk(&ChunkHash, bytes) → Result<()>` — verifies bytes-vs-hash
- `put_manifest_verified(&manifest) → Result<Vec<ChunkHash>>` —
returns the list of chunks missing on disk (empty on success)
- `chunk_path` promoted to `pub` for advanced callers
## Client helpers
- `call_has_chunk` / `call_put_chunk` / `call_get_chunk` / `call_put_manifest`
- `push_blob_missing_chunks(conn, local_store, blob_id) →
Result<(uploaded, total)>` — high-level partial-sync helper
`push_blob_missing_chunks` loads the local manifest, calls HasChunk
for each chunk, uploads only the missing ones via PutChunk, then
commits via PutManifest. On a fully-overlapping cache the uploaded
count is 0 and only the ~small manifest crosses the wire.
## Tests (17 new, all real filesystem + real QUIC — no mocks)
Blob store (6):
- has_chunk_is_false_before_put_and_true_after
- read_chunk_returns_bytes_and_none_when_missing
- put_chunk_rejects_hash_mismatch (nothing written)
- read_chunk_detects_corruption (bit-flip → mismatch error)
- put_manifest_verified_reports_missing_chunks
- put_manifest_verified_writes_when_all_chunks_present
Router dispatch (7):
- phase_2d_method_byte_encoding
- method_reports_streaming_variants — extended for 4 new methods
- has_chunk_returns_ok_for_present_and_not_found_for_missing
- put_chunk_stores_and_returns_status_ok
- put_chunk_rejects_hash_mismatch_over_wire
- get_chunk_returns_content_prefixed_with_status_ok
- get_chunk_returns_not_found_for_missing
- put_manifest_reports_missing_chunks_when_incomplete
- put_manifest_writes_when_chunks_present
- chunk_rpcs_return_not_configured_without_store
End-to-end (2):
- **end_to_end_push_blob_missing_chunks_replicates_only_needed_bytes**:
Peer A pre-seeded with chunk 0 of a 2-chunk (8 MiB) blob;
`push_blob_missing_chunks` reports `(uploaded=1, total=2)`,
only chunk 1 crosses the wire, A's store then contains the
complete blob and `get_bytes` returns byte-equal content.
- **call_get_chunk_verifies_returned_hash**: real 2-node fetch,
client hashes received bytes and compares to requested hash.
156 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 1053
- cluster/rpc/tests.rs: 940
- cluster/blob.rs: 1186
## Where this fits
With Phase 2c whole-blob streaming + Phase 2d partial-chunk sync,
the storage substrate is now genuinely bandwidth-efficient in the
distributed setting:
- First-ever push of a blob: `push_blob_missing_chunks` uploads
everything (all chunks missing).
- Second push of a similar blob (95% chunk overlap with prior
contents): only the 5% new chunks cross the wire, plus a tiny
manifest.
- Whole-blob download: BlobGetStream, bounded by network bandwidth.
## Follow-on
- Phase 3: CRDT metadata for human-readable namespaces on top of
content hashes.
- Phase 5: the killer feature. Fingerprint cargo target dir → tar
→ hash → PutBlobStream (or push_blob_missing_chunks if a similar
build already lives on the peer). Same fingerprint on the next
node → BlobGetStream. This is the whole cargo-cache design in
one line and it now sits on a substrate that handles all the
hard cases (dedup, verification, resumability, partial sync).
|
||
|
Omar Sobh
|
1fd1027da4 |
Phase 2c: streaming Blob RPC (BlobPutStream / BlobGetStream)
Removes the 16 MiB message cap for blob transfers. The bounded Blob* methods from Phase 2b still exist; the streaming variants let a peer push or pull a many-GB blob without either side holding it in memory. ## Wire format Streaming methods use a slightly different reply shape so the client can route on the first byte alone: Reply : status:u8 || payload:bytes... Where `status` is either `STREAM_STATUS_OK` (0x00, content follows) or a single-byte ErrorCode. `serve_connection` now peeks at the method tag byte via read_exact and hands streaming methods the raw send/recv streams; bounded methods still use the old read_to_end path. ## Method additions - BlobPutStream (0x07): client streams bytes → server pipes into BlobStore::put_stream → reply is 0x00 || 32-byte BlobId - BlobGetStream (0x08): client sends 32-byte BlobId → server verifies existence, writes 0x00 status, then streams chunks from disk into the send stream Method::is_streaming() introspection so callers can decide which wire variant to use. ## BlobStore additions - put_stream<R: AsyncRead + Unpin>(reader) -> BlobId Memory ceiling: one CHUNK_SIZE (4 MiB) buffer regardless of blob size. Handles short-reads correctly (loops until CHUNK_SIZE bytes are available or EOF), including the empty-reader case (produces the empty-blob BlobId, zero chunks). - stream_to<W: AsyncWrite + Unpin>(id, writer) -> bool Ok(false) on NotFound (writer untouched). Verifies each chunk hash before emitting; corruption halts mid-stream with Err. ## Client helpers - call_blob_put_stream(conn, reader) -> Result<BlobId> Uses tokio::io::copy directly onto quinn's SendStream. - call_blob_get_stream(conn, id, writer) -> Result<bool> Ok(false) on NotFound; other errors surface as Err. ## Tests (11 new, all real — no mocks) Blob store (6): - put_stream_produces_same_hash_as_put_bytes (3-chunk blob via Cursor) - put_stream_handles_empty_reader (produces empty-blob BlobId) - put_stream_handles_short_reads (custom Trickle reader that only serves 100 bytes per read call — must still assemble full chunks) - stream_to_writes_full_blob (2-chunk write to Vec<u8>) - stream_to_returns_false_when_missing (writer untouched) - stream_to_detects_chunk_corruption (bit-flip a chunk → Err with "chunk hash mismatch") RPC (5): - method_reports_streaming_variants - end_to_end_stream_put_and_get_over_real_quic — 12 MiB + 777 bytes → 4 chunks, real 2-node QUIC + mTLS + stream round-trip - stream_get_returns_false_for_missing_blob - stream_methods_return_not_configured_without_store - stream_put_deduplicates_with_prior_put_bytes — verify streaming put produces the same BlobId as a prior bounded put on identical content, and the manifest chunk count didn't fork ## Housekeeping rpc.rs was tipping over the 1300-line ceiling with the streaming handlers + helpers + tests. Tests split into `cluster/rpc/tests.rs` via `#[path = "rpc/tests.rs"] mod tests;`. Result: - rpc.rs: 748 lines - rpc/tests.rs: 694 lines - blob.rs: 1002 lines - All under ceiling. 139 tests pass. Pre-existing macOS-only failure unchanged. ## What's next - Phase 2d: chunk-level RPC (BlobPutChunk / BlobGetChunk) so a receiver can `LoadManifest` then request only the chunks it's missing — big bandwidth win on partially-overlapping caches. - Phase 3: CRDT metadata for human-readable namespaces on top of content hashes. - Phase 5: the killer feature — fingerprint the cargo target dir, BlobPutStream it, next node BlobGetStream by the same fingerprint. Now buildable directly on Phase 2c since target dirs run 100 MB to a few GB and the previous 16 MiB cap would have blocked us. |
||
|
Omar Sobh
|
f43b34ad15 |
Phase 2b: Blob RPC (Stat/Get/Put/LoadManifest)
Wires the Phase 2 content-addressed blob store onto the network via
four new methods on the existing RpcRouter. Combined with the mTLS +
gossip stack from Phase 1c-1e, peers can now exchange content-addressed
blobs over real QUIC. This is the substrate the fingerprint-keyed cargo
cache (Phase 5) sits on directly.
New methods:
- BlobStat (0x03): payload = 32-byte BlobId, reply = JSON BlobStat
- BlobGet (0x04): payload = 32-byte BlobId, reply = raw bytes
- BlobPut (0x05): payload = raw bytes, reply = 32-byte BlobId
- BlobLoadManifest (0x06): payload = 32-byte BlobId, reply = JSON manifest
New error codes:
- NotFound (0xf3) — the requested BlobId isn't in the local store
- InvalidRequest (0xf4) — e.g. non-32-byte payload for a hash-keyed method
- NotConfigured (0xf5) — Blob* called on a router without an attached store
Wire-format bump: MAX_MESSAGE_BYTES 16 KiB → 16 MiB so a single 4 MiB
chunk (plus JSON overhead) fits comfortably. Anything above 16 MiB
needs the streaming variants coming in Phase 2c.
Client helpers:
- call_blob_stat / call_blob_get / call_blob_put / call_blob_load_manifest
- All map ErrorCode::NotFound to Ok(None), other codes to Err.
- call_blob_put verifies the peer-assigned BlobId matches local blake3
hash of the payload — corruption or protocol drift surfaces
immediately instead of silently accepting a mismatched receipt.
Router changes:
- RpcRouter grows an Option<Arc<BlobStore>> via with_blob_store(store).
- handle() split into handle_outcome() → HandlerOutcome enum
{Reply(bytes) | Error(ErrorCode)} for cleaner control flow across
the growing method set.
Services / daemon:
- ClusterServices::start gains blob_store_root: Option<PathBuf>.
- ClusterConfig gains blob_store_root: Option<PathBuf>.
- daemon.rs reads it from cluster_cfg + passes through.
- New ClusterServices::blob_store_enabled() introspection.
Tests (11 new, all real — no mocks):
Router (7 new):
- method_round_trips_byte_encoding — updated for 6 methods
- error_code_describe_covers_all_variants — updated for 6 codes
- decode_error_covers_all_known_codes
- blob_rpcs_return_not_configured_without_store — all four Blob*
methods return NotConfigured when the router lacks a store
- blob_stat_returns_not_found_for_missing
- blob_stat_returns_json_for_existing
- blob_stat_returns_invalid_request_for_bad_length
- blob_get_returns_content_bytes
- blob_put_stores_bytes_and_returns_hash — verifies BlobId matches
independent local hash
- blob_load_manifest_returns_json_for_existing (2-chunk case)
- blob_load_manifest_returns_not_found_for_missing
End-to-end over real QUIC (2 new):
- end_to_end_blob_put_stat_get_over_real_quic — full 4-method loop
(Put → Stat → Get → LoadManifest) + NotFound path
- end_to_end_multi_chunk_blob_over_real_quic — 6 MiB blob → 2 chunks,
proves MAX_MESSAGE_BYTES bump took effect
Services (1 new):
- services_with_blob_store_serves_blob_rpc_end_to_end — cut CA, sign
leaves, config includes blob_store_root, start ClusterServices,
dial from B over persisted mTLS, put + get through the router,
then independently verify bytes landed on A's on-disk store
Also fixed a parallel-test port collision: services `next_port()`
now increments by 2 so `port + 1` (the RPC bind) is reserved
alongside `port` (the gossip bind).
128 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 1006
- cluster/services.rs: 535
- cluster/blob.rs: 802
- config.rs: 511
- daemon.rs: 265
Follow-on:
- 2c: streaming variants (AsyncRead/AsyncWrite) so a many-GB blob
transfers without holding it in memory
- 2d: chunk-level RPC (BlobPutChunk / BlobGetChunk) so a receiver
can request only chunks it's missing after LoadManifest
- Phase 5 (the killer feature) can now build on Phase 2b directly —
fingerprint the target dir, PutBlob the compressed tarball, and
next node calls GetBlob keyed by the same fingerprint hash.
|
||
|
Omar Sobh
|
eab10005fd |
Phase 2: content-addressed blob store
The storage substrate everything after Phase 1 sits on. Every blob is
identified by its BLAKE3 whole-content hash (BlobId); on disk it lives
as an ordered sequence of BLAKE3-hashed 4 MB chunks, so blobs that
share a prefix (two cargo target dirs with 95% of the same deps) share
storage at chunk granularity with no special detection logic.
New: cluster/blob.rs (802 lines).
Types:
- BlobId — 32-byte BLAKE3 output, hex-serialised (serde ↔ string)
- ChunkHash — same shape as BlobId but a distinct type so blob and
chunk lookups can't accidentally swap
- BlobStat — { total_size, chunk_count }
- BlobManifest — { blob_id, total_size, chunks: Vec<ChunkHash> },
public because Phase 2b RPC serves it directly so a receiver can
request only the chunks it's missing
- GcReport — { chunks_scanned, chunks_removed, bytes_reclaimed }
- BlobStore — root-directory-based store
Public API:
- BlobStore::open(root)
- put_bytes(&[u8]) → BlobId
- get_bytes(&BlobId) → Option<Vec<u8>> (verifies hash + size on read)
- contains(&BlobId) → bool
- stat(&BlobId) → Option<BlobStat>
- load_manifest(&BlobId) → Option<BlobManifest>
- delete_manifest(&BlobId) → bool (chunks stay; orphan by GC)
- gc_orphan_chunks() → GcReport
On-disk layout:
<root>/
blobs/<bb>/<blob_hash>.manifest.json
chunks/<cc>/<chunk_hash>
.tmp/
Two-char bucket prefixes cap fan-out at 256 entries per level — safe
on a warm-tier ZFS dataset with tens of thousands of blobs.
Every write is atomic (tmp file + rename on same filesystem).
Every chunk write is a no-op if the file already exists — same
content across two put()s stores exactly one physical copy.
Correctness:
- Reads verify each chunk against its hash + recompute the whole-blob
hash before returning; a bit-flipped chunk raises "chunk hash
mismatch" instead of silently corrupting the answer.
- delete_manifest is the only deletion primitive; chunks are only
ever removed by gc_orphan_chunks after a full manifest scan proves
they're unreferenced.
Dep: blake3 = "1".
Tests (21 new, all real filesystem, no mocks):
- BlobId/ChunkHash hex round-trip + serde JSON
- BlobId::from_hex rejects wrong-length + non-hex input
- open creates blobs/, chunks/, .tmp/
- put + get round trip: small, empty, 10 MB (3 chunks)
- put is deterministic (same bytes → same BlobId every time)
- put is idempotent (writing twice → exactly one manifest file)
- Different content → different BlobId
- shared_chunks_are_stored_only_once: two blobs sharing a 4 MB prefix
produce exactly 3 chunk files, not 4
- get_returns_none_when_missing / contains false / stat None
- delete_manifest keeps chunks (proven by counting chunk files)
- delete_manifest on missing returns false
- gc_reclaims_orphan_chunks_but_keeps_referenced: put 2 blobs, delete
one manifest, GC removes exactly the orphaned chunk, keeps
live A readable
- gc_on_empty_store_reports_zero
- corrupted_chunk_detected_on_read: rewrite a chunk with garbage →
get_bytes errors with "chunk hash mismatch"
- load_manifest round-trips the chunk list
116 tests pass. Pre-existing macOS-only failure unchanged.
File size: cluster/blob.rs = 802 lines (ceiling 1300).
Follow-on Phase 2 cuts:
- 2b: RPC methods BlobStat / BlobGet / BlobPut, wired into RpcRouter
and served over the QUIC transport built in Phase 1c-1e.
- 2c: streaming put/get (AsyncRead / AsyncWrite variants) for
many-GB build artifacts.
Phase 3 (metadata + CRDTs) can start independently — this store is the
substrate the fingerprint cache in Phase 5 layers onto.
|
