31721e7657a561c0e679d28ed14323fb3fa66e47
115
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar Sobh
|
d36cec11a6 |
Phase 5g: cache metrics + GetMetrics RPC + peer-metrics CLI
Every RPC handler that answers a hit-or-miss question now increments
lock-free atomic counters. The GetMetrics RPC (0x13) returns a JSON
snapshot of every counter; the new claw-cargo peer-metrics CLI
prints hit rates, byte volumes, and counter uptime.
Placement engines can now poll these across the fleet to bias runner
scheduling toward whichever node has the warmest cache for a given
repo/tag combination.
## New module: cluster/metrics.rs (268 lines)
Types:
- CacheMetrics — atomic counters, all AtomicU64, Relaxed ordering
(metrics are advisory, not consistency-critical)
- MetricsReply — JSON snapshot returned by GetMetrics
Public API:
- CacheMetrics::new() — timestamped start, all counters at 0
- record_get_ref_hit / _miss
- record_get_tag_hit / _miss
- record_blob_get_bytes / record_blob_put_bytes
- record_get_chunk_hit / _miss
- record_has_chunk_hit / _miss
- snapshot() — atomic-load every field into a MetricsReply
MetricsReply derived helpers:
- get_ref_hit_rate() / get_tag_hit_rate() / has_chunk_hit_rate() —
Option<f64> so 0/0 returns None instead of NaN
## RPC method
- GetMetrics (0x13): payload = empty; reply = JSON MetricsReply
Wire-level instrumentation added to RpcRouter dispatch:
- GetRef → record_get_ref_hit / _miss
- GetTag → record_get_tag_hit / _miss
- BlobGet → record_blob_get_bytes (on hit)
- BlobPut → record_blob_put_bytes
- HasChunk → record_has_chunk_hit / _miss
- GetChunk → record_get_chunk_hit + record_blob_get_bytes on hit
/ record_get_chunk_miss
RpcRouter grows Arc<CacheMetrics> unconditionally — every router has
metrics, so a fresh node with no traffic still returns a valid
snapshot with all zeros + started_unix.
Streaming variants (BlobPutStream / BlobGetStream) don't yet track
byte counts — they'd require plumbing the count out of put_stream /
stream_to. Follow-on if it turns out to matter for placement.
## Client helper + CLI
- call_get_metrics(&conn) → Result<MetricsReply>
- claw-cargo peer-metrics [--peer ...] [--peer-addr ...] [--tls-dir ...]
Fetches + pretty-prints:
counter uptime: 42s
GetRef hits/misses: 123 / 45
hit rate: 73.21%
GetTag hits/misses: 8 / 2
hit rate: 80.00%
HasChunk hits/miss: 512 / 88
hit rate: 85.33%
GetChunk hits/miss: 47 / 12
Blob GET bytes: 1.23 GiB
Blob PUT bytes: 3.45 GiB
human_bytes() helper picks GiB / MiB / KiB / B based on magnitude.
Subcommand count now 9: build / prefetch / status / fingerprint /
pin / unpin / list-tags / prewarm / peer-metrics.
## Tests (13 new, all real filesystem / real QUIC — no mocks)
CacheMetrics (6):
- new_starts_all_counters_at_zero_except_timestamp
- recorders_increment_the_right_field (every recorder × 1-2 counts)
- hit_rates_none_when_zero_events (avoids 0/0 NaN)
- hit_rates_compute_correctly (3 hits / 1 miss → 75%)
- snapshot_round_trips_through_json
- snapshots_across_threads_are_consistent_up_to_relaxed_ordering
(8 threads × 1000 increments → exactly 8000)
RPC integration (7):
- phase_5g_method_byte_encoding
- get_metrics_returns_empty_snapshot_before_any_activity
- get_ref_records_hit_and_miss_counters (2 hits + 1 miss)
- get_tag_records_hit_and_miss_counters
- blob_get_and_blob_put_record_byte_counts
- has_chunk_and_get_chunk_record_hit_miss_counters
- **end_to_end_get_metrics_over_real_quic** — seed activity locally,
fire GetRef/GetTag/BlobGet dispatches to move the counters, then
fetch metrics through real QUIC + mTLS and verify each field
including the 0.5 hit rate calculation
238 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/metrics.rs: 268
- cluster/rpc.rs: 818
- cluster/rpc/tests_phase5.rs: 905
- claw_cargo.rs: 894
## What this enables
Fleet-wide visibility into which peer is actually serving traffic:
# From anywhere with connectivity + fleet mTLS
claw-cargo peer-metrics --peer tank
claw-cargo peer-metrics --peer architect
claw-cargo peer-metrics --peer morpheus
Compare hit rates side by side to see which node's cache is warmest.
A placement engine can automate this — poll every 30s, feed the
scheduler.
## Follow-on
- 5h: streaming variant of prewarm (fixed-memory ceiling for many-GB
blobs)
- 5i: metrics also published via gossip so PeerView carries hit rate
without a per-peer GetMetrics roundtrip
- 5j: prometheus /metrics endpoint on the daemon for existing dash
integrations
- 6: FUSE mount for warm-tier git worktrees
|
||
|
Omar Sobh
|
db55903311 |
Phase 5f: claw-cargo prewarm — cross-peer cache copy
The last piece before "Gitea webhook triggers a cache-warm for the
CI runner before its build starts." Adds a `prewarm` subcommand that
copies a tagged cache from one peer (upstream) to another (downstream)
in one shot — same tag, same BlobId, both sides serve it after.
## New subcommand
```
claw-cargo prewarm \
--from-peer tank --from-addr 10.0.0.14:7702 \
--to-peer morpheus --to-addr 10.0.0.15:7702 \
--tls-dir /etc/claw-store/tls \
--pin clawverse:main:latest
```
Flow:
1. Connect to upstream with local mTLS identity
2. `GetTag(tag)` → BlobId; `BlobStat(BlobId)` → size + chunk count
3. `BlobGetStream(BlobId)` → download bytes
4. Connect to downstream (second QUIC endpoint, same identity)
5. `BlobPutStream(bytes)` → returns BlobId; verified equal to upstream's
6. `PutTag(tag → BlobId)` on downstream
Summary output shows tag, blob id, both endpoints, byte count,
download/upload timings, total wall clock.
Assumes upstream + downstream share the same fleet CA (the common
case). Mixed-fleet variant with distinct identities is a follow-on.
## Integrity check
`assigned_id != blob_id` after the downstream upload triggers a
bail — the two BlobIds must match because content is BLAKE3-hashed
end-to-end. If they don't, the wire path corrupted bytes and the
whole prewarm fails loud rather than silently pinning a bad blob.
## Buffered vs streamed
Current implementation buffers the whole blob in memory between
download and upload. Fine for cargo target dirs (~1-5 GB compressed);
would break for a 20 GB blob. A follow-on will pipe upstream → tokio
duplex → downstream to run at fixed memory.
## Tests (1 new, real 2-peer QUIC)
- **`end_to_end_prewarm_copies_tagged_blob_between_two_peers`**
Two full RpcRouters serving in-process (A upstream + C downstream),
each on a distinct port. Seeds A with a blob + tag, then runs the
exact sequence prewarm runs internally: `GetTag → BlobGetStream`
against A, then `BlobPutStream → PutTag` against C. Verifies that
C's blob store returns byte-equal payload and C's tag store now
points at the same BlobId. Proves the composition works.
## Housekeeping
`rpc/tests.rs` hit 1408 lines with the new prewarm test. Phase 5
tests (5b refs + 5d tags + 5e restore + 5f prewarm) split to
`rpc/tests_phase5.rs` via a second `#[path]` module in rpc.rs.
Result:
- rpc/tests.rs: 727 (phase 1-2d tests)
- rpc/tests_phase5.rs: 722 (phase 5 tests)
- rpc.rs: 777
- rpc/client.rs: 589
- claw_cargo.rs: 818
- All under ceiling.
225 tests pass. Pre-existing macOS-only failure unchanged.
## What this enables
The complete CI runner flow now works end-to-end:
```
Primary (e.g. tank):
claw-cargo build # first ever build — MISS, uploads
claw-cargo pin --name clawverse:main:latest
Fleet control plane on PR open:
gitea webhook → shell hook → claw-cargo prewarm \
--from-peer tank --to-peer $RUNNER_LOCAL \
--pin clawverse:main:latest
# runner's local daemon now serves the tag + blob
Runner picks up job:
claw-cargo build --peer 127.0.0.1:7702
# local daemon is warm → prefetch returns HIT
# cargo build runs against restored deps → workspace-crates only
# 50 min → 3 min
```
Every subcommand claw-cargo needs for this pipeline now exists:
build / prefetch / prefetch --pin / status / fingerprint /
pin / unpin / list-tags / prewarm.
## What's next
- 5g: cache hit/miss metrics into gossip so placement engines can
bias runner scheduling toward warm nodes
- 5h: streaming variant of prewarm (tokio duplex) for many-GB blobs
- 6: FUSE mount for warm-tier git worktrees
- 3: full CRDT metadata if the plain-tag model shows conflict problems
|
||
|
Omar Sobh
|
05ba800d01 |
Phase 5e: prefetch --pin <tag>
Small, focused extension to Phase 5c's prefetch: an optional
`--pin <tag-name>` flag that skips fingerprint compute entirely
and resolves the tag → BlobId via GetTag, then downloads that.
## Use case
Restore an old cache into a fresh checkout for regression testing:
$ claw-cargo prefetch --pin clawverse:main:2026-07-12
cache HIT — downloading 3221225472 bytes (768 chunks) to /path/target/dev
...
── claw-cargo prefetch ─────────────────────────────
source: --pin clawverse:main:2026-07-12
blob: 8c2f1a…
downloaded: 3221225472 bytes in 12.3s
restored to: /path/target/dev
────────────────────────────────────────────────────
Or diagnose a "why does this build fail against the pinned cache"
question by prefetching the tagged cache and then running cargo
against your current source. Cargo will detect the mismatched
.fingerprint state and rebuild affected crates — that's the point,
you're diffing behaviour between two known-good cache snapshots.
## Changes
- New PrefetchArgs struct (was reusing PeerArgs) with an optional
`pin: Option<String>` field
- resolve_pin(conn, tag) — internal helper that does
GetTag → BlobStat, returning None on either NotFound
- cmd_prefetch branches at the top: --pin → resolve_pin(); default
→ fingerprint-based peer_lookup()
- Rest of the flow is unchanged: BlobStat → BlobGetStream →
restore_target
- Summary output shows `source: --pin <tag>` instead of
`fingerprint: <hex>` when the pinned path was taken
`peer_lookup` (fingerprint path) and `resolve_pin` (tag path) return
the same `Option<(BlobId, BlobStat)>` shape so the downstream code
is identical.
## Live smoke test
`prefetch --help` now advertises --pin with full description.
Missing-tag path prints "no such tag: <name>" and exits 0
(consistent with the fingerprint-miss path).
## Tests (1 new, real QUIC)
- **`end_to_end_tag_resolve_and_stream_restore_over_real_quic`** —
seeds blob store with a 2 MiB "captured target" payload, publishes
a tag pointing at its BlobId, then runs the exact client
sequence `prefetch --pin <tag>` runs internally:
GetTag → BlobStat → BlobGetStream
Verifies bytes reassemble byte-equal to source. Also covers the
missing-tag path.
The pin flow uses the same underlying calls tested separately in
Phase 5b/5c/5d, so the new test proves the composition works rather
than re-verifying primitives.
224 tests pass. Pre-existing macOS-only failure unchanged.
## What's next
- 5f: Gitea webhook pre-fetch — daemon receives PR-open hints and
warms cache for the predicted fingerprint before CI runner starts
- 6: FUSE mount for warm-tier git worktrees so `~/projects/clawverse`
is transparently fleet-shared
- 3: full CRDT metadata layer (only if real conflicts emerge in the
simple tag model)
|
||
|
Omar Sobh
|
b7904b59a5 |
Phase 5d: named tags + pin/unpin/list-tags CLI
Human-readable pins on top of the raw 32-byte ref layer. Operators
publish `clawverse:main:latest-cache` → BlobId once, then everything
downstream (CI runners, dev laptops) references the tag instead of
passing 64-char hex hashes around.
## Module: cluster/tags.rs (433 lines)
TagStore for string-key → 32-byte-value:
- open(root) — creates layout, safe on existing stores
- put(key, value) / get(key) / delete(key) / contains(key)
- list() — sorted by key
- Atomic writes via tempfile + rename
- Key length capped at MAX_TAG_KEY_BYTES (4 KiB); empty keys rejected
On-disk record: `key_len:u16 (LE) || key_bytes || value:32bytes`.
Filename is `blake3(key)` hex so arbitrary UTF-8 keys land at
deterministic paths without shell escaping.
TagEntry type (public, serde) for list results:
`{ key, value_hex }`. Includes `decode_value() → Result<[u8;32]>`.
## RPC methods
- PutTag (0x0f): payload = encoded record → STREAM_STATUS_OK / err
- GetTag (0x10): payload = key bytes → 32-byte value / NotFound
- DeleteTag (0x11): payload = key bytes → STREAM_STATUS_OK / NotFound
- ListTags (0x12): payload = empty → JSON Vec<TagEntry>
RpcRouter grows optional Arc<TagStore> via `.with_tag_store(store)`.
## Services + config
ClusterServices auto-opens a TagStore at `<blob_store_root>/tags-db`
alongside the ref store. `tag_store` field on ClusterServices, same
enable-with-blob-store semantics.
## claw-cargo new subcommands
- `claw-cargo pin --name clawverse:main:latest`
Compute current fingerprint → look up its BlobId via GetRef →
publish TagStore mapping. Errors cleanly if the fingerprint
hasn't been built yet (nothing to point at).
- `claw-cargo unpin --name clawverse:main:latest`
Delete the tag. Prints "no such tag" if it wasn't set.
- `claw-cargo list-tags`
Print every tag with its 32-byte hex value.
Total subcommand count now 7: build / prefetch / status / fingerprint
/ pin / unpin / list-tags. All share the layered config from Phase 5c.
## Client helpers
- call_put_tag / call_get_tag / call_delete_tag / call_list_tags
- All follow the same error-mapping conventions as prior client helpers
(NotFound → Ok(None) or Ok(false), everything else → Err)
## Housekeeping
rpc.rs was pushing past the 1300-line ceiling with the tag methods
added. Client helpers moved to `cluster/rpc/client.rs` with a
re-export (`pub use client::*;`) so external callers still write
`cluster::rpc::call_*`. Result:
- rpc.rs: 773 (was 1343)
- rpc/client.rs: 593 (new)
- rpc/tests.rs: 1235
- All under ceiling.
## Tests (33 new, all real filesystem / real QUIC — no mocks)
TagStore (16 in cluster/tags.rs):
- open_creates_layout
- get_returns_none_for_missing (+ contains false)
- put_and_get_round_trip
- put_overwrites_prior_value
- delete_returns_true_for_existing_and_false_for_missing
- put_rejects_empty_key
- put_rejects_oversize_key
- list_returns_all_tags_sorted
- list_is_empty_on_fresh_store
- keys_with_slashes_and_colons_round_trip (real-world tag shape)
- encode_and_decode_round_trip (raw wire format)
- decode_rejects_short_record
- decode_rejects_length_mismatch
- decode_rejects_non_utf8_key
- tag_entry_decode_value_round_trip
- tag_entry_decode_value_rejects_bad_hex
RPC dispatch (7 new):
- phase_5d_method_byte_encoding
- tag_rpcs_return_not_configured_without_store
- put_tag_stores_and_get_tag_reads_back
- get_tag_returns_not_found_for_missing
- get_tag_rejects_empty_key
- delete_tag_removes_and_returns_not_found_after
- list_tags_returns_json_sorted
End-to-end over real QUIC (1):
- **end_to_end_pin_lookup_delete_over_real_quic** — publish tag →
look up → list → delete → confirm gone. Full round trip through
the wire layer including JSON deserialization of the list.
Also 8 downstream tests continued passing after the client.rs split
(no test moved, they were untouched).
223 tests pass. Pre-existing macOS-only failure unchanged.
## What this enables
Operator flow:
# Build once on the primary
$ claw-cargo build
→ cache MISS → cargo build (50 min) → capture + upload
→ summary: fingerprint 4a3b…, blob 8c2f…, uploaded 3.2 GiB
# Publish a friendly name
$ claw-cargo pin --name clawverse:main:2026-07-12
pinned: clawverse:main:2026-07-12
fingerprint: 4a3b2c…
blob: 8c2f1a…
# Anyone else can now find it via list-tags
$ claw-cargo list-tags
clawverse:main:2026-07-12 8c2f1a…
clawverse:main:latest 8c2f1a…
# CI runner sees the same fingerprint in its workspace state, hits
# the ref directly via GetRef — the tag is for operator visibility
## Follow-on
- 5e: prefetch --pin <tag> — bypass fingerprint compute, download
the tagged BlobId directly (useful when you want an old cache to
test regression scenarios)
- 5f: gitea webhook pre-fetch — daemon pre-warms cache for known
fingerprints before CI runner starts
- 3: full CRDT metadata layer (namespaces, versioned pointers,
vector clocks) if the plain-tag model turns out to have
real-world conflict scenarios
|
||
|
Omar Sobh
|
57d8358255 |
Phase 5c: claw-cargo UX — config files + status + prefetch
Ships the last-mile ergonomics that make claw-cargo actually usable
day-to-day: layered config files so you don't retype --peer-addr on
every invocation, plus two lightweight subcommands (status +
prefetch) for the "what's in the cache" and "warm my target dir"
workflows respectively.
## Config precedence
Later wins:
1. Built-in defaults (profile=dev, features=[])
2. ~/.claw-cargo/config.toml (per-user defaults)
3. <workspace>/.claw-cargo.toml (per-repo overrides)
4. CLI flags (per-invocation overrides)
Shape:
[peer]
name = "tank"
addr = "10.0.0.14:7702"
tls_dir = "/etc/claw-store/tls"
[build]
profile = "release"
features = ["a", "b"]
## New module: cluster/client_config.rs (496 lines)
- ClientConfig / PeerSection / BuildSection — TOML-serialisable, all
Option<> fields at every layer so partial configs are legal
- ClientConfig::from_toml_str / from_file_or_default (missing file →
default, not error)
- ClientConfig::merge — Option::Some in `other` wins over `self`
- ClientConfig::load_layered(workspace) — user → workspace
- ResolvedClientConfig — final flattened shape after CLI overrides,
with require_peer_name / require_peer_addr / require_tls_dir /
require_peer_bundle helpers that produce a specific error message
instead of "some Option was None"
- write_config_file — for tests + future `claw-cargo config init`
Ships with 11 unit tests including a load_layered test that fakes
HOME + workspace via a tempdir, writes both configs, verifies the
workspace override takes precedence.
## claw-cargo (rewritten to 469 lines)
Four subcommands with layered config:
claw-cargo fingerprint [--profile ...] [--features ...] [--workspace ...]
→ local-only, no network
claw-cargo status <peer args>
→ connect + GetRef + BlobStat, print hit/miss + size, no download
claw-cargo prefetch <peer args>
→ hit → BlobGetStream + restore_target, no cargo
claw-cargo build <peer args> [-- extra cargo args]
→ same as Phase 5b flow, now with layered config for peer args
Refactored internals:
- setup_local / setup_peer — figure out workspace, load config,
resolve CLI overrides, compute fingerprint
- connect_peer — load NodeIdentity, open QUIC connection
- peer_lookup — GetRef → BlobStat, handle the "ref points at a
garbage-collected blob" case as a miss
## Live smoke test
Verified end-to-end on this workspace:
# No config file → built-in defaults
$ claw-cargo fingerprint
profile: dev, features: (none), fingerprint: 8ee4cf…
# Add .claw-cargo.toml with profile=release + features=some-feature
$ claw-cargo fingerprint
profile: release, features: some-feature, fingerprint: 2dfcb1…
# CLI overrides just the profile; features fall through from config
$ claw-cargo fingerprint --profile dev
profile: dev, features: some-feature, fingerprint: 84a756…
# `status` without peer args → clean validation error
$ claw-cargo status
Error: peer.name not set (config file or --peer)
## Tests (11 new, all real filesystem — no mocks)
- from_toml_str_parses_full_config
- from_toml_str_handles_partial_sections (peer.name only)
- from_file_or_default_returns_default_when_missing
- merge_prefers_later_over_earlier (unset fields fall through)
- resolve_applies_cli_overrides_over_layered
- resolve_falls_through_to_default_profile_when_unset_everywhere
- require_peer_bundle_errors_when_incomplete (specific error text)
- validate_peer_errors_on_missing_field
- load_layered_reads_both_files — fake HOME + workspace, verifies
workspace override takes precedence
- user_config_path_uses_home
- write_and_read_round_trip_via_disk (nested dir creation)
199 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (well under 1300-line ceiling):
- cluster/client_config.rs: 496
- claw_cargo.rs: 469
## What's next
The CLI is now usable day-to-day. Realistic next steps:
- 5d: publish cache hit/miss metrics into gossip so the placement
engine can bias runner scheduling toward warm nodes
- 5e: pre-fetch on Gitea webhook — daemon receives a "PR opened for
fingerprint X" hint and warms the local cache before the runner
even starts pulling
- 3: CRDT metadata for human-readable pins (`clawverse:main:latest`
→ fingerprint hex) so operators can pin cache versions without
passing raw hashes around
- 6: FUSE mount so `~/projects/clawverse` on any node is transparently
the tank-hosted canonical warm-tier copy
|
||
 osobh
|
fdc943b2ff |
Merge pull request 'Phase 5b: KV refs + claw-cargo CLI (killer feature, live)' (#11) from phase-5b-claw-cargo into main
Reviewed-on: #11 |
||
|
osobh
|
ced6fc6d78 |
Merge pull request 'Phase 5a: fingerprint + capture + restore for build-artifact cache' (#10) from phase-5a-build-cache into main
Reviewed-on: #10 |
||
|
osobh
|
0455c561ee |
Merge pull request 'Phase 2d: chunk-level RPC (HasChunk/PutChunk/GetChunk/PutManifest)' (#9) from phase-2d-chunk-rpc into main
Reviewed-on: #9 |
||
|
osobh
|
831bbb0de0 |
Merge pull request 'Phase 2c: streaming Blob RPC' (#8) from phase-2c-blob-streaming into main
Reviewed-on: #8 |
||
|
osobh
|
01f21afc00 |
Merge pull request 'Phase 2b: Blob RPC (Stat/Get/Put/LoadManifest)' (#7) from phase-2b-blob-rpc into main
Reviewed-on: #7 |
||
|
osobh
|
4342c2053f |
Merge pull request 'Phase 2: content-addressed blob store' (#6) from phase-2-blob-store into main
Reviewed-on: #6 |
||
|
Omar Sobh
|
2d09b4687c |
Phase 5b: KV refs + claw-cargo CLI (the killer feature, live)
Ships the actual user-facing cargo build cache. Combined with Phase 5a
(fingerprint + capture + restore) + the whole Phase 2 blob substrate,
`claw-cargo build` now runs `cargo build` with a peer-cache lookup:
hit → download+restore, miss → build+capture+upload.
## What ships
### cluster/refs.rs (243 lines)
A dumb 32-byte-key → 32-byte-value directory-backed store. Used to map
fingerprints → BlobIds. Layout mirrors BlobStore:
<root>/
refs/<kk>/<key_hex>.ref — 32 raw bytes
.tmp/ — atomic-rename staging
Public API: RefStore::open / get / put / delete / contains. All writes
atomic via tempfile + rename. Deliberately no versioning or CRDT
semantics — that's Phase 3. Every real cargo-cache lookup is a
single-key-single-value shape.
### New RPC methods
- GetRef (0x0d): payload = 32-byte RefKey; reply = 32 bytes / NotFound
- PutRef (0x0e): payload = 32-byte RefKey || 32-byte RefValue;
reply = STREAM_STATUS_OK / error
### RpcRouter + services
- RpcRouter grows optional Arc<RefStore> via `with_ref_store`
- ClusterServices opens a RefStore alongside the BlobStore when
`blob_store_root` is configured (co-located at `<blob_root>/refs-db`)
- `blob_store_enabled()` / `ref_store_enabled()` introspection
### claw-cargo binary (319 lines)
New bin target `claw-cargo` — thin CLI wrapping the whole stack:
claw-cargo fingerprint --profile release --features "a,b"
→ prints the workspace fingerprint (no network)
claw-cargo build \
--peer <name> --peer-addr <ip:port> --tls-dir <dir> \
--profile release --features "a,b" \
-- --workspace=x --frozen ...
→ 1. compute fingerprint
2. QUIC + mTLS connect to peer
3. GetRef(fingerprint) → BlobId?
HIT: BlobStat → BlobGetStream → restore_target → cargo build
MISS: cargo build → capture_target → BlobPutStream → PutRef
4. Print summary: fingerprint, hit/miss, bytes, cargo elapsed
## Live smoke test
Ran claw-cargo fingerprint on this workspace with three profile/feature
combos — got three distinct 32-byte fingerprints. Same profile+features
on the same workspace state → same fingerprint (Phase 5a's guarantee
carried through the CLI).
## Tests (14 new, all real — no mocks)
Refs store (7):
- open creates layout
- get returns None for missing
- put + get round-trips
- put overwrites prior value
- delete removes ref + reports (false on second delete)
- distinct keys produce distinct on-disk files (bucket fan-out proof)
- rejects_wrong_length_on_disk (corruption detection)
RPC (7):
- phase_5b_method_byte_encoding
- get_ref_returns_not_found_for_missing
- put_ref_stores_and_get_ref_reads_back
- put_ref_rejects_wrong_length_payload
- get_ref_rejects_wrong_length_payload
- ref_rpcs_return_not_configured_without_store
- end_to_end_put_ref_get_ref_over_real_quic — full 2-node QUIC + mTLS
round trip proving PutRef/GetRef work at the wire level
188 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/refs.rs: 243
- cluster/rpc.rs: 1169
- cluster/rpc/tests.rs: 1073
- cluster/services.rs: 565
- claw_cargo.rs: 319
## Where this leaves us
The distributed FS + cargo cache is functionally complete for the
happy path:
Node A builds clawverse for the first time
→ cargo build (50 min cold)
→ capture_target (a few seconds)
→ push to node B via BlobPutStream (network-bound)
→ PutRef(fingerprint → BlobId)
Node B on the same workspace state runs `claw-cargo build …`
→ compute_fingerprint (ms)
→ GetRef → hit
→ BlobGetStream (network-bound)
→ restore_target (a few seconds)
→ cargo build → sees valid deps/.fingerprint, builds only
workspace crates (~3 min instead of 50)
Same workspace state on a third machine? Same fingerprint → same
cache hit. That's the whole design.
## Follow-on
- Phase 5c: pre-fetch on Gitea webhook so CI runners never wait
- Phase 5d: metric ticker publishes cache hit rate into gossip so
the placement engine can bias runner scheduling toward warm nodes
- Phase 3: CRDT metadata for human-readable pins on top of raw
32-byte refs (`clawverse:main:latest-cache` → fingerprint hex)
- Phase 6+: FUSE mount for the warm-tier git worktrees
|
||
|
Omar Sobh
|
242ef527b4 |
Phase 5a: fingerprint + capture + restore for build-artifact cache
The substrate for the killer feature. Given a cargo workspace, compute
a deterministic 32-byte BLAKE3 fingerprint over the inputs that
determine what artifacts should be produced, then bundle the portable
subset of `target/<profile>/` into a zstd-compressed tarball ready to
hand to BlobStore.
## Module: cluster/build_cache.rs (661 lines)
Types:
- FingerprintInputs { cargo_lock, rustc_version_verbose, cargo_config,
rust_toolchain, profile, features, rustflags, target_triple }
- Fingerprint(32 bytes) — parallel shape to BlobId
Public API:
- FingerprintInputs::collect(workspace, profile, features) — reads
Cargo.lock, shells out to `rustc --version --verbose`, reads
optional config files, extracts host triple, sorts+dedups features
- FingerprintInputs::compute() → Fingerprint — domain-separated
BLAKE3 with per-field labels + null separators so field boundaries
can't collide
- capture_target(target_dir) → zstd-tarball bytes
- restore_target(bytes, target_dir) → unpacks
- capture_workspace(workspace, profile) — resolves target dir
- compute_workspace_fingerprint(workspace, profile, features) —
returns (inputs, fingerprint)
Captured: deps/, .fingerprint/, build/, examples/, plus small
top-level files (.cargo-lock, .rustc_info.json, CACHEDIR.TAG).
Explicitly NOT captured: incremental/ (per-machine, not portable —
tied to absolute paths + rustc state; restoring on another host
silently corrupts the build).
Determinism guarantees:
- HeaderMode::Deterministic on the tar builder — identical trees
produce byte-identical tarballs (proven by
`capture_yields_identical_bytes_for_identical_input`)
- follow_symlinks(false) — symlinks archived as symlinks, not their
targets, so the fingerprint doesn't drift with symlink destinations
- Features sorted + deduped so `["b","a"]` and `["a","b"]` hash the same
- Missing optional files treated as empty strings so `absent ==
empty` (add-then-remove doesn't churn the hash)
Deps added: tar 0.4, zstd 0.13.
## Tests (18 new, all real filesystem — no mocks)
Fingerprint (7):
- fingerprint_is_deterministic
- fingerprint_changes_when_cargo_lock_changes
- fingerprint_changes_when_profile_changes
- fingerprint_changes_when_features_change
- fingerprint_is_feature_order_independent (proves sort semantics)
- fingerprint_domain_separation_prevents_field_collision — swap
content between two string fields; naive concat hasher would
collide, ours doesn't
- fingerprint_hex_length_and_stability
Input collection (4):
- read_optional_returns_empty_for_missing
- read_optional_returns_content_for_existing
- collect_errors_when_cargo_lock_absent
- collect_reads_cargo_lock_and_computes — actually shells out to
`rustc --version --verbose`, verifies triple extraction
Capture/restore (5):
- capture_target_errors_when_dir_missing
- capture_and_restore_round_trip_preserves_files — full 6-file
tree including a captured example + an intentionally-excluded
incremental/ dir; verifies incremental/ is absent after restore
- capture_yields_identical_bytes_for_identical_input — byte-equal
tarballs across two identical source trees
- capture_skips_top_level_files_not_in_allowlist
- capture_workspace_resolves_profile_dir
End-to-end (1):
- **end_to_end_fingerprint_capture_blob_restore** — the full
workspace → fingerprint → capture → BlobStore::put_bytes →
BlobStore::get_bytes → restore_target loop. Proves the whole
round-trip lands byte-equal for both captured file trees, and
the BlobId is deterministic across runs. This is the primitive
the cargo-cache CLI sits on top of.
174 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (well under 1300-line ceiling):
- cluster/build_cache.rs: 661
- cluster.rs (submodule declarations): 279
## What this unlocks
Phase 5a is the substrate. The CLI wrapper (`claw-cargo build`) is
Phase 5b — thin glue that:
1. Runs compute_workspace_fingerprint()
2. Asks the peer BlobStat(fingerprint_as_blob_id)
3. Hit → BlobGetStream + restore_target + cargo build (just the
workspace's own crates, ~seconds)
4. Miss → cargo build (full), then capture_target +
push_blob_missing_chunks + PutManifest
Every piece of infrastructure that Phase 5b needs — content-addressed
blob store, streaming and partial-chunk RPC, mTLS transport, gossip-
driven peer discovery — is already merged. Phase 5b is CLI polish,
not new distributed-systems machinery.
## Follow-on
- Phase 5b: claw-cargo CLI wrapper (small — 200-400 lines)
- Phase 5c: pre-fetch on Gitea webhook (workflow triggers → daemon
pre-warms the fingerprint on the target runner)
- Phase 3 remains a parallel track — human-readable namespace layer
on top of raw fingerprints so operators can pin
`clawverse:main:latest-cache` instead of a hex string
|
||
|
Omar Sobh
|
2e984b924d |
Phase 2d: chunk-level RPC (HasChunk / PutChunk / GetChunk / PutManifest)
Unlocks partial-sync replication — a peer that already has some
chunks of a blob (typical when two nodes share overlapping cargo
build caches) only receives the chunks it's missing.
## New methods
| Byte | Method | Payload | Reply |
|---|---|---|---|
| 0x09 | HasChunk | 32-byte ChunkHash | STREAM_STATUS_OK / NotFound |
| 0x0a | PutChunk | ChunkHash \|\| bytes | STREAM_STATUS_OK / error |
| 0x0b | GetChunk | ChunkHash | STREAM_STATUS_OK \|\| bytes / NotFound |
| 0x0c | PutManifest | JSON BlobManifest | JSON PutManifestReply |
`PutManifestReply { blob_id, missing: Vec<ChunkHash> }`: empty
`missing` means the manifest was written; non-empty tells the
client which chunks to upload before retrying.
Server verifies bytes hash to claimed hash on PutChunk; a
mismatch surfaces as InvalidRequest and the store is untouched.
## BlobStore additions
- `has_chunk(&ChunkHash) → bool`
- `read_chunk(&ChunkHash) → Option<Vec<u8>>` — verifies hash on read
- `put_chunk(&ChunkHash, bytes) → Result<()>` — verifies bytes-vs-hash
- `put_manifest_verified(&manifest) → Result<Vec<ChunkHash>>` —
returns the list of chunks missing on disk (empty on success)
- `chunk_path` promoted to `pub` for advanced callers
## Client helpers
- `call_has_chunk` / `call_put_chunk` / `call_get_chunk` / `call_put_manifest`
- `push_blob_missing_chunks(conn, local_store, blob_id) →
Result<(uploaded, total)>` — high-level partial-sync helper
`push_blob_missing_chunks` loads the local manifest, calls HasChunk
for each chunk, uploads only the missing ones via PutChunk, then
commits via PutManifest. On a fully-overlapping cache the uploaded
count is 0 and only the ~small manifest crosses the wire.
## Tests (17 new, all real filesystem + real QUIC — no mocks)
Blob store (6):
- has_chunk_is_false_before_put_and_true_after
- read_chunk_returns_bytes_and_none_when_missing
- put_chunk_rejects_hash_mismatch (nothing written)
- read_chunk_detects_corruption (bit-flip → mismatch error)
- put_manifest_verified_reports_missing_chunks
- put_manifest_verified_writes_when_all_chunks_present
Router dispatch (7):
- phase_2d_method_byte_encoding
- method_reports_streaming_variants — extended for 4 new methods
- has_chunk_returns_ok_for_present_and_not_found_for_missing
- put_chunk_stores_and_returns_status_ok
- put_chunk_rejects_hash_mismatch_over_wire
- get_chunk_returns_content_prefixed_with_status_ok
- get_chunk_returns_not_found_for_missing
- put_manifest_reports_missing_chunks_when_incomplete
- put_manifest_writes_when_chunks_present
- chunk_rpcs_return_not_configured_without_store
End-to-end (2):
- **end_to_end_push_blob_missing_chunks_replicates_only_needed_bytes**:
Peer A pre-seeded with chunk 0 of a 2-chunk (8 MiB) blob;
`push_blob_missing_chunks` reports `(uploaded=1, total=2)`,
only chunk 1 crosses the wire, A's store then contains the
complete blob and `get_bytes` returns byte-equal content.
- **call_get_chunk_verifies_returned_hash**: real 2-node fetch,
client hashes received bytes and compares to requested hash.
156 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 1053
- cluster/rpc/tests.rs: 940
- cluster/blob.rs: 1186
## Where this fits
With Phase 2c whole-blob streaming + Phase 2d partial-chunk sync,
the storage substrate is now genuinely bandwidth-efficient in the
distributed setting:
- First-ever push of a blob: `push_blob_missing_chunks` uploads
everything (all chunks missing).
- Second push of a similar blob (95% chunk overlap with prior
contents): only the 5% new chunks cross the wire, plus a tiny
manifest.
- Whole-blob download: BlobGetStream, bounded by network bandwidth.
## Follow-on
- Phase 3: CRDT metadata for human-readable namespaces on top of
content hashes.
- Phase 5: the killer feature. Fingerprint cargo target dir → tar
→ hash → PutBlobStream (or push_blob_missing_chunks if a similar
build already lives on the peer). Same fingerprint on the next
node → BlobGetStream. This is the whole cargo-cache design in
one line and it now sits on a substrate that handles all the
hard cases (dedup, verification, resumability, partial sync).
|
||
|
Omar Sobh
|
1fd1027da4 |
Phase 2c: streaming Blob RPC (BlobPutStream / BlobGetStream)
Removes the 16 MiB message cap for blob transfers. The bounded Blob* methods from Phase 2b still exist; the streaming variants let a peer push or pull a many-GB blob without either side holding it in memory. ## Wire format Streaming methods use a slightly different reply shape so the client can route on the first byte alone: Reply : status:u8 || payload:bytes... Where `status` is either `STREAM_STATUS_OK` (0x00, content follows) or a single-byte ErrorCode. `serve_connection` now peeks at the method tag byte via read_exact and hands streaming methods the raw send/recv streams; bounded methods still use the old read_to_end path. ## Method additions - BlobPutStream (0x07): client streams bytes → server pipes into BlobStore::put_stream → reply is 0x00 || 32-byte BlobId - BlobGetStream (0x08): client sends 32-byte BlobId → server verifies existence, writes 0x00 status, then streams chunks from disk into the send stream Method::is_streaming() introspection so callers can decide which wire variant to use. ## BlobStore additions - put_stream<R: AsyncRead + Unpin>(reader) -> BlobId Memory ceiling: one CHUNK_SIZE (4 MiB) buffer regardless of blob size. Handles short-reads correctly (loops until CHUNK_SIZE bytes are available or EOF), including the empty-reader case (produces the empty-blob BlobId, zero chunks). - stream_to<W: AsyncWrite + Unpin>(id, writer) -> bool Ok(false) on NotFound (writer untouched). Verifies each chunk hash before emitting; corruption halts mid-stream with Err. ## Client helpers - call_blob_put_stream(conn, reader) -> Result<BlobId> Uses tokio::io::copy directly onto quinn's SendStream. - call_blob_get_stream(conn, id, writer) -> Result<bool> Ok(false) on NotFound; other errors surface as Err. ## Tests (11 new, all real — no mocks) Blob store (6): - put_stream_produces_same_hash_as_put_bytes (3-chunk blob via Cursor) - put_stream_handles_empty_reader (produces empty-blob BlobId) - put_stream_handles_short_reads (custom Trickle reader that only serves 100 bytes per read call — must still assemble full chunks) - stream_to_writes_full_blob (2-chunk write to Vec<u8>) - stream_to_returns_false_when_missing (writer untouched) - stream_to_detects_chunk_corruption (bit-flip a chunk → Err with "chunk hash mismatch") RPC (5): - method_reports_streaming_variants - end_to_end_stream_put_and_get_over_real_quic — 12 MiB + 777 bytes → 4 chunks, real 2-node QUIC + mTLS + stream round-trip - stream_get_returns_false_for_missing_blob - stream_methods_return_not_configured_without_store - stream_put_deduplicates_with_prior_put_bytes — verify streaming put produces the same BlobId as a prior bounded put on identical content, and the manifest chunk count didn't fork ## Housekeeping rpc.rs was tipping over the 1300-line ceiling with the streaming handlers + helpers + tests. Tests split into `cluster/rpc/tests.rs` via `#[path = "rpc/tests.rs"] mod tests;`. Result: - rpc.rs: 748 lines - rpc/tests.rs: 694 lines - blob.rs: 1002 lines - All under ceiling. 139 tests pass. Pre-existing macOS-only failure unchanged. ## What's next - Phase 2d: chunk-level RPC (BlobPutChunk / BlobGetChunk) so a receiver can `LoadManifest` then request only the chunks it's missing — big bandwidth win on partially-overlapping caches. - Phase 3: CRDT metadata for human-readable namespaces on top of content hashes. - Phase 5: the killer feature — fingerprint the cargo target dir, BlobPutStream it, next node BlobGetStream by the same fingerprint. Now buildable directly on Phase 2c since target dirs run 100 MB to a few GB and the previous 16 MiB cap would have blocked us. |
||
|
Omar Sobh
|
f43b34ad15 |
Phase 2b: Blob RPC (Stat/Get/Put/LoadManifest)
Wires the Phase 2 content-addressed blob store onto the network via
four new methods on the existing RpcRouter. Combined with the mTLS +
gossip stack from Phase 1c-1e, peers can now exchange content-addressed
blobs over real QUIC. This is the substrate the fingerprint-keyed cargo
cache (Phase 5) sits on directly.
New methods:
- BlobStat (0x03): payload = 32-byte BlobId, reply = JSON BlobStat
- BlobGet (0x04): payload = 32-byte BlobId, reply = raw bytes
- BlobPut (0x05): payload = raw bytes, reply = 32-byte BlobId
- BlobLoadManifest (0x06): payload = 32-byte BlobId, reply = JSON manifest
New error codes:
- NotFound (0xf3) — the requested BlobId isn't in the local store
- InvalidRequest (0xf4) — e.g. non-32-byte payload for a hash-keyed method
- NotConfigured (0xf5) — Blob* called on a router without an attached store
Wire-format bump: MAX_MESSAGE_BYTES 16 KiB → 16 MiB so a single 4 MiB
chunk (plus JSON overhead) fits comfortably. Anything above 16 MiB
needs the streaming variants coming in Phase 2c.
Client helpers:
- call_blob_stat / call_blob_get / call_blob_put / call_blob_load_manifest
- All map ErrorCode::NotFound to Ok(None), other codes to Err.
- call_blob_put verifies the peer-assigned BlobId matches local blake3
hash of the payload — corruption or protocol drift surfaces
immediately instead of silently accepting a mismatched receipt.
Router changes:
- RpcRouter grows an Option<Arc<BlobStore>> via with_blob_store(store).
- handle() split into handle_outcome() → HandlerOutcome enum
{Reply(bytes) | Error(ErrorCode)} for cleaner control flow across
the growing method set.
Services / daemon:
- ClusterServices::start gains blob_store_root: Option<PathBuf>.
- ClusterConfig gains blob_store_root: Option<PathBuf>.
- daemon.rs reads it from cluster_cfg + passes through.
- New ClusterServices::blob_store_enabled() introspection.
Tests (11 new, all real — no mocks):
Router (7 new):
- method_round_trips_byte_encoding — updated for 6 methods
- error_code_describe_covers_all_variants — updated for 6 codes
- decode_error_covers_all_known_codes
- blob_rpcs_return_not_configured_without_store — all four Blob*
methods return NotConfigured when the router lacks a store
- blob_stat_returns_not_found_for_missing
- blob_stat_returns_json_for_existing
- blob_stat_returns_invalid_request_for_bad_length
- blob_get_returns_content_bytes
- blob_put_stores_bytes_and_returns_hash — verifies BlobId matches
independent local hash
- blob_load_manifest_returns_json_for_existing (2-chunk case)
- blob_load_manifest_returns_not_found_for_missing
End-to-end over real QUIC (2 new):
- end_to_end_blob_put_stat_get_over_real_quic — full 4-method loop
(Put → Stat → Get → LoadManifest) + NotFound path
- end_to_end_multi_chunk_blob_over_real_quic — 6 MiB blob → 2 chunks,
proves MAX_MESSAGE_BYTES bump took effect
Services (1 new):
- services_with_blob_store_serves_blob_rpc_end_to_end — cut CA, sign
leaves, config includes blob_store_root, start ClusterServices,
dial from B over persisted mTLS, put + get through the router,
then independently verify bytes landed on A's on-disk store
Also fixed a parallel-test port collision: services `next_port()`
now increments by 2 so `port + 1` (the RPC bind) is reserved
alongside `port` (the gossip bind).
128 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 1006
- cluster/services.rs: 535
- cluster/blob.rs: 802
- config.rs: 511
- daemon.rs: 265
Follow-on:
- 2c: streaming variants (AsyncRead/AsyncWrite) so a many-GB blob
transfers without holding it in memory
- 2d: chunk-level RPC (BlobPutChunk / BlobGetChunk) so a receiver
can request only chunks it's missing after LoadManifest
- Phase 5 (the killer feature) can now build on Phase 2b directly —
fingerprint the target dir, PutBlob the compressed tarball, and
next node calls GetBlob keyed by the same fingerprint hash.
|
||
|
Omar Sobh
|
eab10005fd |
Phase 2: content-addressed blob store
The storage substrate everything after Phase 1 sits on. Every blob is
identified by its BLAKE3 whole-content hash (BlobId); on disk it lives
as an ordered sequence of BLAKE3-hashed 4 MB chunks, so blobs that
share a prefix (two cargo target dirs with 95% of the same deps) share
storage at chunk granularity with no special detection logic.
New: cluster/blob.rs (802 lines).
Types:
- BlobId — 32-byte BLAKE3 output, hex-serialised (serde ↔ string)
- ChunkHash — same shape as BlobId but a distinct type so blob and
chunk lookups can't accidentally swap
- BlobStat — { total_size, chunk_count }
- BlobManifest — { blob_id, total_size, chunks: Vec<ChunkHash> },
public because Phase 2b RPC serves it directly so a receiver can
request only the chunks it's missing
- GcReport — { chunks_scanned, chunks_removed, bytes_reclaimed }
- BlobStore — root-directory-based store
Public API:
- BlobStore::open(root)
- put_bytes(&[u8]) → BlobId
- get_bytes(&BlobId) → Option<Vec<u8>> (verifies hash + size on read)
- contains(&BlobId) → bool
- stat(&BlobId) → Option<BlobStat>
- load_manifest(&BlobId) → Option<BlobManifest>
- delete_manifest(&BlobId) → bool (chunks stay; orphan by GC)
- gc_orphan_chunks() → GcReport
On-disk layout:
<root>/
blobs/<bb>/<blob_hash>.manifest.json
chunks/<cc>/<chunk_hash>
.tmp/
Two-char bucket prefixes cap fan-out at 256 entries per level — safe
on a warm-tier ZFS dataset with tens of thousands of blobs.
Every write is atomic (tmp file + rename on same filesystem).
Every chunk write is a no-op if the file already exists — same
content across two put()s stores exactly one physical copy.
Correctness:
- Reads verify each chunk against its hash + recompute the whole-blob
hash before returning; a bit-flipped chunk raises "chunk hash
mismatch" instead of silently corrupting the answer.
- delete_manifest is the only deletion primitive; chunks are only
ever removed by gc_orphan_chunks after a full manifest scan proves
they're unreferenced.
Dep: blake3 = "1".
Tests (21 new, all real filesystem, no mocks):
- BlobId/ChunkHash hex round-trip + serde JSON
- BlobId::from_hex rejects wrong-length + non-hex input
- open creates blobs/, chunks/, .tmp/
- put + get round trip: small, empty, 10 MB (3 chunks)
- put is deterministic (same bytes → same BlobId every time)
- put is idempotent (writing twice → exactly one manifest file)
- Different content → different BlobId
- shared_chunks_are_stored_only_once: two blobs sharing a 4 MB prefix
produce exactly 3 chunk files, not 4
- get_returns_none_when_missing / contains false / stat None
- delete_manifest keeps chunks (proven by counting chunk files)
- delete_manifest on missing returns false
- gc_reclaims_orphan_chunks_but_keeps_referenced: put 2 blobs, delete
one manifest, GC removes exactly the orphaned chunk, keeps
live A readable
- gc_on_empty_store_reports_zero
- corrupted_chunk_detected_on_read: rewrite a chunk with garbage →
get_bytes errors with "chunk hash mismatch"
- load_manifest round-trips the chunk list
116 tests pass. Pre-existing macOS-only failure unchanged.
File size: cluster/blob.rs = 802 lines (ceiling 1300).
Follow-on Phase 2 cuts:
- 2b: RPC methods BlobStat / BlobGet / BlobPut, wired into RpcRouter
and served over the QUIC transport built in Phase 1c-1e.
- 2c: streaming put/get (AsyncRead / AsyncWrite variants) for
many-GB build artifacts.
Phase 3 (metadata + CRDTs) can start independently — this store is the
substrate the fingerprint cache in Phase 5 layers onto.
|
||
|
osobh
|
91ab43000e |
Merge pull request 'Phase 1e: daemon-integrated cluster services + PeerStatus RPC' (#5) from phase-1e-daemon-integration into main
Reviewed-on: #5 |
||
|
osobh
|
783c64ad5c |
Merge pull request 'Phase 1d: persistent identity + fleet-ca CLI' (#4) from phase-1d-persistent-identity into main
Reviewed-on: #4 |
||
|
Omar Sobh
|
2ae079fbd4 |
Phase 1e: daemon-integrated cluster services + PeerStatus RPC
Ties Phase 1a-1d together into a live subsystem the daemon actually
runs. When `[cluster]` is present in config, `claw-store daemon` now:
1. Starts chitchat gossip via ClusterServices::start.
2. Publishes hot.max_gb immediately and re-measures the hot dir
every 30s, updating `clawstor.hot.used`.
3. If `[cluster.tls]` is also configured, loads NodeIdentity from
PEM files and binds a QUIC RPC server that accepts + dispatches
incoming Ping / PeerStatus requests via RpcRouter.
New modules:
- cluster/rpc.rs (510 lines):
- Method enum (Ping=0x01, PeerStatus=0x02)
- ErrorCode enum (EmptyRequest / UnknownMethod / HandlerFailure)
- PeerStatusReply { local_name, local_zone, peers: Vec<PeerView> }
- RpcRouter — dispatch state (holds Arc<ClusterGossip> + local
name/zone)
- rpc_call / call_ping / call_peer_status — client helpers
- serve_connection — server accept-bidi loop
- Wire format: `method:u8 || payload:bytes` → `reply:bytes` or
single-byte ErrorCode
- cluster/services.rs (418 lines):
- ClusterServices { gossip, accept_task, metric_task }
- start(cluster_cfg, name, hot_dir, hot_max_bytes) → bootstraps
everything above
- shutdown() aborts background tasks cleanly
- Recursive dir-walker (spawn_blocking) for hot-tier metric
PeerView now derives Serialize/Deserialize so it round-trips through
JSON over the RPC.
daemon.rs integration (~35 lines added):
- Bootstraps ClusterServices before entering the select loop
- Held for daemon lifetime
- Shutdown on SIGTERM
- Gossip-less config still runs standalone (backwards compat)
CLI: `cluster-peer-status --peer <name> --rpc-addr <addr>
--tls-dir <dir>`
Loads a persistent identity, dials the peer, calls PeerStatus,
prints the peer's local view as a table.
Tests (15 new, all real — no mocks, real UDP + TLS + JSON round trip):
RPC (9):
- method_round_trips_byte_encoding
- dispatch_returns_pong_for_ping
- dispatch_returns_json_for_peer_status
- dispatch_returns_empty_request_error
- dispatch_returns_unknown_method_error
- rpc_call_rejects_oversize_payload (with real quinn connection)
- end_to_end_ping_and_peer_status_over_real_quic — full 2-node quinn
with mTLS + both RPCs
- peer_status_reflects_peer_gossip_state — 2 gossip instances converge,
RPC caller from a third identity sees the converged view
- error_code_describe_covers_all_variants
Services (6):
- dir_bytes_sync_returns_zero_for_missing_path
- dir_bytes_sync_sums_recursive_file_sizes (3-level nesting)
- services_start_without_tls_leaves_rpc_disabled
- services_start_with_tls_serves_rpc_end_to_end — full stack: cut CA on
disk, sign leaves, start ClusterServices for A, dial from B via
persisted mTLS, run ping + PeerStatus over the wire
- services_publish_hot_used_metric_periodically
- services_gossip_sees_peer_after_convergence
95 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 510
- cluster/services.rs: 418
- cluster/gossip.rs: 579
- cluster/transport.rs: 916
- daemon.rs: 262
- main.rs: 749
Phase 1 done end-to-end. `claw-store daemon` now boots a real distributed
cluster stack when config asks for one; peers can call each other's
PeerStatus RPC and see live membership. Next: Phase 2 (content-addressed
blob store) can hook new RPC methods into the same RpcRouter with a
one-line dispatch arm.
|
||
|
Omar Sobh
|
916add37df |
Phase 1d: persistent NodeIdentity + FleetCa + fleet-ca CLI
Closes out Phase 1. A production operator can now cut a fleet CA,
sign per-node leaves, drop the resulting PEMs at
/etc/claw-store/tls/, point [cluster.tls] at them, and the daemon
loads real mTLS material on startup — no more ephemeral in-process
CA hack.
New public API in cluster::transport:
- FleetCa::generate(cn) — new self-signed root CA
- FleetCa::save(dir) / FleetCa::load(dir) — round-trip PEM
- FleetCa::sign_leaf(name) — mint an in-memory NodeIdentity
- FleetCa::sign_leaf_to_pem(name, out_dir) — write ca.crt + node.crt +
node.key (node.key at 0o600 on Unix)
- NodeIdentity::from_pem_files(ca, cert, key) — production load path
- NodeIdentity::from_pem_dir(dir) — canonical filename layout
- NodeIdentity::from_cluster_config(cfg) — pick up [cluster.tls] paths
Manual Debug for FleetCa redacts the private key.
Config extension:
- [cluster.tls] ca_cert / node_cert / node_key (all PathBuf).
- Optional at the top level; callers that need mTLS surface a clear
error when it's absent.
Deps:
- rcgen features += "x509-parser" (for FleetCa::load's from_ca_cert_pem).
- rustls-pemfile 2 (parse PEM back into DER for rustls).
CLI (new commands; short-circuit config load so they run on fresh
boxes without /etc/claw-store/config.toml):
- fleet-ca-init --dir <dir> [--cn <name>]
generates ca.crt + ca.key (both 0o600).
- fleet-ca-sign --ca-dir <dir> --node <name> --out-dir <dir>
writes ca.crt + node.crt + node.key (node.key at 0o600).
- cluster-ping now accepts --tls-dir <dir> to load persistent
NodeIdentity from disk (produced by fleet-ca-sign).
Tests (8 new, all real — no mocks, real filesystem, real TLS handshake):
- fleet_ca_rejects_empty_common_name
- fleet_ca_save_and_load_round_trip_preserves_signing (asserts 0o600
on ca.key)
- fleet_ca_load_errors_when_files_missing
- sign_leaf_to_pem_writes_all_three_files_with_correct_permissions
(asserts 0o600 on node.key)
- persistent_identity_round_trips_through_disk_and_pings — end-to-end:
cut CA on disk, reload it, sign two leaves via sign_leaf_to_pem,
reload them via from_pem_dir, run real QUIC ping/pong. This is the
operator flow.
- node_identity_from_cluster_config_errors_without_tls_section
- node_identity_from_cluster_config_loads_pem_paths
- from_pem_files_errors_on_missing_ca_file
80 tests pass. Pre-existing macOS-only hot test unchanged.
Also verified live CLI smoke test:
fleet-ca-init → ca.crt + ca.key at 0o600
fleet-ca-sign → ca.crt + node.crt + node.key at 0o600
Files parse as valid X.509.
File sizes (all under 1300-line ceiling):
- cluster/transport.rs: 916
- cluster/gossip.rs: 576
- cluster.rs: 275
- config.rs: 503
- main.rs: 662
Phase 1 complete. Next up:
- Phase 1e (daemon integration): gossip + QUIC RPC server wired into
claw-store daemon; hot-tier metrics periodically pushed; a real
PeerStatus RPC alongside ping.
- Phase 2: content-addressed blob store (BLAKE3 chunking, put/get).
|
||
|
osobh
|
776f28e3a3 |
Merge pull request 'Phase 1c: QUIC transport with fleet-CA mTLS' (#3) from phase-1c-quic-transport into main
Reviewed-on: #3 |
||
|
osobh
|
e1d0e9cb21 |
Merge pull request 'Phase 1b: chitchat SWIM gossip + cluster-status' (#2) from phase-1b-chitchat-gossip into main
Reviewed-on: #2 |
||
|
osobh
|
c0e1ee5a85 |
Merge pull request 'Phase 1a: cluster module + LAN-first peer probe' (#1) from phase-1a-cluster-peer-probe into main
Reviewed-on: #1 |
||
|
Omar Sobh
|
4d93f955c5 |
Phase 1c: QUIC transport with fleet-CA mTLS + cluster-ping
Wraps quinn 0.11 in cluster/transport.rs with rustls 0.23 + rcgen 0.13 for identity management. Every peer connection is mTLS: both sides must present certs signed by the shared fleet CA, and rustls verifies the peer's cert SAN matches the requested server name. Public API on `cluster::transport`: - `NodeIdentity` (cert chain + private key + trusted CA) - `NodeIdentity::generate_test_pair(a, b)` — ephemeral CA + two signed leaves; shape matches the production PEM-file loader (Phase 1d) - `QuicServer::bind(addr, identity)` — bind mTLS-enforced listener - `QuicServer::accept()` — accept one connection (Option<Result<_>>) - `QuicClient::new(local_addr, identity)` — build client endpoint - `QuicClient::connect(peer_addr, expected_name)` — outbound with SAN check - `ping(&conn, payload)` — bidi stream RPC; server echoes as `pong:<payload>` - `ping_handler_loop(conn)` — server-side accept/echo forever - `CLAWSTOR_RPC_ALPN` constant, single ALPN "clawstor-rpc/1" Config extension: - `bind_rpc_lan`, `bind_rpc_tailscale` on `ClusterConfig` (both optional). - Defaults: gossip port + 1 (so gossip UDP and QUIC UDP don't collide). - Helper: `ClusterConfig::rpc_lan()`, `rpc_tailscale()`, `PeerEntry::rpc_lan()`, `rpc_tailscale()`. Gossip.rs now advertises the RPC address, not the gossip address, on the well-known `clawstor.rpc.lan` / `clawstor.rpc.tailscale` keys. CLI: - `cluster-ping --name <me> --peer <you> --rpc-addr <addr> [--payload X]` Runs a full mTLS handshake and single ping. For dev/loopback use today (both sides need to share a CA); persistent-identity ping lands in Phase 1d. Tests (6 new, real UDP + TLS handshake, no mocks): - generate_test_pair produces two distinct leaves that share a CA - ping_pong_between_two_mtls_peers: real 2-node QUIC round trip with full mTLS chain verification, `open_bi()`/`accept_bi()`, byte-exact response check - client_rejects_peer_with_wrong_ca: TLS chain verification failure when the server presents a cert signed by a different CA - client_rejects_wrong_server_name: SAN mismatch is enforced - server_binds_wildcard_and_reports_concrete_local_addr: port 0 → real - ping_rejects_oversize_payload: MAX_MESSAGE_BYTES cap enforced client-side Fixed-port allocator range 42000+ so transport tests don't conflict with gossip tests (41000+). 72 tests pass. Pre-existing `hot::test_project_target_size_bytes` macOS-only failure unchanged. File sizes (all under 1300-line ceiling): - cluster/transport.rs: 485 - cluster/gossip.rs: 570 - cluster.rs: 275 - config.rs: 480 - main.rs: 564 Follow-on Phase 1 cut (1d): - Load NodeIdentity from persistent PEM files at /etc/claw-store/tls/ - Fleet CA bootstrap ceremony (rcgen → write CA cert; per-node leaf CSR) - Daemon-level RPC server that runs alongside gossip + serves real operations (blob get/put, metadata sync) - cluster-status reads from live daemon via API instead of standalone |
||
|
Omar Sobh
|
5283ab8655 |
Phase 1b: chitchat SWIM gossip layer + cluster-status CLI
Wraps chitchat 0.11 in cluster/gossip.rs. Each node publishes its
zone, RPC endpoints, hot-tier usage, warm-tier project list, and
uptime under well-known kv keys; peers propagate this via gossip and
are classified alive/dead by phi-accrual failure detection.
Bootstraps from static [[cluster.peers]] seed nodes in the config.
Membership extends dynamically as nodes join or leave.
Public API on `ClusterGossip`:
- bootstrap(cluster_cfg, local_name) — start UDP gossip service
- set(key, value), set_hot_used, set_hot_max, set_warm_projects
- peers() — all known peers with typed PeerView + liveness
- peer(name) — one peer by advertised name
- peers_in_zone(zone) — live peers filtered by zone
- shutdown() — abort gossip task
CLI:
- `claw-store cluster-status --name <me> [--wait-secs N]`
bootstraps gossip, waits for convergence, prints a peer table
with zone, alive/dead, RPC LAN/Tailscale addresses, hot usage.
Tests (7 new):
- bootstrap_publishes_our_own_state — self-ID + advertised address
- bootstrap_fails_when_local_name_empty
- bootstrap_fails_when_no_bind_address — cluster.validate error chain
- two_node_cluster_converges_and_shares_state — REAL two-node
in-process UDP gossip. Publish state on A, wait for phi-accrual
liveness on B, verify all fields propagated (zone, rpc_lan, hot
bytes, warm_projects list, fill ratio, zone filtering).
- peers_excludes_self — solo cluster reports zero peers
- peer_view_hot_fill_ratio_handles_missing_or_zero
No mocks. Real UDP transport, real chitchat runtime, real gossip
protocol. Ports allocated from a fixed range (41001+) via atomic
counter — deterministic and race-free within the test binary.
66 tests pass. Pre-existing `hot::test_project_target_size_bytes`
macOS-only failure unchanged.
File sizes (under 1300-line ceiling):
- cluster/gossip.rs: 549
- cluster.rs: 274
- main.rs: 506
Follow-on Phase 1 cuts:
- 1c: `quinn` QUIC transport + fleet-CA mTLS for RPC (separate UDP
port so gossip and QUIC don't collide)
- 1d: daemon wire-up — gossip runs in background, hot-tier usage
pushed periodically, `cluster-status` reads from live daemon
|
||
|
Omar Sobh
|
5c19c60292 |
Phase 1a: cluster module + LAN-first peer probe
First cut of the v2 distributed FS. Captures the architecture design in ARCHITECTURE-v2.md and lands the smallest useful new capability: probing cluster peers with a LAN-first policy so subsequent transport + gossip layers (Phase 1b, 1c) can build on a real routing decision. New: - ARCHITECTURE-v2.md: zones (fabric-10g/lan-1g/roaming), tier lifecycle (hot/warm/cold), fingerprint-keyed build cache design, smart-clean policy, phase plan, explicit non-goals. - claw-store/src/cluster.rs: RouteKind, RouteWinner, LanFirstProbe. LAN 200ms timeout, Tailscale 500ms fallback. 8 tests use real TCP listeners on 127.0.0.1 (no mocks); cover happy path, fall-through, both-fail, single-address, and elapsed reporting. - claw-store/src/config.rs: ClusterConfig + PeerEntry with validation (bind-address presence, no duplicate peer names, per-peer reachable address required). Optional at top level so pre-v2 configs still load unchanged. 6 new tests. - claw-store/src/main.rs: `claw-store cluster-probe <peer>` CLI subcommand that reads config, resolves the peer, probes, prints the winning route + elapsed time. All 16 new tests pass. Existing 45 pass. Sole failure (hot::tests::test_project_target_size_bytes) is a pre-existing macOS-only issue with `du -sb`; Linux CI unaffected. Follow-on Phase 1 cuts (subsequent sessions): - 1b: chitchat SWIM gossip for live membership state - 1c: quinn QUIC transport with fleet-CA mTLS - 1d: `claw-store cluster status` — live membership view Every file well under the 1300-line ceiling (cluster.rs 268, config.rs 428, main.rs 450). |
||
|
Omar Sobh
|
aefa1cce58 |
URGENT FIX: stale-gc must not treat last_active=None as stale
Previous commit
|
||
|
Omar Sobh
|
643ba170b6 |
daemon: proactively deactivate stale (>48h idle) projects each tick
Answers a real operational question — 'why does /hot/targets stay near
full even when I haven't touched most of these projects for weeks?'
Previously the stale sweep was gated behind 'hot tier > 90% full', so
an idle project held NVMe until the operator manually deactivated it
or space ran out and LRU came for it.
The change:
* new claw-store/src/actions.rs — shared deactivate_project(cfg,
manifest_path, project) with the FULL flow: sync-to-peer +
hot rm + .cargo/config.toml removal + manifest update, plus a
DeactivateOutcome { synced, sync_error, freed_bytes } return.
Extracted from main.rs::cmd_deactivate so both CLI and daemon
run the same code — no drift between manual and auto semantics.
* daemon.rs poll_tick now runs the stale sweep on EVERY tick,
independent of space pressure. Each project idle > stale_hours
(48 by config) goes through the full deactivate, logs
'stale-gc: X freed N MB (synced=Y)'.
* gc_by_space (LRU) still gates on >90% full — it's the emergency
'even after the stale sweep we're still tight' path.
* main.rs cmd_deactivate is now a thin CLI wrapper that adds
println! feedback + reports freed MB in the terminal output.
Space-pressure LRU keeps its original .cargo/config.toml-preserving
semantics (rm hot only, not the shim) so a project marked 'active'
by manual activation but hard-evicted for space can still be
re-activated cheaply. Stale sweep is the aggressive one because the
project genuinely hasn't been touched.
Tests: two new unit tests in actions.rs cover the full flow +
idempotency; the freed_bytes assertion is Linux-only-safe (BSD du
returns 0 for -sb, same limitation as the existing hot test).
|
||
|
Omar Sobh
|
eb0ef29398 |
dashboard: active-only filter + row checkboxes + batch deactivate
Three additions to ProjectBrowser that address concrete UX pain
found while managing 174 warm projects:
1. 'active only' filter chip next to 'all'. Toggles a state that
filters visible list to is_active === true. Colored green to
distinguish from the neutral org chips.
2. Per-row checkbox on active projects (warm-only rows leave the
slot empty — deactivate is meaningless on them). A select-all
row appears above the list whenever at least one active project
is visible; it selects/deselects only the visible subset so
the operator can filter down then bulk-select.
3. Sticky action bar (appears when selected > 0) showing the count
and a 'Deactivate N' button. Click prompts a native confirm that
explains: sync-first, hot-tier freed, manifest cleared, warm
clone untouched. Sequential execution with a live progress
counter ('Deactivating 3/10...'); mid-batch failures are
collected and reported at the end without halting the run.
Selection is stable across filter changes — narrowing to a different
org keeps prior selections. Explicit 'Clear' button (X) resets.
Deactivation semantics documented above match cmd_deactivate in
main.rs:196-238: sync → evict hot → remove .cargo/config.toml →
manifest.projects.retain — never touches /slab/projects.
|
||
|
Omar Sobh
|
92b2dd751e |
daemon: auto-enqueue sync when a project's git HEAD moves
Removes the biggest workflow footgun in the architect↔tank flow: the
operator no longer has to remember to run 'claw-store sync <project>'.
An ordinary 'git commit' anywhere under /slab/projects/*/* is now
picked up automatically on the next 5-min poll tick and the peer is
notified — same code path deactivate/sync used to trigger by hand.
New module claw-store/src/head_watch.rs owns the state:
* HeadCache — {project → last-seen HEAD sha}, persisted at
/var/lib/claw-store/head-cache.toml (same directory as the
existing sync queue and manifest).
* scan_and_enqueue walks warm_root/<org>/<repo>, calls
'git rev-parse HEAD' on each, compares to cache, enqueues on
change. Non-git dirs, symlinks, and unreadable entries are
skipped (matches the project-list walker's behavior).
First-scan policy: if the cache is empty on startup we populate it
silently instead of enqueueing every project — otherwise a fresh
install with 55 warm projects would flood the peer with 55 syncs on
the first tick. Existing state gets baselined; only movement from
that point on triggers work.
daemon.rs poll_tick was one 'retry the queue' block; it's now
'scan for HEAD changes → save cache → drain queue' in that order,
so a commit from between ticks gets enqueued AND drained the
same tick.
Four unit tests cover the invariants: silent first scan, enqueue
on real HEAD move (only for the moved repo), cache roundtrip,
non-git directory skip.
|
||
|
Omar Sobh
|
17640a7bb1 |
dashboard: surface sync_queue_depth + sync_queue_stuck
Extends the React UI to render the two new NodeStatus fields shipped
in commit
|
||
|
Omar Sobh
|
29a6616c59 |
sync: never silently drop; branch-aware pull; expose queue depth in /api/status
Three fixes bundled — all defensive around the architect↔tank dev flow:
1. drain_sync_queue no longer drops jobs after 48 attempts. The
previous cutoff (~4h at 5-min drain intervals) silently lost every
pending sync whenever the peer was down longer than that — the
exact scenario the user hit last week when architect was offline.
Now jobs are retried indefinitely; past SYNC_STUCK_ATTEMPTS (12,
~1h) they escalate from WARN to ERROR and appear in /api/status
under sync_queue_stuck so the operator sees them.
2. pull_project is now branch-aware. Old behavior was 'git pull
--ff-only' on whatever branch happened to be checked out. If the
peer was on main but the sender committed on develop, the peer
never picked up the new branch's ref. New behavior:
- git fetch --all --prune --tags first (gets every branch)
- fast-forward the checked-out branch if it has an upstream
- fast-forward every OTHER branch that has an upstream via
update-ref, without switching HEAD — so main can advance while
the operator is off hacking on feature/x.
Diverged branches are left alone (loud error, not silent merge).
Detached HEAD is skipped after fetch.
3. NodeStatus gains sync_queue_depth + sync_queue_stuck fields, wired
into both handle_status (GET /api/status) and the SSE stream. The
dashboard now has ground truth for 'is anything backing up?'
Tests updated: the old test_sync_queue_drops_after_max_attempts is
replaced by test_sync_queue_never_drops_stuck_jobs (invariant: never
drops) and test_sync_queue_stuck_count_threshold (invariant: counts
jobs past the escalation threshold).
|
||
|
Omar SobhandClaude Sonnet 4.6
|
720b331527 |
fix: skip all symlinks in project list walker
Replace the ends_with(".git") name check with a file_type().is_symlink()
check at both the org and repo levels of build_projects_list.
This handles three classes of symlink that all produced duplicate entries:
1. Gitea's repo.git → repo aliases (previously caught by name check)
2. Within-org shortcuts: quantumclaw/bbq → quantum-bbq
3. Cross-org aliases: clawverse/clawhdf5 → ../quantumclaw/clawhdf5
The name check only caught class 1; the symlink check catches all three.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
|
||
|
Omar SobhandClaude Sonnet 4.6
|
c101cbe5ed |
fix: hot tier chart units GB + tooltip text color
- Convert size_mb → GB (÷1024) so axis and tooltip show GB not raw MB - Add itemStyle to Tooltip so value text is zinc-400, not Recharts' default near-black which was unreadable on the dark background - Add subtle hover cursor fill so bar highlight doesn't flash white - Widen right margin so GB tick labels don't clip Co-Authored-By: Claude Sonnet 4.6 <[email protected]> |
||
|
Omar SobhandClaude Sonnet 4.6
|
70d1d410bb |
fix: add hot/slab ReadWritePaths to serve service unit
ProtectSystem=strict in claw-store-serve.service only listed
/var/lib/claw-store as writable. The serve process shells out to
`claw-store activate/deactivate` which also needs to write to:
/hot/targets — create/remove hot target dirs
/slab/projects — write/remove .cargo/config.toml
Subprocess inherits the service's mount namespace, so both paths were
silently read-only, causing activate/deactivate from the dashboard to
return {"ok":false,"error":"Read-only file system (os error 30)"} while
the same commands worked fine from a login shell.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
|
||
|
Omar SobhandClaude Sonnet 4.6
|
f6ff04e9dc |
fix: skip *.git symlinks in project list scanner
Gitea creates a repo.git → repo symlink alongside every real checkout
in /slab/projects. The directory walker resolved both, found .git inside
each (since the symlink points at the real dir), and emitted a duplicate
entry for every project — e.g. quantumclaw/clawhdf5 and
quantumclaw/clawhdf5.git both appeared in the dashboard.
Add ends_with(".git") guard at both the org and repo levels of
build_projects_list so symlinked aliases are always skipped.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
|
||
|
Omar SobhandClaude Sonnet 4.6
|
1320d2df9e |
feat: v0.3.0 — HTTP handler tests + README
- Extract build_app() from run_server() so tests can construct the router without binding a port (tower::ServiceExt::oneshot pattern) - Add 10 new tests in serve::tests: validate_project_name (valid + invalid slugs), daemon_uptime when file absent, GET /api/status, GET /api/projects, POST /api/activate with invalid name, auth middleware (no header → 401, wrong token → 401, correct token → pass-through, GET bypasses auth entirely) - Add tower + http-body-util to dev-dependencies - Add project README covering architecture, install (Makefile), config options, SSH setup, snapshot schedule, CLI reference, HTTP API table, pinning, and troubleshooting guide - Bump version to 0.3.0 Test suite: 39 tests, 0 failures Co-Authored-By: Claude Sonnet 4.6 <[email protected]> |
||
|
Omar SobhandClaude Sonnet 4.6
|
39ddac431b |
chore: stop tracking compiled dashboard assets
claw-store/static/ is now in .gitignore. The bundle is installed via `make install-dashboard` which runs `npm run build` and copies to /usr/share/claw-store/static — it should not live in the repo. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> |
||
|
Omar SobhandClaude Sonnet 4.6
|
1b1d66eac9 |
feat(v0.3.0): weekly snapshots, incremental replication, auth, uptime, hardening
- snapshot: implement weekly ZFS snapshots (Sunday midnight, retain_weekly config) - snapshot: incremental cold replication via zfs send -i; tracks last replicated snapshot in /var/lib/claw-store/last-replicated-snapshot - daemon: write start-time file for uptime reporting; add SIGTERM graceful shutdown - serve: daemon_uptime_secs now reads the start-time file (was hardcoded 0) - serve: validate project names in POST handlers (org/repo slug, no traversal) - serve: Bearer token auth middleware on all POST endpoints via cfg.api_token - config: add optional api_token field (backward compatible, defaults to None) - sync: add SSH timeouts (ConnectTimeout=10, ServerAliveInterval=5) to peer notify - hot/sync/main: replace unwrap() on path-to-str with proper anyhow errors - config/tank.toml: document 10G fabric IP and nightly_at field intent - .gitignore: exclude compiled dashboard assets (claw-store/static/) - Makefile: add build, install, install-systemd, install-dashboard, deploy targets - tests: 29 passing (up from 25); 4 new weekly snapshot tests Co-Authored-By: Claude Sonnet 4.6 <[email protected]> |
||
|
Omar SobhandClaude Opus 4.7
|
d076fac619 |
fix(serve): single-source the project list — stops dashboard flashing
The HTTP `/api/projects` handler and the SSE `/api/events` stream each
built their own ProjectInfo list — independent inline code, ~50 lines
of mostly identical duplicate. But the two diverged on ONE field:
/api/projects: is_active = active.is_some() || has_hot_cargo_config(...)
/api/events: is_active = active.is_some()
The dashboard hits BOTH every few seconds (a 10-second setInterval
polling /api/projects + a 5-second SSE pushing events). For any
project that was hot-wired via `.cargo/config.toml` but not in the
manifest, the two endpoints disagreed about its `is_active` bit.
The dashboard's last-write-won, and the row visibly flickered as
the active flag toggled. The 'X active' count in the header bounced
along with it.
Extracted `build_projects_list(cfg, &manifest)` and call it from both
sides. Same answer in both code paths. Verified post-deploy:
architect: HTTP=59 active, SSE=59 active (identical)
tank: HTTP=57 active, SSE=57 active (identical)
This also makes the 'total' number stable. The 354 in the dashboard
header is correct semantics — discoverable git repos on disk — but
the label is misleading. Cosmetic cleanup for a follow-up:
- "Projects — N discovered · M active · K shown" reads truer than
"N shown · M active · K total"
- The status card's `active_project_count` (manifest.projects.len)
is a third meaning of 'active' that doesn't match either dashboard
number; consider renaming to `manifest_size` to disambiguate.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.7
|
d5651614ba |
fix(serve): drop ProtectHome — was blocking SSH peer probe
The v0.2.0 unit set ProtectHome=true to lock down /home from a
hypothetical RCE in axum. Side effect: ssh in peer_reachable
(serve.rs:145) failed with "Host key verification failed: Permission
denied" because ~/.ssh/known_hosts was unreachable. Adding
BindReadOnlyPaths=/home/osobh/.ssh didn't help — systemd applies
ProtectHome before the bind mounts run, so /home is already an
inaccessible barrier when the bind lands. Result: both dashboards
showed peer_reachable=false even though LAN ping + SSH worked fine
from a shell.
Two options to keep some sandboxing:
1. ProtectHome=tmpfs + BindReadOnlyPaths=/home/osobh/.ssh — bind
into an empty tmpfs view of /home.
2. Drop ProtectHome entirely — keep ProtectSystem=strict +
ReadWritePaths=/var/lib/claw-store + NoNewPrivileges + PrivateTmp.
Going with (2) for now. The threat model is local-host RCE in a
read-mostly axum service the dashboard pokes; ProtectSystem alone
prevents writing anywhere outside /var/lib/claw-store. Re-introduce
(1) when we have a clean justification.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.7
|
af50adec19 |
feat(v0.2.0): atomic+flocked manifest, pinned projects, serve systemd unit
The two biggest pain points coming out of the architecture review:
1. The manifest at /var/lib/claw-store/projects.toml was the only
piece of writable state but had no locking, no atomic writes,
and three concurrent writers (daemon poll tick, every CLI verb,
and the dashboard shelling out via /api/activate). Two writers
interleaving silently dropped one of them; a crash mid-write
left a corrupt half-written TOML that the next reader parsed
as an empty manifest.
2. Reboot survival: the dashboard had no systemd unit and was a
stray hand-launched process. Architect lost its dashboard on
todays reboot.
This commit lands:
- Manifest::update(path, FnOnce(&mut Manifest)) — locked-atomic
load-mutate-save in one transaction. Uses libc::flock(LOCK_EX) on
a sidecar .lock file (so the data file can be replaced by rename
without invalidating the lock) and tempfile + persist for the
rename. Concurrent writers serialise; readers see the previous
state or the new state, never a torn write. Manifest::load uses
LOCK_SH so it never races a mid-rename.
- Project.pinned: bool with #[serde(default)] so legacy manifests
parse cleanly. hot::gc_stale_targets and hot::gc_by_space both
skip pinned projects, with a WARN log when every remaining
project is pinned but were still over budget — operator intent
beats space pressure.
- claw-store pin <project> / unpin <project> CLI verbs.
status command surfaces pin marker (📌).
activate preserves an existing rows pinned flag so re-activating
doesnt silently unpin.
- claw-store-serve.service systemd unit. Type=simple, Restart=
on-failure, RestartSec=15, ProtectSystem=strict + ReadWritePaths
=/var/lib/claw-store, ProtectHome, NoNewPrivileges, PrivateTmp.
- daemon poll tick reloads the manifest from disk at the start of
each cycle (so CLI activations between ticks are visible) and
routes its GC write through Manifest::update (so it cant race a
concurrent CLI pin).
- libc + tempfile move from dev-deps into runtime deps.
- empty-manifest fallthrough on load (treat "" as default) so a
half-written tempfile crashed pre-rename doesnt hard-fail the
daemon next boot.
- 25 tests passing incl. new ones: legacy-toml-parses, update-
serializes-two-sequential-writers, pinned-survives-stale-gc,
pinned-survives-space-gc-even-when-lru.
Version bumped 0.1.0 → 0.2.0.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.7
|
2fcfb16160 |
feat(serve): HTTP+SSE API + static dashboard bundle
Adds the serve subcommand: an axum 0.7 + tower-http server on
:7700 backing the React dashboard. Routes:
GET /api/status /api/projects /api/snapshots
/api/sync-queue /api/hot /api/events (SSE 5s tick)
POST /api/activate /api/deactivate /api/sync
/api/gc /api/snapshot
Mutating endpoints shell out to /usr/local/bin/claw-store so
all CLI logic stays single-sourced. claw-store/static/ holds
the prebuilt dashboard Vite bundle for --static-dir.
cargo_init.rs gets a minor wiring tweak so the API can read
back the managed cargo config marker.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar Sobh
|
7f57b6e3bd |
feat(dashboard): React+Vite SPA for the claw-store HTTP API
Single-page app with NodeCard, HotTierChart, SyncQueueCard, SnapshotTimeline, and ProjectBrowser. Hits the new HTTP/SSE API surface in claw-store serve (next commit). Built with Vite/Tailwind/lucide-react; consumed by mounting the dist/ output at /usr/share/claw-store/static (or wherever --static-dir points) and serving via claw-store serve. |
||
|
Omar SobhandClaude Sonnet 4.6
|
8b4d0088ca |
fix: list command only shows git repos, not org subdirs
Skip non-git directories inside org folders so flat layout dirs like claw-store/target or claw-store/.cargo don't appear as fake repos. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> |
||
|
Omar SobhandClaude Sonnet 4.6
|
a9c12173fe |
feat: activate/deactivate/list/sync/pull commands + peer sync queue
- activate <org/repo>: wire hot tier, write .cargo/config.toml, register - deactivate <org/repo>: auto-sync to peer, evict hot tier, unregister - list: show all warm repos with activation status and last-active time - sync <org/repo>: git push origin then SSH peer to pull - pull <org/repo>: git pull from origin, stamp last_sync in manifest - SyncQueue: file-based retry queue (/var/lib/claw-store/sync-queue.toml) - daemon: drains sync queue on every 5-min poll tick - config: [peer] host/user for cross-node notification - manifest: Project.name is now org/repo, adds last_sync field - hot tier paths are org/repo-namespaced to avoid collisions Co-Authored-By: Claude Sonnet 4.6 <[email protected]> |
||
|
Omar Sobh
|
3fb108edfc | fix: manifest path to /var/lib/claw-store (user-writable) | ||
|
Omar Sobh
|
f596ae28b9 | feat: systemd unit files and node configs | ||
|
Omar Sobh
|
ac3079b56c | fix: SystemZfs uses sudo for write ops (snapshot/destroy/clone/send) |