clawverse/clawstor
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
77 Commits 10 Branches 0 Tags
e70f5d74e0b8cc73f7c2c4e476ccf21b54f74375
Commit Graph
8 Commits
Author SHA1 Message Date
Omar Sobh e70f5d74e0 Pilot findings: 5 real-world fixes from the 2026-07-12 deploy 
Bundles the profile→dir bug (PR #20 supersede) with four new fixes
discovered by running clawstor against itself + across the fabric:

* target_subdir_for: `dev`/`test` → `debug/`, `release`/`bench` →
  `release/`, custom passes through. Was silently skipping upload.

* rustc release via gossip: daemon probes `rustc --version` at start,
  publishes the release string as `clawstor.rustc.release`. PeerView
  carries it; `cluster-peer-status` prints it in a new column and
  emits a warning line when the fleet has mixed versions. Would have
  surfaced the tank/architect 1.96.1 vs 1.95.0 drift instantly.

* prewarm publishes fingerprint→blob ref downstream: `pin` now writes
  a companion tag `<name>.fingerprint` holding the fingerprint bytes.
  `prewarm` reads the companion, PutTag's it downstream, then
  PutRef(fp→blob) so a subsequent fingerprint-based `build` HITS.
  Without this, prewarm was almost useless for the runner path
  (build always missed even with matching source + rustc).

* streaming byte counters: BlobPutStream + BlobGetStream now record
  the transferred bytes via `record_blob_{put,get}_bytes`. Metric
  used to stay at 0 no matter how much you moved.

* capture determinism: replaced `tar::Builder::append_dir_all` (uses
  `read_dir`'s native order) with `append_dir_sorted` that walks the
  tree recursively and sorts by filename bytes at every level. Two
  byte-identical trees now produce byte-identical tars regardless of
  filesystem ordering.

+3 tests:
- target_subdir_matches_cargo_layout (from #20)
- fingerprint_companion_tag_uses_dotted_suffix
- capture_is_order_independent_of_filesystem_readdir (guard against
  the exact bug we saw in the field)

252 tests pass (+1 from Phase 5h's 251). Pre-existing macOS `du -sb`
failure unchanged.

Supersedes #20 (also included here). Ready for re-deploy to
tank + architect for the retest run.
 2026-07-12 05:34:36 -07:00
Omar Sobh 523b22f148 Phase 5j: Prometheus /metrics endpoint 
Adds a tiny axum-served HTTP endpoint that exposes the same
CacheMetrics counters that back GetMetrics + gossip, in Prometheus
text exposition format (v0.0.4). Enable per node by setting
`cluster.prom_bind` in the daemon config.

* metrics.rs: MetricsReply::to_prometheus() emits one HELP + TYPE +
  sample line per counter. started_unix is a gauge; everything else is
  a counter. Preallocates ~1 KiB so no reallocs mid-format.
* prom.rs (new): PromServer::bind spins up axum on a TcpListener,
  serves GET /metrics, returns 404 elsewhere. Graceful shutdown via
  oneshot channel; abort() variant for the sync-drop path in
  ClusterServices. Snapshots on every scrape (no cache) — Relaxed
  atomic loads are cheap enough that even 1 Hz is sub-microsecond.
* config.rs: new optional `cluster.prom_bind: SocketAddr` field.
  Default None means no server; typical value is 127.0.0.1:7702.
* services.rs: wires PromServer into ClusterServices when both a
  router and prom_bind exist. Warns (doesn't fail) if prom_bind is set
  without RPC — nothing would ever change on a scrape.

+8 tests:
- metrics: to_prometheus emits every counter with correct type; zeros
  still produce valid exposition (fresh daemon scrape)
- prom: content-type is text/plain; version=0.0.4; live updates
  between requests (no cache); unknown paths 404; shutdown stops
  serving
- services: end-to-end scrape returns router-driven counters;
  prom_addr() is None when unconfigured (no accidentally-leaked port)

Raw-TCP HTTP client in tests instead of pulling in reqwest — 30 lines
of tokio::net + string split for GET / read-to-close is small enough
to justify not adding a dep.

249 tests pass (+8 from Phase 5i). Pre-existing macOS `du -sb` failure
unchanged.
 2026-07-12 04:36:34 -07:00
Omar Sobh 29be728089 Phase 5i: publish cache metrics via gossip 
ClusterServices now periodically snapshots the router's CacheMetrics
and republishes four raw counters — GetRef hits/misses and blob
GET/PUT byte totals — through chitchat as clawstor.cache.* keys.
Peers derive the hit rate locally via PeerView::cache_get_ref_hit_rate,
eliminating a per-peer GetMetrics roundtrip for placement decisions.

* gossip.rs: 4 new well-known keys, PeerView carries the counters +
  a saturating-add derived hit rate that returns None on 0/0 or when
  either counter is missing (guards against half-writes reading as
  100% hits).
* services.rs: router hoisted out of the TLS branch so a
  cache_metric_task can hold Arc<RpcRouter>. Publishes once at start
  (initial zeros so peers don't wait 60s for first read) then every
  CACHE_METRIC_INTERVAL. Ticker skipped when RPC isn't up — counters
  only fire inside dispatch.

+3 tests:
- gossip: two-node convergence with cache counters + hit-rate math
- gossip: half-write / 0-0 / normal PeerView cases return correct rates
- services: fresh cluster sees Some(0) for all four keys, then after
  in-process router counters + explicit set_cache_metrics the peer sees
  the updated values with hit rate 0.8

241 tests pass (+3 from Phase 5g). Pre-existing macOS `du -sb` failure
in hot::tests unchanged.
 2026-07-12 04:20:45 -07:00
Omar Sobh f43b34ad15 Phase 2b: Blob RPC (Stat/Get/Put/LoadManifest) 
Wires the Phase 2 content-addressed blob store onto the network via
four new methods on the existing RpcRouter. Combined with the mTLS +
gossip stack from Phase 1c-1e, peers can now exchange content-addressed
blobs over real QUIC. This is the substrate the fingerprint-keyed cargo
cache (Phase 5) sits on directly.

New methods:
- BlobStat        (0x03): payload = 32-byte BlobId, reply = JSON BlobStat
- BlobGet         (0x04): payload = 32-byte BlobId, reply = raw bytes
- BlobPut         (0x05): payload = raw bytes, reply = 32-byte BlobId
- BlobLoadManifest (0x06): payload = 32-byte BlobId, reply = JSON manifest

New error codes:
- NotFound (0xf3)      — the requested BlobId isn't in the local store
- InvalidRequest (0xf4) — e.g. non-32-byte payload for a hash-keyed method
- NotConfigured (0xf5) — Blob* called on a router without an attached store

Wire-format bump: MAX_MESSAGE_BYTES 16 KiB → 16 MiB so a single 4 MiB
chunk (plus JSON overhead) fits comfortably. Anything above 16 MiB
needs the streaming variants coming in Phase 2c.

Client helpers:
- call_blob_stat / call_blob_get / call_blob_put / call_blob_load_manifest
- All map ErrorCode::NotFound to Ok(None), other codes to Err.
- call_blob_put verifies the peer-assigned BlobId matches local blake3
  hash of the payload — corruption or protocol drift surfaces
  immediately instead of silently accepting a mismatched receipt.

Router changes:
- RpcRouter grows an Option<Arc<BlobStore>> via with_blob_store(store).
- handle() split into handle_outcome() → HandlerOutcome enum
  {Reply(bytes) | Error(ErrorCode)} for cleaner control flow across
  the growing method set.

Services / daemon:
- ClusterServices::start gains blob_store_root: Option<PathBuf>.
- ClusterConfig gains blob_store_root: Option<PathBuf>.
- daemon.rs reads it from cluster_cfg + passes through.
- New ClusterServices::blob_store_enabled() introspection.

Tests (11 new, all real — no mocks):

Router (7 new):
- method_round_trips_byte_encoding — updated for 6 methods
- error_code_describe_covers_all_variants — updated for 6 codes
- decode_error_covers_all_known_codes
- blob_rpcs_return_not_configured_without_store — all four Blob*
  methods return NotConfigured when the router lacks a store
- blob_stat_returns_not_found_for_missing
- blob_stat_returns_json_for_existing
- blob_stat_returns_invalid_request_for_bad_length
- blob_get_returns_content_bytes
- blob_put_stores_bytes_and_returns_hash — verifies BlobId matches
  independent local hash
- blob_load_manifest_returns_json_for_existing (2-chunk case)
- blob_load_manifest_returns_not_found_for_missing

End-to-end over real QUIC (2 new):
- end_to_end_blob_put_stat_get_over_real_quic — full 4-method loop
  (Put → Stat → Get → LoadManifest) + NotFound path
- end_to_end_multi_chunk_blob_over_real_quic — 6 MiB blob → 2 chunks,
  proves MAX_MESSAGE_BYTES bump took effect

Services (1 new):
- services_with_blob_store_serves_blob_rpc_end_to_end — cut CA, sign
  leaves, config includes blob_store_root, start ClusterServices,
  dial from B over persisted mTLS, put + get through the router,
  then independently verify bytes landed on A's on-disk store

Also fixed a parallel-test port collision: services `next_port()`
now increments by 2 so `port + 1` (the RPC bind) is reserved
alongside `port` (the gossip bind).

128 tests pass. Pre-existing macOS-only failure unchanged.

File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 1006
- cluster/services.rs: 535
- cluster/blob.rs: 802
- config.rs: 511
- daemon.rs: 265

Follow-on:
- 2c: streaming variants (AsyncRead/AsyncWrite) so a many-GB blob
  transfers without holding it in memory
- 2d: chunk-level RPC (BlobPutChunk / BlobGetChunk) so a receiver
  can request only chunks it's missing after LoadManifest
- Phase 5 (the killer feature) can now build on Phase 2b directly —
  fingerprint the target dir, PutBlob the compressed tarball, and
  next node calls GetBlob keyed by the same fingerprint hash.
 2026-07-11 22:42:17 -07:00
Omar Sobh 2ae079fbd4 Phase 1e: daemon-integrated cluster services + PeerStatus RPC 
Ties Phase 1a-1d together into a live subsystem the daemon actually
runs. When `[cluster]` is present in config, `claw-store daemon` now:

  1. Starts chitchat gossip via ClusterServices::start.
  2. Publishes hot.max_gb immediately and re-measures the hot dir
     every 30s, updating `clawstor.hot.used`.
  3. If `[cluster.tls]` is also configured, loads NodeIdentity from
     PEM files and binds a QUIC RPC server that accepts + dispatches
     incoming Ping / PeerStatus requests via RpcRouter.

New modules:

- cluster/rpc.rs (510 lines):
  - Method enum (Ping=0x01, PeerStatus=0x02)
  - ErrorCode enum (EmptyRequest / UnknownMethod / HandlerFailure)
  - PeerStatusReply { local_name, local_zone, peers: Vec<PeerView> }
  - RpcRouter — dispatch state (holds Arc<ClusterGossip> + local
    name/zone)
  - rpc_call / call_ping / call_peer_status — client helpers
  - serve_connection — server accept-bidi loop
  - Wire format: `method:u8 || payload:bytes` → `reply:bytes` or
    single-byte ErrorCode

- cluster/services.rs (418 lines):
  - ClusterServices { gossip, accept_task, metric_task }
  - start(cluster_cfg, name, hot_dir, hot_max_bytes) → bootstraps
    everything above
  - shutdown() aborts background tasks cleanly
  - Recursive dir-walker (spawn_blocking) for hot-tier metric

PeerView now derives Serialize/Deserialize so it round-trips through
JSON over the RPC.

daemon.rs integration (~35 lines added):
  - Bootstraps ClusterServices before entering the select loop
  - Held for daemon lifetime
  - Shutdown on SIGTERM
  - Gossip-less config still runs standalone (backwards compat)

CLI: `cluster-peer-status --peer <name> --rpc-addr <addr>
                          --tls-dir <dir>`
  Loads a persistent identity, dials the peer, calls PeerStatus,
  prints the peer's local view as a table.

Tests (15 new, all real — no mocks, real UDP + TLS + JSON round trip):

RPC (9):
- method_round_trips_byte_encoding
- dispatch_returns_pong_for_ping
- dispatch_returns_json_for_peer_status
- dispatch_returns_empty_request_error
- dispatch_returns_unknown_method_error
- rpc_call_rejects_oversize_payload (with real quinn connection)
- end_to_end_ping_and_peer_status_over_real_quic — full 2-node quinn
  with mTLS + both RPCs
- peer_status_reflects_peer_gossip_state — 2 gossip instances converge,
  RPC caller from a third identity sees the converged view
- error_code_describe_covers_all_variants

Services (6):
- dir_bytes_sync_returns_zero_for_missing_path
- dir_bytes_sync_sums_recursive_file_sizes (3-level nesting)
- services_start_without_tls_leaves_rpc_disabled
- services_start_with_tls_serves_rpc_end_to_end — full stack: cut CA on
  disk, sign leaves, start ClusterServices for A, dial from B via
  persisted mTLS, run ping + PeerStatus over the wire
- services_publish_hot_used_metric_periodically
- services_gossip_sees_peer_after_convergence

95 tests pass. Pre-existing macOS-only failure unchanged.

File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 510
- cluster/services.rs: 418
- cluster/gossip.rs: 579
- cluster/transport.rs: 916
- daemon.rs: 262
- main.rs: 749

Phase 1 done end-to-end. `claw-store daemon` now boots a real distributed
cluster stack when config asks for one; peers can call each other's
PeerStatus RPC and see live membership. Next: Phase 2 (content-addressed
blob store) can hook new RPC methods into the same RpcRouter with a
one-line dispatch arm.
 2026-07-11 22:15:41 -07:00
Omar Sobh 916add37df Phase 1d: persistent NodeIdentity + FleetCa + fleet-ca CLI 
Closes out Phase 1. A production operator can now cut a fleet CA,
sign per-node leaves, drop the resulting PEMs at
/etc/claw-store/tls/, point [cluster.tls] at them, and the daemon
loads real mTLS material on startup — no more ephemeral in-process
CA hack.

New public API in cluster::transport:
- FleetCa::generate(cn) — new self-signed root CA
- FleetCa::save(dir) / FleetCa::load(dir) — round-trip PEM
- FleetCa::sign_leaf(name) — mint an in-memory NodeIdentity
- FleetCa::sign_leaf_to_pem(name, out_dir) — write ca.crt + node.crt +
  node.key (node.key at 0o600 on Unix)
- NodeIdentity::from_pem_files(ca, cert, key) — production load path
- NodeIdentity::from_pem_dir(dir) — canonical filename layout
- NodeIdentity::from_cluster_config(cfg) — pick up [cluster.tls] paths

Manual Debug for FleetCa redacts the private key.

Config extension:
- [cluster.tls] ca_cert / node_cert / node_key (all PathBuf).
- Optional at the top level; callers that need mTLS surface a clear
  error when it's absent.

Deps:
- rcgen features += "x509-parser" (for FleetCa::load's from_ca_cert_pem).
- rustls-pemfile 2 (parse PEM back into DER for rustls).

CLI (new commands; short-circuit config load so they run on fresh
boxes without /etc/claw-store/config.toml):
- fleet-ca-init --dir <dir> [--cn <name>]
    generates ca.crt + ca.key (both 0o600).
- fleet-ca-sign --ca-dir <dir> --node <name> --out-dir <dir>
    writes ca.crt + node.crt + node.key (node.key at 0o600).
- cluster-ping now accepts --tls-dir <dir> to load persistent
  NodeIdentity from disk (produced by fleet-ca-sign).

Tests (8 new, all real — no mocks, real filesystem, real TLS handshake):
- fleet_ca_rejects_empty_common_name
- fleet_ca_save_and_load_round_trip_preserves_signing (asserts 0o600
  on ca.key)
- fleet_ca_load_errors_when_files_missing
- sign_leaf_to_pem_writes_all_three_files_with_correct_permissions
  (asserts 0o600 on node.key)
- persistent_identity_round_trips_through_disk_and_pings — end-to-end:
  cut CA on disk, reload it, sign two leaves via sign_leaf_to_pem,
  reload them via from_pem_dir, run real QUIC ping/pong. This is the
  operator flow.
- node_identity_from_cluster_config_errors_without_tls_section
- node_identity_from_cluster_config_loads_pem_paths
- from_pem_files_errors_on_missing_ca_file

80 tests pass. Pre-existing macOS-only hot test unchanged.

Also verified live CLI smoke test:
  fleet-ca-init → ca.crt + ca.key at 0o600
  fleet-ca-sign → ca.crt + node.crt + node.key at 0o600
  Files parse as valid X.509.

File sizes (all under 1300-line ceiling):
- cluster/transport.rs: 916
- cluster/gossip.rs: 576
- cluster.rs: 275
- config.rs: 503
- main.rs: 662

Phase 1 complete. Next up:
- Phase 1e (daemon integration): gossip + QUIC RPC server wired into
  claw-store daemon; hot-tier metrics periodically pushed; a real
  PeerStatus RPC alongside ping.
- Phase 2: content-addressed blob store (BLAKE3 chunking, put/get).
 2026-07-11 22:04:54 -07:00
Omar Sobh 4d93f955c5 Phase 1c: QUIC transport with fleet-CA mTLS + cluster-ping 
Wraps quinn 0.11 in cluster/transport.rs with rustls 0.23 + rcgen 0.13
for identity management. Every peer connection is mTLS: both sides must
present certs signed by the shared fleet CA, and rustls verifies the
peer's cert SAN matches the requested server name.

Public API on `cluster::transport`:
- `NodeIdentity` (cert chain + private key + trusted CA)
- `NodeIdentity::generate_test_pair(a, b)` — ephemeral CA + two signed
  leaves; shape matches the production PEM-file loader (Phase 1d)
- `QuicServer::bind(addr, identity)` — bind mTLS-enforced listener
- `QuicServer::accept()` — accept one connection (Option<Result<_>>)
- `QuicClient::new(local_addr, identity)` — build client endpoint
- `QuicClient::connect(peer_addr, expected_name)` — outbound with SAN check
- `ping(&conn, payload)` — bidi stream RPC; server echoes as `pong:<payload>`
- `ping_handler_loop(conn)` — server-side accept/echo forever
- `CLAWSTOR_RPC_ALPN` constant, single ALPN "clawstor-rpc/1"

Config extension:
- `bind_rpc_lan`, `bind_rpc_tailscale` on `ClusterConfig` (both optional).
- Defaults: gossip port + 1 (so gossip UDP and QUIC UDP don't collide).
- Helper: `ClusterConfig::rpc_lan()`, `rpc_tailscale()`,
  `PeerEntry::rpc_lan()`, `rpc_tailscale()`.

Gossip.rs now advertises the RPC address, not the gossip address, on
the well-known `clawstor.rpc.lan` / `clawstor.rpc.tailscale` keys.

CLI:
- `cluster-ping --name <me> --peer <you> --rpc-addr <addr> [--payload X]`
  Runs a full mTLS handshake and single ping. For dev/loopback use
  today (both sides need to share a CA); persistent-identity ping
  lands in Phase 1d.

Tests (6 new, real UDP + TLS handshake, no mocks):
- generate_test_pair produces two distinct leaves that share a CA
- ping_pong_between_two_mtls_peers: real 2-node QUIC round trip with
  full mTLS chain verification, `open_bi()`/`accept_bi()`, byte-exact
  response check
- client_rejects_peer_with_wrong_ca: TLS chain verification failure
  when the server presents a cert signed by a different CA
- client_rejects_wrong_server_name: SAN mismatch is enforced
- server_binds_wildcard_and_reports_concrete_local_addr: port 0 → real
- ping_rejects_oversize_payload: MAX_MESSAGE_BYTES cap enforced client-side

Fixed-port allocator range 42000+ so transport tests don't conflict
with gossip tests (41000+).

72 tests pass. Pre-existing `hot::test_project_target_size_bytes`
macOS-only failure unchanged.

File sizes (all under 1300-line ceiling):
- cluster/transport.rs: 485
- cluster/gossip.rs: 570
- cluster.rs: 275
- config.rs: 480
- main.rs: 564

Follow-on Phase 1 cut (1d):
- Load NodeIdentity from persistent PEM files at /etc/claw-store/tls/
- Fleet CA bootstrap ceremony (rcgen → write CA cert; per-node leaf CSR)
- Daemon-level RPC server that runs alongside gossip + serves real
  operations (blob get/put, metadata sync)
- cluster-status reads from live daemon via API instead of standalone
 2026-07-11 21:50:28 -07:00
Omar Sobh 5283ab8655 Phase 1b: chitchat SWIM gossip layer + cluster-status CLI 
Wraps chitchat 0.11 in cluster/gossip.rs. Each node publishes its
zone, RPC endpoints, hot-tier usage, warm-tier project list, and
uptime under well-known kv keys; peers propagate this via gossip and
are classified alive/dead by phi-accrual failure detection.

Bootstraps from static [[cluster.peers]] seed nodes in the config.
Membership extends dynamically as nodes join or leave.

Public API on `ClusterGossip`:
- bootstrap(cluster_cfg, local_name) — start UDP gossip service
- set(key, value), set_hot_used, set_hot_max, set_warm_projects
- peers() — all known peers with typed PeerView + liveness
- peer(name) — one peer by advertised name
- peers_in_zone(zone) — live peers filtered by zone
- shutdown() — abort gossip task

CLI:
- `claw-store cluster-status --name <me> [--wait-secs N]`
  bootstraps gossip, waits for convergence, prints a peer table
  with zone, alive/dead, RPC LAN/Tailscale addresses, hot usage.

Tests (7 new):
- bootstrap_publishes_our_own_state — self-ID + advertised address
- bootstrap_fails_when_local_name_empty
- bootstrap_fails_when_no_bind_address — cluster.validate error chain
- two_node_cluster_converges_and_shares_state — REAL two-node
  in-process UDP gossip. Publish state on A, wait for phi-accrual
  liveness on B, verify all fields propagated (zone, rpc_lan, hot
  bytes, warm_projects list, fill ratio, zone filtering).
- peers_excludes_self — solo cluster reports zero peers
- peer_view_hot_fill_ratio_handles_missing_or_zero

No mocks. Real UDP transport, real chitchat runtime, real gossip
protocol. Ports allocated from a fixed range (41001+) via atomic
counter — deterministic and race-free within the test binary.

66 tests pass. Pre-existing `hot::test_project_target_size_bytes`
macOS-only failure unchanged.

File sizes (under 1300-line ceiling):
- cluster/gossip.rs: 549
- cluster.rs: 274
- main.rs: 506

Follow-on Phase 1 cuts:
- 1c: `quinn` QUIC transport + fleet-CA mTLS for RPC (separate UDP
      port so gossip and QUIC don't collide)
- 1d: daemon wire-up — gossip runs in background, hot-tier usage
      pushed periodically, `cluster-status` reads from live daemon
 2026-07-11 21:41:52 -07:00