f30ea04ab6fc7368be46bee30154f7c3e541ee3c
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 osobh
|
baefd95427 |
Phase 9 F4: namespaced tokens for multi-tenant aggregator (#104)
Build with clawstor cache / Cargo build (clawstor-cached) (push) Failing after 2s
|
||
 Omar Sobh
|
2f3055a3aa |
blob: size-based LRU eviction + auto-cap in the GC ticker
Orphan-chunk GC alone doesn't stop unbounded growth: as long as fingerprint→blob refs keep getting PutRef'd, the manifest set keeps growing and no chunk is ever an orphan. * `BlobStore::evict_to_size_cap(max_bytes)` — walks manifests oldest first by mtime, deletes them, refcount-decrements each chunk they used, unlinks + reclaims size for any chunk whose refcount hits zero. Shared chunks stay put until the last blob referencing them is evicted. * `ManifestSummary` internal type keeps the diff-set bookkeeping cheap (one HashMap<ChunkHash, u32>, no repeated tree walks). * `claw-store cluster-gc --evict-to-gb <N>` extends the CLI: still runs the orphan sweep first, then optionally caps the store. * Config: `cluster.blob_max_gb: Option<u64>`. The auto-GC ticker runs eviction after every orphan sweep when this is set. Silent when the store is already under cap; INFO log when it evicts. +3 tests: - evict_to_size_cap_reclaims_oldest_blobs_first: 3 blobs with distinct mtimes, cap below combined size → oldest evicted, newer blobs survive - evict_keeps_shared_chunks_when_still_referenced: guards the refcount decrement path (content-addressed dedup keeps identical content as one blob → chunk survives until manifest deleted) - evict_on_empty_store_is_a_noop: sanity 257 tests pass (baseline +3). Pre-existing macOS failure unchanged. |
||
|
Omar Sobh
|
84aa758fd4 |
Two pilot follow-ons: rustc drift warning + blob GC
Both surfaced by the 2026-07-12 pilot as real operator concerns: ## rustc drift warning at build time Runners silently silo their cache when rustc versions differ across peers (fingerprint depends on rustc verbose output). The pilot's first flow burned a full cold+upload before we realized the silo. - `PeerStatusReply.local_rustc_release` — new field, populated from the peer's own gossip `RUSTC_RELEASE` key via a new `ClusterGossip::self_kv(key)` accessor. - `claw-cargo build`: on cache MISS, calls `PeerStatus`; if the peer's rustc release ≠ our local `rustc --version`, emits a WARN with both versions + hint to add `rust-toolchain.toml`. - Best-effort: absence of either release string is a shrug, not an error. ## blob GC Blob store grows unbounded on a runner; disk-full is a real incident. `gc_orphan_chunks` already existed but wasn't exposed. - New CLI: `claw-store cluster-gc` — runs `gc_orphan_chunks`, prints report. Safe to run any time, safe to interrupt. - New config: `cluster.gc_interval_hours: Option<u64>`. When set to a positive integer, the daemon spawns a periodic ticker that invokes GC in-process. Skips the first tick (nothing to reclaim on boot). Errors are logged and retried next tick. - Shutdown aborts the ticker cleanly. 254 tests pass (baseline unchanged). Pre-existing macOS failure untouched. |
||
|
Omar Sobh
|
523b22f148 |
Phase 5j: Prometheus /metrics endpoint
Adds a tiny axum-served HTTP endpoint that exposes the same CacheMetrics counters that back GetMetrics + gossip, in Prometheus text exposition format (v0.0.4). Enable per node by setting `cluster.prom_bind` in the daemon config. * metrics.rs: MetricsReply::to_prometheus() emits one HELP + TYPE + sample line per counter. started_unix is a gauge; everything else is a counter. Preallocates ~1 KiB so no reallocs mid-format. * prom.rs (new): PromServer::bind spins up axum on a TcpListener, serves GET /metrics, returns 404 elsewhere. Graceful shutdown via oneshot channel; abort() variant for the sync-drop path in ClusterServices. Snapshots on every scrape (no cache) — Relaxed atomic loads are cheap enough that even 1 Hz is sub-microsecond. * config.rs: new optional `cluster.prom_bind: SocketAddr` field. Default None means no server; typical value is 127.0.0.1:7702. * services.rs: wires PromServer into ClusterServices when both a router and prom_bind exist. Warns (doesn't fail) if prom_bind is set without RPC — nothing would ever change on a scrape. +8 tests: - metrics: to_prometheus emits every counter with correct type; zeros still produce valid exposition (fresh daemon scrape) - prom: content-type is text/plain; version=0.0.4; live updates between requests (no cache); unknown paths 404; shutdown stops serving - services: end-to-end scrape returns router-driven counters; prom_addr() is None when unconfigured (no accidentally-leaked port) Raw-TCP HTTP client in tests instead of pulling in reqwest — 30 lines of tokio::net + string split for GET / read-to-close is small enough to justify not adding a dep. 249 tests pass (+8 from Phase 5i). Pre-existing macOS `du -sb` failure unchanged. |
||
|
Omar Sobh
|
f43b34ad15 |
Phase 2b: Blob RPC (Stat/Get/Put/LoadManifest)
Wires the Phase 2 content-addressed blob store onto the network via
four new methods on the existing RpcRouter. Combined with the mTLS +
gossip stack from Phase 1c-1e, peers can now exchange content-addressed
blobs over real QUIC. This is the substrate the fingerprint-keyed cargo
cache (Phase 5) sits on directly.
New methods:
- BlobStat (0x03): payload = 32-byte BlobId, reply = JSON BlobStat
- BlobGet (0x04): payload = 32-byte BlobId, reply = raw bytes
- BlobPut (0x05): payload = raw bytes, reply = 32-byte BlobId
- BlobLoadManifest (0x06): payload = 32-byte BlobId, reply = JSON manifest
New error codes:
- NotFound (0xf3) — the requested BlobId isn't in the local store
- InvalidRequest (0xf4) — e.g. non-32-byte payload for a hash-keyed method
- NotConfigured (0xf5) — Blob* called on a router without an attached store
Wire-format bump: MAX_MESSAGE_BYTES 16 KiB → 16 MiB so a single 4 MiB
chunk (plus JSON overhead) fits comfortably. Anything above 16 MiB
needs the streaming variants coming in Phase 2c.
Client helpers:
- call_blob_stat / call_blob_get / call_blob_put / call_blob_load_manifest
- All map ErrorCode::NotFound to Ok(None), other codes to Err.
- call_blob_put verifies the peer-assigned BlobId matches local blake3
hash of the payload — corruption or protocol drift surfaces
immediately instead of silently accepting a mismatched receipt.
Router changes:
- RpcRouter grows an Option<Arc<BlobStore>> via with_blob_store(store).
- handle() split into handle_outcome() → HandlerOutcome enum
{Reply(bytes) | Error(ErrorCode)} for cleaner control flow across
the growing method set.
Services / daemon:
- ClusterServices::start gains blob_store_root: Option<PathBuf>.
- ClusterConfig gains blob_store_root: Option<PathBuf>.
- daemon.rs reads it from cluster_cfg + passes through.
- New ClusterServices::blob_store_enabled() introspection.
Tests (11 new, all real — no mocks):
Router (7 new):
- method_round_trips_byte_encoding — updated for 6 methods
- error_code_describe_covers_all_variants — updated for 6 codes
- decode_error_covers_all_known_codes
- blob_rpcs_return_not_configured_without_store — all four Blob*
methods return NotConfigured when the router lacks a store
- blob_stat_returns_not_found_for_missing
- blob_stat_returns_json_for_existing
- blob_stat_returns_invalid_request_for_bad_length
- blob_get_returns_content_bytes
- blob_put_stores_bytes_and_returns_hash — verifies BlobId matches
independent local hash
- blob_load_manifest_returns_json_for_existing (2-chunk case)
- blob_load_manifest_returns_not_found_for_missing
End-to-end over real QUIC (2 new):
- end_to_end_blob_put_stat_get_over_real_quic — full 4-method loop
(Put → Stat → Get → LoadManifest) + NotFound path
- end_to_end_multi_chunk_blob_over_real_quic — 6 MiB blob → 2 chunks,
proves MAX_MESSAGE_BYTES bump took effect
Services (1 new):
- services_with_blob_store_serves_blob_rpc_end_to_end — cut CA, sign
leaves, config includes blob_store_root, start ClusterServices,
dial from B over persisted mTLS, put + get through the router,
then independently verify bytes landed on A's on-disk store
Also fixed a parallel-test port collision: services `next_port()`
now increments by 2 so `port + 1` (the RPC bind) is reserved
alongside `port` (the gossip bind).
128 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 1006
- cluster/services.rs: 535
- cluster/blob.rs: 802
- config.rs: 511
- daemon.rs: 265
Follow-on:
- 2c: streaming variants (AsyncRead/AsyncWrite) so a many-GB blob
transfers without holding it in memory
- 2d: chunk-level RPC (BlobPutChunk / BlobGetChunk) so a receiver
can request only chunks it's missing after LoadManifest
- Phase 5 (the killer feature) can now build on Phase 2b directly —
fingerprint the target dir, PutBlob the compressed tarball, and
next node calls GetBlob keyed by the same fingerprint hash.
|
||
|
Omar Sobh
|
916add37df |
Phase 1d: persistent NodeIdentity + FleetCa + fleet-ca CLI
Closes out Phase 1. A production operator can now cut a fleet CA,
sign per-node leaves, drop the resulting PEMs at
/etc/claw-store/tls/, point [cluster.tls] at them, and the daemon
loads real mTLS material on startup — no more ephemeral in-process
CA hack.
New public API in cluster::transport:
- FleetCa::generate(cn) — new self-signed root CA
- FleetCa::save(dir) / FleetCa::load(dir) — round-trip PEM
- FleetCa::sign_leaf(name) — mint an in-memory NodeIdentity
- FleetCa::sign_leaf_to_pem(name, out_dir) — write ca.crt + node.crt +
node.key (node.key at 0o600 on Unix)
- NodeIdentity::from_pem_files(ca, cert, key) — production load path
- NodeIdentity::from_pem_dir(dir) — canonical filename layout
- NodeIdentity::from_cluster_config(cfg) — pick up [cluster.tls] paths
Manual Debug for FleetCa redacts the private key.
Config extension:
- [cluster.tls] ca_cert / node_cert / node_key (all PathBuf).
- Optional at the top level; callers that need mTLS surface a clear
error when it's absent.
Deps:
- rcgen features += "x509-parser" (for FleetCa::load's from_ca_cert_pem).
- rustls-pemfile 2 (parse PEM back into DER for rustls).
CLI (new commands; short-circuit config load so they run on fresh
boxes without /etc/claw-store/config.toml):
- fleet-ca-init --dir <dir> [--cn <name>]
generates ca.crt + ca.key (both 0o600).
- fleet-ca-sign --ca-dir <dir> --node <name> --out-dir <dir>
writes ca.crt + node.crt + node.key (node.key at 0o600).
- cluster-ping now accepts --tls-dir <dir> to load persistent
NodeIdentity from disk (produced by fleet-ca-sign).
Tests (8 new, all real — no mocks, real filesystem, real TLS handshake):
- fleet_ca_rejects_empty_common_name
- fleet_ca_save_and_load_round_trip_preserves_signing (asserts 0o600
on ca.key)
- fleet_ca_load_errors_when_files_missing
- sign_leaf_to_pem_writes_all_three_files_with_correct_permissions
(asserts 0o600 on node.key)
- persistent_identity_round_trips_through_disk_and_pings — end-to-end:
cut CA on disk, reload it, sign two leaves via sign_leaf_to_pem,
reload them via from_pem_dir, run real QUIC ping/pong. This is the
operator flow.
- node_identity_from_cluster_config_errors_without_tls_section
- node_identity_from_cluster_config_loads_pem_paths
- from_pem_files_errors_on_missing_ca_file
80 tests pass. Pre-existing macOS-only hot test unchanged.
Also verified live CLI smoke test:
fleet-ca-init → ca.crt + ca.key at 0o600
fleet-ca-sign → ca.crt + node.crt + node.key at 0o600
Files parse as valid X.509.
File sizes (all under 1300-line ceiling):
- cluster/transport.rs: 916
- cluster/gossip.rs: 576
- cluster.rs: 275
- config.rs: 503
- main.rs: 662
Phase 1 complete. Next up:
- Phase 1e (daemon integration): gossip + QUIC RPC server wired into
claw-store daemon; hot-tier metrics periodically pushed; a real
PeerStatus RPC alongside ping.
- Phase 2: content-addressed blob store (BLAKE3 chunking, put/get).
|
||
|
Omar Sobh
|
4d93f955c5 |
Phase 1c: QUIC transport with fleet-CA mTLS + cluster-ping
Wraps quinn 0.11 in cluster/transport.rs with rustls 0.23 + rcgen 0.13 for identity management. Every peer connection is mTLS: both sides must present certs signed by the shared fleet CA, and rustls verifies the peer's cert SAN matches the requested server name. Public API on `cluster::transport`: - `NodeIdentity` (cert chain + private key + trusted CA) - `NodeIdentity::generate_test_pair(a, b)` — ephemeral CA + two signed leaves; shape matches the production PEM-file loader (Phase 1d) - `QuicServer::bind(addr, identity)` — bind mTLS-enforced listener - `QuicServer::accept()` — accept one connection (Option<Result<_>>) - `QuicClient::new(local_addr, identity)` — build client endpoint - `QuicClient::connect(peer_addr, expected_name)` — outbound with SAN check - `ping(&conn, payload)` — bidi stream RPC; server echoes as `pong:<payload>` - `ping_handler_loop(conn)` — server-side accept/echo forever - `CLAWSTOR_RPC_ALPN` constant, single ALPN "clawstor-rpc/1" Config extension: - `bind_rpc_lan`, `bind_rpc_tailscale` on `ClusterConfig` (both optional). - Defaults: gossip port + 1 (so gossip UDP and QUIC UDP don't collide). - Helper: `ClusterConfig::rpc_lan()`, `rpc_tailscale()`, `PeerEntry::rpc_lan()`, `rpc_tailscale()`. Gossip.rs now advertises the RPC address, not the gossip address, on the well-known `clawstor.rpc.lan` / `clawstor.rpc.tailscale` keys. CLI: - `cluster-ping --name <me> --peer <you> --rpc-addr <addr> [--payload X]` Runs a full mTLS handshake and single ping. For dev/loopback use today (both sides need to share a CA); persistent-identity ping lands in Phase 1d. Tests (6 new, real UDP + TLS handshake, no mocks): - generate_test_pair produces two distinct leaves that share a CA - ping_pong_between_two_mtls_peers: real 2-node QUIC round trip with full mTLS chain verification, `open_bi()`/`accept_bi()`, byte-exact response check - client_rejects_peer_with_wrong_ca: TLS chain verification failure when the server presents a cert signed by a different CA - client_rejects_wrong_server_name: SAN mismatch is enforced - server_binds_wildcard_and_reports_concrete_local_addr: port 0 → real - ping_rejects_oversize_payload: MAX_MESSAGE_BYTES cap enforced client-side Fixed-port allocator range 42000+ so transport tests don't conflict with gossip tests (41000+). 72 tests pass. Pre-existing `hot::test_project_target_size_bytes` macOS-only failure unchanged. File sizes (all under 1300-line ceiling): - cluster/transport.rs: 485 - cluster/gossip.rs: 570 - cluster.rs: 275 - config.rs: 480 - main.rs: 564 Follow-on Phase 1 cut (1d): - Load NodeIdentity from persistent PEM files at /etc/claw-store/tls/ - Fleet CA bootstrap ceremony (rcgen → write CA cert; per-node leaf CSR) - Daemon-level RPC server that runs alongside gossip + serves real operations (blob get/put, metadata sync) - cluster-status reads from live daemon via API instead of standalone |
||
|
Omar Sobh
|
5c19c60292 |
Phase 1a: cluster module + LAN-first peer probe
First cut of the v2 distributed FS. Captures the architecture design in ARCHITECTURE-v2.md and lands the smallest useful new capability: probing cluster peers with a LAN-first policy so subsequent transport + gossip layers (Phase 1b, 1c) can build on a real routing decision. New: - ARCHITECTURE-v2.md: zones (fabric-10g/lan-1g/roaming), tier lifecycle (hot/warm/cold), fingerprint-keyed build cache design, smart-clean policy, phase plan, explicit non-goals. - claw-store/src/cluster.rs: RouteKind, RouteWinner, LanFirstProbe. LAN 200ms timeout, Tailscale 500ms fallback. 8 tests use real TCP listeners on 127.0.0.1 (no mocks); cover happy path, fall-through, both-fail, single-address, and elapsed reporting. - claw-store/src/config.rs: ClusterConfig + PeerEntry with validation (bind-address presence, no duplicate peer names, per-peer reachable address required). Optional at top level so pre-v2 configs still load unchanged. 6 new tests. - claw-store/src/main.rs: `claw-store cluster-probe <peer>` CLI subcommand that reads config, resolves the peer, probes, prints the winning route + elapsed time. All 16 new tests pass. Existing 45 pass. Sole failure (hot::tests::test_project_target_size_bytes) is a pre-existing macOS-only issue with `du -sb`; Linux CI unaffected. Follow-on Phase 1 cuts (subsequent sessions): - 1b: chitchat SWIM gossip for live membership state - 1c: quinn QUIC transport with fleet-CA mTLS - 1d: `claw-store cluster status` — live membership view Every file well under the 1300-line ceiling (cluster.rs 268, config.rs 428, main.rs 450). |
||
|
Omar SobhandClaude Sonnet 4.6
|
1b1d66eac9 |
feat(v0.3.0): weekly snapshots, incremental replication, auth, uptime, hardening
- snapshot: implement weekly ZFS snapshots (Sunday midnight, retain_weekly config) - snapshot: incremental cold replication via zfs send -i; tracks last replicated snapshot in /var/lib/claw-store/last-replicated-snapshot - daemon: write start-time file for uptime reporting; add SIGTERM graceful shutdown - serve: daemon_uptime_secs now reads the start-time file (was hardcoded 0) - serve: validate project names in POST handlers (org/repo slug, no traversal) - serve: Bearer token auth middleware on all POST endpoints via cfg.api_token - config: add optional api_token field (backward compatible, defaults to None) - sync: add SSH timeouts (ConnectTimeout=10, ServerAliveInterval=5) to peer notify - hot/sync/main: replace unwrap() on path-to-str with proper anyhow errors - config/tank.toml: document 10G fabric IP and nightly_at field intent - .gitignore: exclude compiled dashboard assets (claw-store/static/) - Makefile: add build, install, install-systemd, install-dashboard, deploy targets - tests: 29 passing (up from 25); 4 new weekly snapshot tests Co-Authored-By: Claude Sonnet 4.6 <[email protected]> |
||
|
Omar SobhandClaude Sonnet 4.6
|
a9c12173fe |
feat: activate/deactivate/list/sync/pull commands + peer sync queue
- activate <org/repo>: wire hot tier, write .cargo/config.toml, register - deactivate <org/repo>: auto-sync to peer, evict hot tier, unregister - list: show all warm repos with activation status and last-active time - sync <org/repo>: git push origin then SSH peer to pull - pull <org/repo>: git pull from origin, stamp last_sync in manifest - SyncQueue: file-based retry queue (/var/lib/claw-store/sync-queue.toml) - daemon: drains sync queue on every 5-min poll tick - config: [peer] host/user for cross-node notification - manifest: Project.name is now org/repo, adds last_sync field - hot tier paths are org/repo-namespaced to avoid collisions Co-Authored-By: Claude Sonnet 4.6 <[email protected]> |
||
|
Omar SobhandClaude Sonnet 4.6
|
8ab3bb2d12 |
feat: config types and loader
Co-Authored-By: Claude Sonnet 4.6 <[email protected]> |