Commit Graph
86 Commits
Author SHA1 Message Date
Omar Sobh 2ae079fbd4 Phase 1e: daemon-integrated cluster services + PeerStatus RPC
Ties Phase 1a-1d together into a live subsystem the daemon actually
runs. When `[cluster]` is present in config, `claw-store daemon` now:

  1. Starts chitchat gossip via ClusterServices::start.
  2. Publishes hot.max_gb immediately and re-measures the hot dir
     every 30s, updating `clawstor.hot.used`.
  3. If `[cluster.tls]` is also configured, loads NodeIdentity from
     PEM files and binds a QUIC RPC server that accepts + dispatches
     incoming Ping / PeerStatus requests via RpcRouter.

New modules:

- cluster/rpc.rs (510 lines):
  - Method enum (Ping=0x01, PeerStatus=0x02)
  - ErrorCode enum (EmptyRequest / UnknownMethod / HandlerFailure)
  - PeerStatusReply { local_name, local_zone, peers: Vec<PeerView> }
  - RpcRouter — dispatch state (holds Arc<ClusterGossip> + local
    name/zone)
  - rpc_call / call_ping / call_peer_status — client helpers
  - serve_connection — server accept-bidi loop
  - Wire format: `method:u8 || payload:bytes` → `reply:bytes` or
    single-byte ErrorCode

- cluster/services.rs (418 lines):
  - ClusterServices { gossip, accept_task, metric_task }
  - start(cluster_cfg, name, hot_dir, hot_max_bytes) → bootstraps
    everything above
  - shutdown() aborts background tasks cleanly
  - Recursive dir-walker (spawn_blocking) for hot-tier metric

PeerView now derives Serialize/Deserialize so it round-trips through
JSON over the RPC.

daemon.rs integration (~35 lines added):
  - Bootstraps ClusterServices before entering the select loop
  - Held for daemon lifetime
  - Shutdown on SIGTERM
  - Gossip-less config still runs standalone (backwards compat)

CLI: `cluster-peer-status --peer <name> --rpc-addr <addr>
                          --tls-dir <dir>`
  Loads a persistent identity, dials the peer, calls PeerStatus,
  prints the peer's local view as a table.

Tests (15 new, all real — no mocks, real UDP + TLS + JSON round trip):

RPC (9):
- method_round_trips_byte_encoding
- dispatch_returns_pong_for_ping
- dispatch_returns_json_for_peer_status
- dispatch_returns_empty_request_error
- dispatch_returns_unknown_method_error
- rpc_call_rejects_oversize_payload (with real quinn connection)
- end_to_end_ping_and_peer_status_over_real_quic — full 2-node quinn
  with mTLS + both RPCs
- peer_status_reflects_peer_gossip_state — 2 gossip instances converge,
  RPC caller from a third identity sees the converged view
- error_code_describe_covers_all_variants

Services (6):
- dir_bytes_sync_returns_zero_for_missing_path
- dir_bytes_sync_sums_recursive_file_sizes (3-level nesting)
- services_start_without_tls_leaves_rpc_disabled
- services_start_with_tls_serves_rpc_end_to_end — full stack: cut CA on
  disk, sign leaves, start ClusterServices for A, dial from B via
  persisted mTLS, run ping + PeerStatus over the wire
- services_publish_hot_used_metric_periodically
- services_gossip_sees_peer_after_convergence

95 tests pass. Pre-existing macOS-only failure unchanged.

File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 510
- cluster/services.rs: 418
- cluster/gossip.rs: 579
- cluster/transport.rs: 916
- daemon.rs: 262
- main.rs: 749

Phase 1 done end-to-end. `claw-store daemon` now boots a real distributed
cluster stack when config asks for one; peers can call each other's
PeerStatus RPC and see live membership. Next: Phase 2 (content-addressed
blob store) can hook new RPC methods into the same RpcRouter with a
one-line dispatch arm.
2026-07-11 22:15:41 -07:00
Omar Sobh 916add37df Phase 1d: persistent NodeIdentity + FleetCa + fleet-ca CLI
Closes out Phase 1. A production operator can now cut a fleet CA,
sign per-node leaves, drop the resulting PEMs at
/etc/claw-store/tls/, point [cluster.tls] at them, and the daemon
loads real mTLS material on startup — no more ephemeral in-process
CA hack.

New public API in cluster::transport:
- FleetCa::generate(cn) — new self-signed root CA
- FleetCa::save(dir) / FleetCa::load(dir) — round-trip PEM
- FleetCa::sign_leaf(name) — mint an in-memory NodeIdentity
- FleetCa::sign_leaf_to_pem(name, out_dir) — write ca.crt + node.crt +
  node.key (node.key at 0o600 on Unix)
- NodeIdentity::from_pem_files(ca, cert, key) — production load path
- NodeIdentity::from_pem_dir(dir) — canonical filename layout
- NodeIdentity::from_cluster_config(cfg) — pick up [cluster.tls] paths

Manual Debug for FleetCa redacts the private key.

Config extension:
- [cluster.tls] ca_cert / node_cert / node_key (all PathBuf).
- Optional at the top level; callers that need mTLS surface a clear
  error when it's absent.

Deps:
- rcgen features += "x509-parser" (for FleetCa::load's from_ca_cert_pem).
- rustls-pemfile 2 (parse PEM back into DER for rustls).

CLI (new commands; short-circuit config load so they run on fresh
boxes without /etc/claw-store/config.toml):
- fleet-ca-init --dir <dir> [--cn <name>]
    generates ca.crt + ca.key (both 0o600).
- fleet-ca-sign --ca-dir <dir> --node <name> --out-dir <dir>
    writes ca.crt + node.crt + node.key (node.key at 0o600).
- cluster-ping now accepts --tls-dir <dir> to load persistent
  NodeIdentity from disk (produced by fleet-ca-sign).

Tests (8 new, all real — no mocks, real filesystem, real TLS handshake):
- fleet_ca_rejects_empty_common_name
- fleet_ca_save_and_load_round_trip_preserves_signing (asserts 0o600
  on ca.key)
- fleet_ca_load_errors_when_files_missing
- sign_leaf_to_pem_writes_all_three_files_with_correct_permissions
  (asserts 0o600 on node.key)
- persistent_identity_round_trips_through_disk_and_pings — end-to-end:
  cut CA on disk, reload it, sign two leaves via sign_leaf_to_pem,
  reload them via from_pem_dir, run real QUIC ping/pong. This is the
  operator flow.
- node_identity_from_cluster_config_errors_without_tls_section
- node_identity_from_cluster_config_loads_pem_paths
- from_pem_files_errors_on_missing_ca_file

80 tests pass. Pre-existing macOS-only hot test unchanged.

Also verified live CLI smoke test:
  fleet-ca-init → ca.crt + ca.key at 0o600
  fleet-ca-sign → ca.crt + node.crt + node.key at 0o600
  Files parse as valid X.509.

File sizes (all under 1300-line ceiling):
- cluster/transport.rs: 916
- cluster/gossip.rs: 576
- cluster.rs: 275
- config.rs: 503
- main.rs: 662

Phase 1 complete. Next up:
- Phase 1e (daemon integration): gossip + QUIC RPC server wired into
  claw-store daemon; hot-tier metrics periodically pushed; a real
  PeerStatus RPC alongside ping.
- Phase 2: content-addressed blob store (BLAKE3 chunking, put/get).
2026-07-11 22:04:54 -07:00
Omar Sobh 4d93f955c5 Phase 1c: QUIC transport with fleet-CA mTLS + cluster-ping
Wraps quinn 0.11 in cluster/transport.rs with rustls 0.23 + rcgen 0.13
for identity management. Every peer connection is mTLS: both sides must
present certs signed by the shared fleet CA, and rustls verifies the
peer's cert SAN matches the requested server name.

Public API on `cluster::transport`:
- `NodeIdentity` (cert chain + private key + trusted CA)
- `NodeIdentity::generate_test_pair(a, b)` — ephemeral CA + two signed
  leaves; shape matches the production PEM-file loader (Phase 1d)
- `QuicServer::bind(addr, identity)` — bind mTLS-enforced listener
- `QuicServer::accept()` — accept one connection (Option<Result<_>>)
- `QuicClient::new(local_addr, identity)` — build client endpoint
- `QuicClient::connect(peer_addr, expected_name)` — outbound with SAN check
- `ping(&conn, payload)` — bidi stream RPC; server echoes as `pong:<payload>`
- `ping_handler_loop(conn)` — server-side accept/echo forever
- `CLAWSTOR_RPC_ALPN` constant, single ALPN "clawstor-rpc/1"

Config extension:
- `bind_rpc_lan`, `bind_rpc_tailscale` on `ClusterConfig` (both optional).
- Defaults: gossip port + 1 (so gossip UDP and QUIC UDP don't collide).
- Helper: `ClusterConfig::rpc_lan()`, `rpc_tailscale()`,
  `PeerEntry::rpc_lan()`, `rpc_tailscale()`.

Gossip.rs now advertises the RPC address, not the gossip address, on
the well-known `clawstor.rpc.lan` / `clawstor.rpc.tailscale` keys.

CLI:
- `cluster-ping --name <me> --peer <you> --rpc-addr <addr> [--payload X]`
  Runs a full mTLS handshake and single ping. For dev/loopback use
  today (both sides need to share a CA); persistent-identity ping
  lands in Phase 1d.

Tests (6 new, real UDP + TLS handshake, no mocks):
- generate_test_pair produces two distinct leaves that share a CA
- ping_pong_between_two_mtls_peers: real 2-node QUIC round trip with
  full mTLS chain verification, `open_bi()`/`accept_bi()`, byte-exact
  response check
- client_rejects_peer_with_wrong_ca: TLS chain verification failure
  when the server presents a cert signed by a different CA
- client_rejects_wrong_server_name: SAN mismatch is enforced
- server_binds_wildcard_and_reports_concrete_local_addr: port 0 → real
- ping_rejects_oversize_payload: MAX_MESSAGE_BYTES cap enforced client-side

Fixed-port allocator range 42000+ so transport tests don't conflict
with gossip tests (41000+).

72 tests pass. Pre-existing `hot::test_project_target_size_bytes`
macOS-only failure unchanged.

File sizes (all under 1300-line ceiling):
- cluster/transport.rs: 485
- cluster/gossip.rs: 570
- cluster.rs: 275
- config.rs: 480
- main.rs: 564

Follow-on Phase 1 cut (1d):
- Load NodeIdentity from persistent PEM files at /etc/claw-store/tls/
- Fleet CA bootstrap ceremony (rcgen → write CA cert; per-node leaf CSR)
- Daemon-level RPC server that runs alongside gossip + serves real
  operations (blob get/put, metadata sync)
- cluster-status reads from live daemon via API instead of standalone
2026-07-11 21:50:28 -07:00
Omar Sobh 5283ab8655 Phase 1b: chitchat SWIM gossip layer + cluster-status CLI
Wraps chitchat 0.11 in cluster/gossip.rs. Each node publishes its
zone, RPC endpoints, hot-tier usage, warm-tier project list, and
uptime under well-known kv keys; peers propagate this via gossip and
are classified alive/dead by phi-accrual failure detection.

Bootstraps from static [[cluster.peers]] seed nodes in the config.
Membership extends dynamically as nodes join or leave.

Public API on `ClusterGossip`:
- bootstrap(cluster_cfg, local_name) — start UDP gossip service
- set(key, value), set_hot_used, set_hot_max, set_warm_projects
- peers() — all known peers with typed PeerView + liveness
- peer(name) — one peer by advertised name
- peers_in_zone(zone) — live peers filtered by zone
- shutdown() — abort gossip task

CLI:
- `claw-store cluster-status --name <me> [--wait-secs N]`
  bootstraps gossip, waits for convergence, prints a peer table
  with zone, alive/dead, RPC LAN/Tailscale addresses, hot usage.

Tests (7 new):
- bootstrap_publishes_our_own_state — self-ID + advertised address
- bootstrap_fails_when_local_name_empty
- bootstrap_fails_when_no_bind_address — cluster.validate error chain
- two_node_cluster_converges_and_shares_state — REAL two-node
  in-process UDP gossip. Publish state on A, wait for phi-accrual
  liveness on B, verify all fields propagated (zone, rpc_lan, hot
  bytes, warm_projects list, fill ratio, zone filtering).
- peers_excludes_self — solo cluster reports zero peers
- peer_view_hot_fill_ratio_handles_missing_or_zero

No mocks. Real UDP transport, real chitchat runtime, real gossip
protocol. Ports allocated from a fixed range (41001+) via atomic
counter — deterministic and race-free within the test binary.

66 tests pass. Pre-existing `hot::test_project_target_size_bytes`
macOS-only failure unchanged.

File sizes (under 1300-line ceiling):
- cluster/gossip.rs: 549
- cluster.rs: 274
- main.rs: 506

Follow-on Phase 1 cuts:
- 1c: `quinn` QUIC transport + fleet-CA mTLS for RPC (separate UDP
      port so gossip and QUIC don't collide)
- 1d: daemon wire-up — gossip runs in background, hot-tier usage
      pushed periodically, `cluster-status` reads from live daemon
2026-07-11 21:41:52 -07:00
Omar Sobh 5c19c60292 Phase 1a: cluster module + LAN-first peer probe
First cut of the v2 distributed FS. Captures the architecture design
in ARCHITECTURE-v2.md and lands the smallest useful new capability:
probing cluster peers with a LAN-first policy so subsequent transport
+ gossip layers (Phase 1b, 1c) can build on a real routing decision.

New:
- ARCHITECTURE-v2.md: zones (fabric-10g/lan-1g/roaming), tier
  lifecycle (hot/warm/cold), fingerprint-keyed build cache design,
  smart-clean policy, phase plan, explicit non-goals.
- claw-store/src/cluster.rs: RouteKind, RouteWinner, LanFirstProbe.
  LAN 200ms timeout, Tailscale 500ms fallback. 8 tests use real
  TCP listeners on 127.0.0.1 (no mocks); cover happy path,
  fall-through, both-fail, single-address, and elapsed reporting.
- claw-store/src/config.rs: ClusterConfig + PeerEntry with
  validation (bind-address presence, no duplicate peer names,
  per-peer reachable address required). Optional at top level so
  pre-v2 configs still load unchanged. 6 new tests.
- claw-store/src/main.rs: `claw-store cluster-probe <peer>` CLI
  subcommand that reads config, resolves the peer, probes, prints
  the winning route + elapsed time.

All 16 new tests pass. Existing 45 pass. Sole failure
(hot::tests::test_project_target_size_bytes) is a pre-existing
macOS-only issue with `du -sb`; Linux CI unaffected.

Follow-on Phase 1 cuts (subsequent sessions):
- 1b: chitchat SWIM gossip for live membership state
- 1c: quinn QUIC transport with fleet-CA mTLS
- 1d: `claw-store cluster status` — live membership view

Every file well under the 1300-line ceiling
(cluster.rs 268, config.rs 428, main.rs 450).
2026-07-11 21:30:50 -07:00
Omar Sobh aefa1cce58 URGENT FIX: stale-gc must not treat last_active=None as stale
Previous commit 643ba17 unmasked a latent bug in the None-handling
branch of both gc_stale_targets AND the new stale_project_names:
they treated 'no last_active timestamp' as 'stale, evict'. This
combined with the new proactive sweep (fires every tick, not just
under space pressure) meant daemon restart wiped runtime state,
saw every project as None-timestamped, and mass-deactivated
everything active on the next tick.

Real damage in this session on live daemons:
  * architect: clawverse/omni-cortex — 134 GB hot artifacts freed
  * tank: clawverse/omni-cortex (44 GB), rustyverse/rustytorch (4 GB),
    plus ~10 more with empty hot targets

Warm clones under /slab/projects are intact (per deactivate flow) —
impact is only rebuild cost on next activation.

Fix: None is treated as NOT stale in both places. Absence of a
timestamp is normal: update_active_projects only stamps projects
whose cargo/rustc it catches mid-run. A freshly-activated project
with nothing built yet, or a project whose builds all finished
between poll ticks, will have None. That's not stale — that's
'we haven't seen it hit the threshold'. Staleness must always be
a positive assertion.

Added test test_gc_skips_none_last_active to lock the invariant.
2026-07-02 17:00:58 -07:00
Omar Sobh 643ba170b6 daemon: proactively deactivate stale (>48h idle) projects each tick
Answers a real operational question — 'why does /hot/targets stay near
full even when I haven't touched most of these projects for weeks?'
Previously the stale sweep was gated behind 'hot tier > 90% full', so
an idle project held NVMe until the operator manually deactivated it
or space ran out and LRU came for it.

The change:
  * new claw-store/src/actions.rs — shared deactivate_project(cfg,
    manifest_path, project) with the FULL flow: sync-to-peer +
    hot rm + .cargo/config.toml removal + manifest update, plus a
    DeactivateOutcome { synced, sync_error, freed_bytes } return.
    Extracted from main.rs::cmd_deactivate so both CLI and daemon
    run the same code — no drift between manual and auto semantics.
  * daemon.rs poll_tick now runs the stale sweep on EVERY tick,
    independent of space pressure. Each project idle > stale_hours
    (48 by config) goes through the full deactivate, logs
    'stale-gc: X freed N MB (synced=Y)'.
  * gc_by_space (LRU) still gates on >90% full — it's the emergency
    'even after the stale sweep we're still tight' path.
  * main.rs cmd_deactivate is now a thin CLI wrapper that adds
    println! feedback + reports freed MB in the terminal output.

Space-pressure LRU keeps its original .cargo/config.toml-preserving
semantics (rm hot only, not the shim) so a project marked 'active'
by manual activation but hard-evicted for space can still be
re-activated cheaply. Stale sweep is the aggressive one because the
project genuinely hasn't been touched.

Tests: two new unit tests in actions.rs cover the full flow +
idempotency; the freed_bytes assertion is Linux-only-safe (BSD du
returns 0 for -sb, same limitation as the existing hot test).
2026-07-02 16:57:44 -07:00
Omar Sobh 92b2dd751e daemon: auto-enqueue sync when a project's git HEAD moves
Removes the biggest workflow footgun in the architect↔tank flow: the
operator no longer has to remember to run 'claw-store sync <project>'.
An ordinary 'git commit' anywhere under /slab/projects/*/* is now
picked up automatically on the next 5-min poll tick and the peer is
notified — same code path deactivate/sync used to trigger by hand.

New module claw-store/src/head_watch.rs owns the state:
  * HeadCache — {project → last-seen HEAD sha}, persisted at
    /var/lib/claw-store/head-cache.toml (same directory as the
    existing sync queue and manifest).
  * scan_and_enqueue walks warm_root/<org>/<repo>, calls
    'git rev-parse HEAD' on each, compares to cache, enqueues on
    change. Non-git dirs, symlinks, and unreadable entries are
    skipped (matches the project-list walker's behavior).

First-scan policy: if the cache is empty on startup we populate it
silently instead of enqueueing every project — otherwise a fresh
install with 55 warm projects would flood the peer with 55 syncs on
the first tick. Existing state gets baselined; only movement from
that point on triggers work.

daemon.rs poll_tick was one 'retry the queue' block; it's now
'scan for HEAD changes → save cache → drain queue' in that order,
so a commit from between ticks gets enqueued AND drained the
same tick.

Four unit tests cover the invariants: silent first scan, enqueue
on real HEAD move (only for the moved repo), cache roundtrip,
non-git directory skip.
2026-07-02 16:15:16 -07:00
Omar Sobh 29a6616c59 sync: never silently drop; branch-aware pull; expose queue depth in /api/status
Three fixes bundled — all defensive around the architect↔tank dev flow:

1. drain_sync_queue no longer drops jobs after 48 attempts. The
   previous cutoff (~4h at 5-min drain intervals) silently lost every
   pending sync whenever the peer was down longer than that — the
   exact scenario the user hit last week when architect was offline.
   Now jobs are retried indefinitely; past SYNC_STUCK_ATTEMPTS (12,
   ~1h) they escalate from WARN to ERROR and appear in /api/status
   under sync_queue_stuck so the operator sees them.

2. pull_project is now branch-aware. Old behavior was 'git pull
   --ff-only' on whatever branch happened to be checked out. If the
   peer was on main but the sender committed on develop, the peer
   never picked up the new branch's ref. New behavior:
     - git fetch --all --prune --tags first (gets every branch)
     - fast-forward the checked-out branch if it has an upstream
     - fast-forward every OTHER branch that has an upstream via
       update-ref, without switching HEAD — so main can advance while
       the operator is off hacking on feature/x.
   Diverged branches are left alone (loud error, not silent merge).
   Detached HEAD is skipped after fetch.

3. NodeStatus gains sync_queue_depth + sync_queue_stuck fields, wired
   into both handle_status (GET /api/status) and the SSE stream. The
   dashboard now has ground truth for 'is anything backing up?'

Tests updated: the old test_sync_queue_drops_after_max_attempts is
replaced by test_sync_queue_never_drops_stuck_jobs (invariant: never
drops) and test_sync_queue_stuck_count_threshold (invariant: counts
jobs past the escalation threshold).
2026-07-02 14:19:09 -07:00
Omar SobhandClaude Sonnet 4.6 720b331527 fix: skip all symlinks in project list walker
Replace the ends_with(".git") name check with a file_type().is_symlink()
check at both the org and repo levels of build_projects_list.

This handles three classes of symlink that all produced duplicate entries:
  1. Gitea's repo.git → repo aliases (previously caught by name check)
  2. Within-org shortcuts: quantumclaw/bbq → quantum-bbq
  3. Cross-org aliases:   clawverse/clawhdf5 → ../quantumclaw/clawhdf5

The name check only caught class 1; the symlink check catches all three.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-30 11:24:16 +00:00
Omar SobhandClaude Sonnet 4.6 f6ff04e9dc fix: skip *.git symlinks in project list scanner
Gitea creates a repo.git → repo symlink alongside every real checkout
in /slab/projects. The directory walker resolved both, found .git inside
each (since the symlink points at the real dir), and emitted a duplicate
entry for every project — e.g. quantumclaw/clawhdf5 and
quantumclaw/clawhdf5.git both appeared in the dashboard.

Add ends_with(".git") guard at both the org and repo levels of
build_projects_list so symlinked aliases are always skipped.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-30 10:56:54 +00:00
Omar SobhandClaude Sonnet 4.6 1320d2df9e feat: v0.3.0 — HTTP handler tests + README
- Extract build_app() from run_server() so tests can construct the
  router without binding a port (tower::ServiceExt::oneshot pattern)
- Add 10 new tests in serve::tests: validate_project_name (valid +
  invalid slugs), daemon_uptime when file absent, GET /api/status,
  GET /api/projects, POST /api/activate with invalid name, auth
  middleware (no header → 401, wrong token → 401, correct token →
  pass-through, GET bypasses auth entirely)
- Add tower + http-body-util to dev-dependencies
- Add project README covering architecture, install (Makefile),
  config options, SSH setup, snapshot schedule, CLI reference,
  HTTP API table, pinning, and troubleshooting guide
- Bump version to 0.3.0

Test suite: 39 tests, 0 failures

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-30 10:51:52 +00:00
Omar SobhandClaude Sonnet 4.6 39ddac431b chore: stop tracking compiled dashboard assets
claw-store/static/ is now in .gitignore. The bundle is installed via
`make install-dashboard` which runs `npm run build` and copies to
/usr/share/claw-store/static — it should not live in the repo.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-30 10:43:49 +00:00
Omar SobhandClaude Sonnet 4.6 1b1d66eac9 feat(v0.3.0): weekly snapshots, incremental replication, auth, uptime, hardening
- snapshot: implement weekly ZFS snapshots (Sunday midnight, retain_weekly config)
- snapshot: incremental cold replication via zfs send -i; tracks last replicated
  snapshot in /var/lib/claw-store/last-replicated-snapshot
- daemon: write start-time file for uptime reporting; add SIGTERM graceful shutdown
- serve: daemon_uptime_secs now reads the start-time file (was hardcoded 0)
- serve: validate project names in POST handlers (org/repo slug, no traversal)
- serve: Bearer token auth middleware on all POST endpoints via cfg.api_token
- config: add optional api_token field (backward compatible, defaults to None)
- sync: add SSH timeouts (ConnectTimeout=10, ServerAliveInterval=5) to peer notify
- hot/sync/main: replace unwrap() on path-to-str with proper anyhow errors
- config/tank.toml: document 10G fabric IP and nightly_at field intent
- .gitignore: exclude compiled dashboard assets (claw-store/static/)
- Makefile: add build, install, install-systemd, install-dashboard, deploy targets
- tests: 29 passing (up from 25); 4 new weekly snapshot tests

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-30 10:41:20 +00:00
Omar SobhandClaude Opus 4.7 d076fac619 fix(serve): single-source the project list — stops dashboard flashing
The HTTP `/api/projects` handler and the SSE `/api/events` stream each
built their own ProjectInfo list — independent inline code, ~50 lines
of mostly identical duplicate.  But the two diverged on ONE field:

  /api/projects: is_active = active.is_some() || has_hot_cargo_config(...)
  /api/events:   is_active = active.is_some()

The dashboard hits BOTH every few seconds (a 10-second setInterval
polling /api/projects + a 5-second SSE pushing events).  For any
project that was hot-wired via `.cargo/config.toml` but not in the
manifest, the two endpoints disagreed about its `is_active` bit.
The dashboard's last-write-won, and the row visibly flickered as
the active flag toggled.  The 'X active' count in the header bounced
along with it.

Extracted `build_projects_list(cfg, &manifest)` and call it from both
sides.  Same answer in both code paths.  Verified post-deploy:

  architect: HTTP=59 active, SSE=59 active (identical)
  tank:      HTTP=57 active, SSE=57 active (identical)

This also makes the 'total' number stable.  The 354 in the dashboard
header is correct semantics — discoverable git repos on disk — but
the label is misleading.  Cosmetic cleanup for a follow-up:

  - "Projects — N discovered · M active · K shown" reads truer than
    "N shown · M active · K total"
  - The status card's `active_project_count` (manifest.projects.len)
    is a third meaning of 'active' that doesn't match either dashboard
    number; consider renaming to `manifest_size` to disambiguate.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-06-28 13:23:44 +00:00
Omar SobhandClaude Opus 4.7 af50adec19 feat(v0.2.0): atomic+flocked manifest, pinned projects, serve systemd unit
The two biggest pain points coming out of the architecture review:

  1. The manifest at /var/lib/claw-store/projects.toml was the only
     piece of writable state but had no locking, no atomic writes,
     and three concurrent writers (daemon poll tick, every CLI verb,
     and the dashboard shelling out via /api/activate). Two writers
     interleaving silently dropped one of them; a crash mid-write
     left a corrupt half-written TOML that the next reader parsed
     as an empty manifest.

  2. Reboot survival: the dashboard had no systemd unit and was a
     stray hand-launched process. Architect lost its dashboard on
     todays reboot.

This commit lands:

- Manifest::update(path, FnOnce(&mut Manifest)) — locked-atomic
  load-mutate-save in one transaction. Uses libc::flock(LOCK_EX) on
  a sidecar .lock file (so the data file can be replaced by rename
  without invalidating the lock) and tempfile + persist for the
  rename. Concurrent writers serialise; readers see the previous
  state or the new state, never a torn write. Manifest::load uses
  LOCK_SH so it never races a mid-rename.

- Project.pinned: bool with #[serde(default)] so legacy manifests
  parse cleanly. hot::gc_stale_targets and hot::gc_by_space both
  skip pinned projects, with a WARN log when every remaining
  project is pinned but were still over budget — operator intent
  beats space pressure.

- claw-store pin <project> / unpin <project> CLI verbs.
  status command surfaces pin marker (📌).
  activate preserves an existing rows pinned flag so re-activating
  doesnt silently unpin.

- claw-store-serve.service systemd unit. Type=simple, Restart=
  on-failure, RestartSec=15, ProtectSystem=strict + ReadWritePaths
  =/var/lib/claw-store, ProtectHome, NoNewPrivileges, PrivateTmp.

- daemon poll tick reloads the manifest from disk at the start of
  each cycle (so CLI activations between ticks are visible) and
  routes its GC write through Manifest::update (so it cant race a
  concurrent CLI pin).

- libc + tempfile move from dev-deps into runtime deps.
- empty-manifest fallthrough on load (treat "" as default) so a
  half-written tempfile crashed pre-rename doesnt hard-fail the
  daemon next boot.

- 25 tests passing incl. new ones: legacy-toml-parses, update-
  serializes-two-sequential-writers, pinned-survives-stale-gc,
  pinned-survives-space-gc-even-when-lru.

Version bumped 0.1.0 → 0.2.0.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-06-28 12:59:41 +00:00
Omar SobhandClaude Opus 4.7 2fcfb16160 feat(serve): HTTP+SSE API + static dashboard bundle
Adds the serve subcommand: an axum 0.7 + tower-http server on
:7700 backing the React dashboard. Routes:

  GET  /api/status   /api/projects   /api/snapshots
       /api/sync-queue   /api/hot   /api/events (SSE 5s tick)
  POST /api/activate /api/deactivate /api/sync
       /api/gc /api/snapshot

Mutating endpoints shell out to /usr/local/bin/claw-store so
all CLI logic stays single-sourced. claw-store/static/ holds
the prebuilt dashboard Vite bundle for --static-dir.

cargo_init.rs gets a minor wiring tweak so the API can read
back the managed cargo config marker.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-06-28 12:52:46 +00:00
Omar SobhandClaude Sonnet 4.6 8b4d0088ca fix: list command only shows git repos, not org subdirs
Skip non-git directories inside org folders so flat layout dirs like
claw-store/target or claw-store/.cargo don't appear as fake repos.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 11:50:02 +00:00
Omar SobhandClaude Sonnet 4.6 a9c12173fe feat: activate/deactivate/list/sync/pull commands + peer sync queue
- activate <org/repo>: wire hot tier, write .cargo/config.toml, register
- deactivate <org/repo>: auto-sync to peer, evict hot tier, unregister
- list: show all warm repos with activation status and last-active time
- sync <org/repo>: git push origin then SSH peer to pull
- pull <org/repo>: git pull from origin, stamp last_sync in manifest
- SyncQueue: file-based retry queue (/var/lib/claw-store/sync-queue.toml)
- daemon: drains sync queue on every 5-min poll tick
- config: [peer] host/user for cross-node notification
- manifest: Project.name is now org/repo, adds last_sync field
- hot tier paths are org/repo-namespaced to avoid collisions

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 11:45:04 +00:00
Omar Sobh 3fb108edfc fix: manifest path to /var/lib/claw-store (user-writable) 2026-06-16 04:00:31 +00:00
Omar Sobh ac3079b56c fix: SystemZfs uses sudo for write ops (snapshot/destroy/clone/send) 2026-06-16 03:58:39 +00:00
Omar Sobh 7ae1c4f950 feat: daemon event loop and full CLI (init, status, gc, snapshot, restore, replicate) 2026-06-16 03:55:22 +00:00
Omar Sobh 11c32df3cd merge: restore.rs 2026-06-16 03:53:19 +00:00
Omar Sobh 426323723c merge: snapshot.rs 2026-06-16 03:53:19 +00:00
Omar Sobh 5f032bae25 merge: hot.rs 2026-06-16 03:53:19 +00:00
Omar Sobh 8cd8ce9d16 merge: zfs.rs 2026-06-16 03:53:19 +00:00
Omar Sobh c20fdcb484 merge: manifest.rs 2026-06-16 03:53:19 +00:00
Omar Sobh fbd8840a70 merge: config.rs 2026-06-16 03:53:19 +00:00
Omar SobhandClaude Sonnet 4.6 f093a06e4b feat: restore from ZFS snapshot via clone
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 03:51:40 +00:00
Omar SobhandClaude Sonnet 4.6 96293ef794 feat: snapshot scheduling, retention pruning, cold replication
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 03:51:30 +00:00
Omar SobhandClaude Sonnet 4.6 f9af03a134 feat: hot tier GC with stale eviction and LRU space eviction
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 03:51:23 +00:00
Omar SobhandClaude Sonnet 4.6 dfb40eb13b feat: cargo config writer for automatic hot tier routing
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 03:50:39 +00:00
Omar SobhandClaude Sonnet 4.6 965eef4db5 feat: ZFS trait abstraction with SystemZfs and MockZfs
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 03:50:24 +00:00
Omar SobhandClaude Sonnet 4.6 bd390f953d feat: project manifest with load/save/upsert
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 03:49:43 +00:00
Omar SobhandClaude Sonnet 4.6 8ab3bb2d12 feat: config types and loader
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 03:49:25 +00:00
Omar Sobh 08b7f94e4c feat: claw-store workspace scaffold 2026-06-16 03:48:06 +00:00