Commit Graph
125 Commits
Author SHA1 Message Date
Omar SobhandClaude Sonnet 4.6 1320d2df9e feat: v0.3.0 — HTTP handler tests + README
- Extract build_app() from run_server() so tests can construct the
  router without binding a port (tower::ServiceExt::oneshot pattern)
- Add 10 new tests in serve::tests: validate_project_name (valid +
  invalid slugs), daemon_uptime when file absent, GET /api/status,
  GET /api/projects, POST /api/activate with invalid name, auth
  middleware (no header → 401, wrong token → 401, correct token →
  pass-through, GET bypasses auth entirely)
- Add tower + http-body-util to dev-dependencies
- Add project README covering architecture, install (Makefile),
  config options, SSH setup, snapshot schedule, CLI reference,
  HTTP API table, pinning, and troubleshooting guide
- Bump version to 0.3.0

Test suite: 39 tests, 0 failures

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-30 10:51:52 +00:00
Omar SobhandClaude Sonnet 4.6 39ddac431b chore: stop tracking compiled dashboard assets
claw-store/static/ is now in .gitignore. The bundle is installed via
`make install-dashboard` which runs `npm run build` and copies to
/usr/share/claw-store/static — it should not live in the repo.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-30 10:43:49 +00:00
Omar SobhandClaude Sonnet 4.6 1b1d66eac9 feat(v0.3.0): weekly snapshots, incremental replication, auth, uptime, hardening
- snapshot: implement weekly ZFS snapshots (Sunday midnight, retain_weekly config)
- snapshot: incremental cold replication via zfs send -i; tracks last replicated
  snapshot in /var/lib/claw-store/last-replicated-snapshot
- daemon: write start-time file for uptime reporting; add SIGTERM graceful shutdown
- serve: daemon_uptime_secs now reads the start-time file (was hardcoded 0)
- serve: validate project names in POST handlers (org/repo slug, no traversal)
- serve: Bearer token auth middleware on all POST endpoints via cfg.api_token
- config: add optional api_token field (backward compatible, defaults to None)
- sync: add SSH timeouts (ConnectTimeout=10, ServerAliveInterval=5) to peer notify
- hot/sync/main: replace unwrap() on path-to-str with proper anyhow errors
- config/tank.toml: document 10G fabric IP and nightly_at field intent
- .gitignore: exclude compiled dashboard assets (claw-store/static/)
- Makefile: add build, install, install-systemd, install-dashboard, deploy targets
- tests: 29 passing (up from 25); 4 new weekly snapshot tests

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-30 10:41:20 +00:00
Omar SobhandClaude Opus 4.7 d076fac619 fix(serve): single-source the project list — stops dashboard flashing
The HTTP `/api/projects` handler and the SSE `/api/events` stream each
built their own ProjectInfo list — independent inline code, ~50 lines
of mostly identical duplicate.  But the two diverged on ONE field:

  /api/projects: is_active = active.is_some() || has_hot_cargo_config(...)
  /api/events:   is_active = active.is_some()

The dashboard hits BOTH every few seconds (a 10-second setInterval
polling /api/projects + a 5-second SSE pushing events).  For any
project that was hot-wired via `.cargo/config.toml` but not in the
manifest, the two endpoints disagreed about its `is_active` bit.
The dashboard's last-write-won, and the row visibly flickered as
the active flag toggled.  The 'X active' count in the header bounced
along with it.

Extracted `build_projects_list(cfg, &manifest)` and call it from both
sides.  Same answer in both code paths.  Verified post-deploy:

  architect: HTTP=59 active, SSE=59 active (identical)
  tank:      HTTP=57 active, SSE=57 active (identical)

This also makes the 'total' number stable.  The 354 in the dashboard
header is correct semantics — discoverable git repos on disk — but
the label is misleading.  Cosmetic cleanup for a follow-up:

  - "Projects — N discovered · M active · K shown" reads truer than
    "N shown · M active · K total"
  - The status card's `active_project_count` (manifest.projects.len)
    is a third meaning of 'active' that doesn't match either dashboard
    number; consider renaming to `manifest_size` to disambiguate.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-06-28 13:23:44 +00:00
Omar SobhandClaude Opus 4.7 af50adec19 feat(v0.2.0): atomic+flocked manifest, pinned projects, serve systemd unit
The two biggest pain points coming out of the architecture review:

  1. The manifest at /var/lib/claw-store/projects.toml was the only
     piece of writable state but had no locking, no atomic writes,
     and three concurrent writers (daemon poll tick, every CLI verb,
     and the dashboard shelling out via /api/activate). Two writers
     interleaving silently dropped one of them; a crash mid-write
     left a corrupt half-written TOML that the next reader parsed
     as an empty manifest.

  2. Reboot survival: the dashboard had no systemd unit and was a
     stray hand-launched process. Architect lost its dashboard on
     todays reboot.

This commit lands:

- Manifest::update(path, FnOnce(&mut Manifest)) — locked-atomic
  load-mutate-save in one transaction. Uses libc::flock(LOCK_EX) on
  a sidecar .lock file (so the data file can be replaced by rename
  without invalidating the lock) and tempfile + persist for the
  rename. Concurrent writers serialise; readers see the previous
  state or the new state, never a torn write. Manifest::load uses
  LOCK_SH so it never races a mid-rename.

- Project.pinned: bool with #[serde(default)] so legacy manifests
  parse cleanly. hot::gc_stale_targets and hot::gc_by_space both
  skip pinned projects, with a WARN log when every remaining
  project is pinned but were still over budget — operator intent
  beats space pressure.

- claw-store pin <project> / unpin <project> CLI verbs.
  status command surfaces pin marker (📌).
  activate preserves an existing rows pinned flag so re-activating
  doesnt silently unpin.

- claw-store-serve.service systemd unit. Type=simple, Restart=
  on-failure, RestartSec=15, ProtectSystem=strict + ReadWritePaths
  =/var/lib/claw-store, ProtectHome, NoNewPrivileges, PrivateTmp.

- daemon poll tick reloads the manifest from disk at the start of
  each cycle (so CLI activations between ticks are visible) and
  routes its GC write through Manifest::update (so it cant race a
  concurrent CLI pin).

- libc + tempfile move from dev-deps into runtime deps.
- empty-manifest fallthrough on load (treat "" as default) so a
  half-written tempfile crashed pre-rename doesnt hard-fail the
  daemon next boot.

- 25 tests passing incl. new ones: legacy-toml-parses, update-
  serializes-two-sequential-writers, pinned-survives-stale-gc,
  pinned-survives-space-gc-even-when-lru.

Version bumped 0.1.0 → 0.2.0.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-06-28 12:59:41 +00:00
Omar SobhandClaude Opus 4.7 2fcfb16160 feat(serve): HTTP+SSE API + static dashboard bundle
Adds the serve subcommand: an axum 0.7 + tower-http server on
:7700 backing the React dashboard. Routes:

  GET  /api/status   /api/projects   /api/snapshots
       /api/sync-queue   /api/hot   /api/events (SSE 5s tick)
  POST /api/activate /api/deactivate /api/sync
       /api/gc /api/snapshot

Mutating endpoints shell out to /usr/local/bin/claw-store so
all CLI logic stays single-sourced. claw-store/static/ holds
the prebuilt dashboard Vite bundle for --static-dir.

cargo_init.rs gets a minor wiring tweak so the API can read
back the managed cargo config marker.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-06-28 12:52:46 +00:00
Omar SobhandClaude Sonnet 4.6 8b4d0088ca fix: list command only shows git repos, not org subdirs
Skip non-git directories inside org folders so flat layout dirs like
claw-store/target or claw-store/.cargo don't appear as fake repos.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 11:50:02 +00:00
Omar SobhandClaude Sonnet 4.6 a9c12173fe feat: activate/deactivate/list/sync/pull commands + peer sync queue
- activate <org/repo>: wire hot tier, write .cargo/config.toml, register
- deactivate <org/repo>: auto-sync to peer, evict hot tier, unregister
- list: show all warm repos with activation status and last-active time
- sync <org/repo>: git push origin then SSH peer to pull
- pull <org/repo>: git pull from origin, stamp last_sync in manifest
- SyncQueue: file-based retry queue (/var/lib/claw-store/sync-queue.toml)
- daemon: drains sync queue on every 5-min poll tick
- config: [peer] host/user for cross-node notification
- manifest: Project.name is now org/repo, adds last_sync field
- hot tier paths are org/repo-namespaced to avoid collisions

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 11:45:04 +00:00
Omar Sobh 3fb108edfc fix: manifest path to /var/lib/claw-store (user-writable) 2026-06-16 04:00:31 +00:00
Omar Sobh ac3079b56c fix: SystemZfs uses sudo for write ops (snapshot/destroy/clone/send) 2026-06-16 03:58:39 +00:00
Omar Sobh 7ae1c4f950 feat: daemon event loop and full CLI (init, status, gc, snapshot, restore, replicate) 2026-06-16 03:55:22 +00:00
Omar Sobh 11c32df3cd merge: restore.rs 2026-06-16 03:53:19 +00:00
Omar Sobh 426323723c merge: snapshot.rs 2026-06-16 03:53:19 +00:00
Omar Sobh 5f032bae25 merge: hot.rs 2026-06-16 03:53:19 +00:00
Omar Sobh 8cd8ce9d16 merge: zfs.rs 2026-06-16 03:53:19 +00:00
Omar Sobh c20fdcb484 merge: manifest.rs 2026-06-16 03:53:19 +00:00
Omar Sobh fbd8840a70 merge: config.rs 2026-06-16 03:53:19 +00:00
Omar SobhandClaude Sonnet 4.6 f093a06e4b feat: restore from ZFS snapshot via clone
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 03:51:40 +00:00
Omar SobhandClaude Sonnet 4.6 96293ef794 feat: snapshot scheduling, retention pruning, cold replication
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 03:51:30 +00:00
Omar SobhandClaude Sonnet 4.6 f9af03a134 feat: hot tier GC with stale eviction and LRU space eviction
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 03:51:23 +00:00
Omar SobhandClaude Sonnet 4.6 dfb40eb13b feat: cargo config writer for automatic hot tier routing
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 03:50:39 +00:00
Omar SobhandClaude Sonnet 4.6 965eef4db5 feat: ZFS trait abstraction with SystemZfs and MockZfs
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 03:50:24 +00:00
Omar SobhandClaude Sonnet 4.6 bd390f953d feat: project manifest with load/save/upsert
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 03:49:43 +00:00
Omar SobhandClaude Sonnet 4.6 8ab3bb2d12 feat: config types and loader
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-16 03:49:25 +00:00
Omar Sobh 08b7f94e4c feat: claw-store workspace scaffold 2026-06-16 03:48:06 +00:00