feat: v0.3.0 — HTTP handler tests + README

- Extract build_app() from run_server() so tests can construct the
  router without binding a port (tower::ServiceExt::oneshot pattern)
- Add 10 new tests in serve::tests: validate_project_name (valid +
  invalid slugs), daemon_uptime when file absent, GET /api/status,
  GET /api/projects, POST /api/activate with invalid name, auth
  middleware (no header → 401, wrong token → 401, correct token →
  pass-through, GET bypasses auth entirely)
- Add tower + http-body-util to dev-dependencies
- Add project README covering architecture, install (Makefile),
  config options, SSH setup, snapshot schedule, CLI reference,
  HTTP API table, pinning, and troubleshooting guide
- Bump version to 0.3.0

Test suite: 39 tests, 0 failures

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
This commit is contained in:
Omar Sobh
2026-06-30 10:51:52 +00:00
co-authored by Claude Sonnet 4.6
parent 39ddac431b
commit 1320d2df9e
4 changed files with 469 additions and 9 deletions
Generated
+2
View File
@@ -262,6 +262,7 @@ dependencies = [
"axum", "axum",
"chrono", "chrono",
"clap", "clap",
"http-body-util",
"libc", "libc",
"serde", "serde",
"serde_json", "serde_json",
@@ -270,6 +271,7 @@ dependencies = [
"tokio", "tokio",
"tokio-stream", "tokio-stream",
"toml", "toml",
"tower",
"tower-http", "tower-http",
"tracing", "tracing",
"tracing-subscriber", "tracing-subscriber",
+266
View File
@@ -0,0 +1,266 @@
# clawstor
ZFS-backed fleet storage daemon for the clawverse two-node cluster (architect + tank). Manages a three-tier hot/warm/cold storage model with automatic GC, hourly/daily/weekly ZFS snapshots, incremental cold replication, and a live SSE dashboard.
---
## Architecture
```
Hot tier /hot/targets/<org>/<repo>/ NVMe — active Cargo target dirs
Warm tier /slab/projects/<org>/<repo> ZFS dataset — canonical git repos
Cold tier /data/archive/ Nightly incremental ZFS snapshot send
```
**Hot (NVMe):** `cargo build` target directories symlinked into the NVMe partition. Activated on demand; GC'd when stale or when the hot tier exceeds `max_gb`.
**Warm (ZFS):** The working checkout of every project, on a ZFS dataset that receives hourly, daily, and weekly snapshots. This is the source of truth for all git history.
**Cold (ZFS send):** Tank sends incremental ZFS snapshots to architect nightly over the 10G fabric link (`10.10.0.9 = architect-fab-tank`). The first run is a full send; subsequent runs are incremental (`zfs send -i <prev>`).
---
## Nodes
| Node | Role | IP (LAN) | ZFS pool |
|---|---|---|---|
| architect | primary | 10.0.0.13 | `slab` |
| tank | secondary | 10.0.0.14 | `slab` |
Architect holds the cold archive. Tank sends to architect. The 10G fabric IP (`10.10.0.9`) is used intentionally for replication bandwidth.
---
## Install
### Prerequisites
- Rust toolchain (`rustup`)
- Node.js + npm (for dashboard build)
- ZFS installed and pools imported
- SSH key from tank → architect configured (see [SSH setup](#ssh-setup))
### Build and install
```sh
# On architect
sudo make deploy NODE=architect
# On tank
sudo make deploy NODE=tank
```
This runs `cargo build --release`, copies the binary to `/usr/local/bin/claw-store`, builds and installs the React dashboard to `/usr/share/claw-store/static`, installs systemd units, and enables all services and timers.
### Makefile targets
| Target | What it does |
|---|---|
| `make build` | Compile release binary |
| `make install` | Install binary to `/usr/local/bin` |
| `make install-systemd` | Install and reload all systemd units |
| `make install-dashboard` | Build React dashboard and copy to `/usr/share/claw-store/static` |
| `make install-config NODE=architect` | Install node-specific config |
| `make deploy NODE=architect` | Full install + enable all units |
| `make uninstall` | Remove binary and systemd units (preserves config and data) |
| `make clean` | Remove build artifacts |
---
## Configuration
Config files live in `config/`. Install the right one with `make install-config NODE=<node>` which copies it to `/etc/claw-store/config.toml`.
### Key options
```toml
[node]
name = "architect"
role = "primary" # primary | secondary
[hot]
path = "/hot/targets"
max_gb = 200 # LRU evicts unpinned projects above this
stale_hours = 48 # GC projects inactive for this long
[warm]
projects_path = "/slab/projects"
zfs_dataset = "slab/projects"
snapshot_retain_hours = 24
snapshot_retain_days = 7
snapshot_retain_weeks = 4
[cold] # architect only
archive_path = "/data/archive"
zfs_dataset = "data/archive"
retain_weeks = 12
[replication] # tank only — sends to architect
send_to_host = "10.10.0.9"
send_to_user = "osobh"
cold_dataset_on_peer = "data/archive/tank-projects"
[peer] # optional — enables reachability probe
host = "10.0.0.13"
user = "osobh"
# Optional — require Bearer token on all HTTP POST endpoints
# api_token = "your-long-random-token-here"
```
To enable API authentication:
```toml
api_token = "$(openssl rand -hex 32)"
```
All `GET` requests (dashboard, status, project list) are always allowed. `POST` requests (activate, deactivate, gc, sync, snapshot) require `Authorization: Bearer <token>`.
---
## Systemd units
| Unit | Runs |
|---|---|
| `claw-store.service` | Background daemon (GC, sync retry) |
| `claw-store-serve.service` | HTTP API + dashboard server (port 3030) |
| `claw-store-snapshot.service` + `.timer` | Hourly snapshot cycle |
| `claw-store-replicate.service` + `.timer` | Nightly cold replication (tank → architect) |
Check status:
```sh
systemctl status claw-store claw-store-serve
journalctl -u claw-store -f
```
---
## CLI reference
```sh
claw-store activate <org/repo> # Link hot target dir, write .cargo/config.toml
claw-store deactivate <org/repo> # Remove hot target dir, restore .cargo/config.toml
claw-store sync <org/repo> # git push, notify peer via SSH
claw-store gc # Evict stale + LRU hot targets
claw-store snapshot # Run snapshot cycle now
claw-store replicate # Run cold replication now (tank only)
claw-store status # Print node status
claw-store list # List all projects and their active state
claw-store serve # Start HTTP API + dashboard server
claw-store daemon # Start background daemon
claw-store restore <snap> <dest> # Clone snapshot into dest directory
claw-store pin <org/repo> # Mark project as pinned (survives all GC)
claw-store unpin <org/repo> # Remove pin
```
---
## HTTP API
Base URL: `http://<node>:3030`
| Method | Path | Description |
|---|---|---|
| GET | `/api/status` | Node status (role, ZFS pool, uptime, hot usage) |
| GET | `/api/projects` | All projects with active/size/branch info |
| GET | `/api/snapshots` | ZFS snapshot list with kind and timestamp |
| GET | `/api/hot` | Hot tier entries with size and last-modified |
| GET | `/api/sync-queue` | Pending sync retry jobs |
| GET | `/api/events` | SSE stream (status + projects every 5s) |
| POST | `/api/activate` | `{"project": "org/repo"}` |
| POST | `/api/deactivate` | `{"project": "org/repo"}` |
| POST | `/api/sync` | `{"project": "org/repo"}` |
| POST | `/api/gc` | Trigger GC pass |
| POST | `/api/snapshot` | Trigger snapshot cycle |
---
## SSH setup
Tank's replication and peer-probe commands SSH to architect. The daemon runs as root (via systemd), so root's key on tank must be authorized on architect.
```sh
# On tank — generate key if not present
sudo ssh-keygen -t ed25519 -f /root/.ssh/id_ed25519 -N ""
# Copy to architect
sudo ssh-copy-id [email protected]
# Verify
sudo ssh -o BatchMode=yes [email protected] true && echo "OK"
```
The peer probe in `serve.rs` uses `BatchMode=yes` (no password prompts) with a 3-second connect timeout. Replication SSH calls use `ConnectTimeout=10` with server-alive checks to prevent indefinite hangs.
---
## Snapshot schedule
Snapshots are taken by `claw-store-snapshot.timer` which fires hourly.
Each run:
- Always takes an **hourly** snapshot; prunes to `snapshot_retain_hours`
- At midnight (`HHMM = 0000`): also takes a **daily** snapshot; prunes to `snapshot_retain_days`
- At Sunday midnight: also takes a **weekly** snapshot; prunes to `snapshot_retain_weeks`
Snapshot names: `<dataset>@<kind>-<YYYY-MM-DD-HHMM>` e.g. `slab/projects@hourly-2026-06-30-0400`.
---
## Project pinning
Pinned projects survive all GC passes — both the stale-hours sweep and the LRU space-pressure eviction. If every unpinned project has been evicted and the hot tier is still over budget, the daemon logs a warning and stops rather than evict pinned projects.
```sh
claw-store pin myorg/critical-service
claw-store unpin myorg/critical-service
```
Pinned status is stored in the manifest (`/var/lib/claw-store/manifest.toml`).
---
## Troubleshooting
**Hot tier not shrinking after gc:**
Check `journalctl -u claw-store` for "over budget but every remaining project is pinned". If so, unpin a project or raise `max_gb`.
**Replication not running on tank:**
```sh
systemctl status claw-store-replicate.timer
journalctl -u claw-store-replicate -n 50
sudo claw-store replicate # run manually, check output
```
**Snapshot cycle missed:**
```sh
systemctl status claw-store-snapshot.timer
sudo claw-store snapshot
```
**Dashboard shows uptime 0:**
The serve process reads `/var/lib/claw-store/daemon-started`. If the daemon (`claw-store.service`) isn't running, uptime will show 0.
**Peer shown as unreachable:**
SSH key not installed, or the peer's `claw-store-serve.service` is down. Run the SSH verify command above.
**`activate` fails with "non-UTF-8 path":**
Project path contains non-UTF-8 bytes. All paths under `/slab/projects` should be ASCII.
---
## Development
```sh
# Run all tests
~/.cargo/bin/cargo test --manifest-path claw-store/Cargo.toml
# Lint
~/.cargo/bin/cargo clippy --manifest-path claw-store/Cargo.toml -- -D warnings
# Release build
~/.cargo/bin/cargo build --release --manifest-path claw-store/Cargo.toml
```
Tests cover: manifest atomicity, hot GC (stale + LRU + pinning), ZFS snapshot lifecycle, weekly snapshot scheduling, incremental replication logic, HTTP handler responses, auth middleware, project name validation, and sync queue.
+3 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "claw-store" name = "claw-store"
version = "0.2.0" version = "0.3.0"
edition = "2021" edition = "2021"
[[bin]] [[bin]]
@@ -31,3 +31,5 @@ tempfile = "3"
[dev-dependencies] [dev-dependencies]
tempfile = "3" tempfile = "3"
tower = { version = "0.5", features = ["util"] }
http-body-util = "0.1"
+198 -8
View File
@@ -643,13 +643,8 @@ async fn auth_middleware(
// ── server entry point ──────────────────────────────────────────────────────── // ── server entry point ────────────────────────────────────────────────────────
pub async fn run_server( /// Builds the Axum Router. Extracted so tests can call it without binding a port.
cfg: Config, pub fn build_app(cfg: Config, manifest_path: PathBuf, static_dir: Option<PathBuf>) -> Router {
_manifest: Manifest,
port: u16,
static_dir: Option<PathBuf>,
) -> Result<()> {
let manifest_path = Manifest::default_path();
let state = Arc::new(AppState { cfg, manifest_path }); let state = Arc::new(AppState { cfg, manifest_path });
let cors = CorsLayer::new() let cors = CorsLayer::new()
@@ -678,7 +673,17 @@ pub async fn run_server(
api = api.fallback_service(tower_http::services::ServeDir::new(dir)); api = api.fallback_service(tower_http::services::ServeDir::new(dir));
} }
let app: Router = api.layer(cors).with_state(state); api.layer(cors).with_state(state)
}
pub async fn run_server(
cfg: Config,
_manifest: Manifest,
port: u16,
static_dir: Option<PathBuf>,
) -> Result<()> {
let manifest_path = Manifest::default_path();
let app = build_app(cfg, manifest_path, static_dir);
let addr = std::net::SocketAddr::from(([0, 0, 0, 0], port)); let addr = std::net::SocketAddr::from(([0, 0, 0, 0], port));
tracing::info!("claw-store API server listening on {}", addr); tracing::info!("claw-store API server listening on {}", addr);
@@ -687,3 +692,188 @@ pub async fn run_server(
axum::serve(listener, app).await?; axum::serve(listener, app).await?;
Ok(()) Ok(())
} }
// ── tests ─────────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
use axum::{
body::Body,
http::{Request, StatusCode},
};
use tempfile::NamedTempFile;
use tower::ServiceExt; // for .oneshot()
fn test_cfg(api_token: Option<&str>) -> Config {
let token_line = match api_token {
Some(t) => format!("api_token = \"{}\"\n", t),
None => String::new(),
};
// api_token is a root-level key — it must appear before any [section] header
toml::from_str(&format!(
r#"
{token_line}
[node]
name = "test-node"
role = "primary"
[hot]
path = "/tmp/claw-test-hot"
max_gb = 10
stale_hours = 48
[warm]
projects_path = "/tmp/claw-test-warm"
zfs_dataset = "test/dataset"
snapshot_retain_hours = 24
snapshot_retain_days = 7
snapshot_retain_weeks = 4
"#
))
.unwrap()
}
async fn body_bytes(body: Body) -> Vec<u8> {
use http_body_util::BodyExt;
body.collect().await.unwrap().to_bytes().to_vec()
}
// ── pure-function tests ───────────────────────────────────────────────────
#[test]
fn test_validate_project_name_accepts_valid() {
assert!(validate_project_name("org/repo").is_ok());
assert!(validate_project_name("my-org/my-repo").is_ok());
assert!(validate_project_name("Org123/Repo.name_v2").is_ok());
}
#[test]
fn test_validate_project_name_rejects_bad() {
assert!(validate_project_name("no-slash").is_err());
assert!(validate_project_name("../../etc/passwd").is_err());
assert!(validate_project_name("; rm -rf /").is_err());
assert!(validate_project_name("a/b/c").is_err());
assert!(validate_project_name("org/").is_err());
assert!(validate_project_name("/repo").is_err());
}
#[test]
fn test_daemon_uptime_absent() {
if std::path::Path::new(crate::daemon::DAEMON_STARTED_PATH).exists() {
return; // daemon is live — skip rather than assert a live value
}
assert_eq!(daemon_uptime_secs(), 0);
}
// ── HTTP handler tests ────────────────────────────────────────────────────
#[tokio::test]
async fn test_get_status_returns_200_json() {
let tmp = NamedTempFile::new().unwrap();
let app = build_app(test_cfg(None), tmp.path().to_path_buf(), None);
let req = Request::builder()
.uri("/api/status")
.body(Body::empty())
.unwrap();
let resp = app.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let bytes = body_bytes(resp.into_body()).await;
let val: serde_json::Value = serde_json::from_slice(&bytes).unwrap();
assert_eq!(val["node_name"], "test-node");
assert_eq!(val["role"], "primary");
}
#[tokio::test]
async fn test_get_projects_returns_200_array() {
let tmp = NamedTempFile::new().unwrap();
let app = build_app(test_cfg(None), tmp.path().to_path_buf(), None);
let req = Request::builder()
.uri("/api/projects")
.body(Body::empty())
.unwrap();
let resp = app.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let bytes = body_bytes(resp.into_body()).await;
let val: serde_json::Value = serde_json::from_slice(&bytes).unwrap();
assert!(val.is_array(), "expected JSON array, got: {}", val);
}
#[tokio::test]
async fn test_activate_invalid_name_returns_error_json() {
let tmp = NamedTempFile::new().unwrap();
let app = build_app(test_cfg(None), tmp.path().to_path_buf(), None);
let body = serde_json::json!({"project": "../../bad"}).to_string();
let req = Request::builder()
.method("POST")
.uri("/api/activate")
.header("content-type", "application/json")
.body(Body::from(body))
.unwrap();
let resp = app.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let bytes = body_bytes(resp.into_body()).await;
let val: serde_json::Value = serde_json::from_slice(&bytes).unwrap();
assert_eq!(val["ok"], false);
assert!(val["error"].as_str().unwrap_or("").contains("invalid project name"));
}
#[tokio::test]
async fn test_auth_no_header_returns_401() {
let tmp = NamedTempFile::new().unwrap();
let app = build_app(test_cfg(Some("secret123")), tmp.path().to_path_buf(), None);
let body = serde_json::json!({"project": "org/repo"}).to_string();
let req = Request::builder()
.method("POST")
.uri("/api/activate")
.header("content-type", "application/json")
.body(Body::from(body))
.unwrap();
let resp = app.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
}
#[tokio::test]
async fn test_auth_wrong_token_returns_401() {
let tmp = NamedTempFile::new().unwrap();
let app = build_app(test_cfg(Some("secret123")), tmp.path().to_path_buf(), None);
let body = serde_json::json!({"project": "org/repo"}).to_string();
let req = Request::builder()
.method("POST")
.uri("/api/activate")
.header("content-type", "application/json")
.header("authorization", "Bearer wrong-token")
.body(Body::from(body))
.unwrap();
let resp = app.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
}
#[tokio::test]
async fn test_auth_correct_token_passes_through() {
let tmp = NamedTempFile::new().unwrap();
let app = build_app(test_cfg(Some("secret123")), tmp.path().to_path_buf(), None);
let body = serde_json::json!({"project": "org/repo"}).to_string();
let req = Request::builder()
.method("POST")
.uri("/api/activate")
.header("content-type", "application/json")
.header("authorization", "Bearer secret123")
.body(Body::from(body))
.unwrap();
let resp = app.oneshot(req).await.unwrap();
// Auth passed — response is from the handler, not the middleware
assert_ne!(resp.status(), StatusCode::UNAUTHORIZED);
}
#[tokio::test]
async fn test_get_requests_bypass_auth() {
// GET endpoints must be accessible even when a token is configured
let tmp = NamedTempFile::new().unwrap();
let app = build_app(test_cfg(Some("secret123")), tmp.path().to_path_buf(), None);
let req = Request::builder()
.uri("/api/status")
.body(Body::empty())
.unwrap();
let resp = app.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
}
}