bb24c77676ead4c41e9c33dcefb8c1a4a9f9cb50
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar Sobh
|
cb07bfc574 |
capture: stream to a Writer instead of buffering the whole tar in RAM
Field finding 2026-07-12 (clawverse measurement): the buffered `capture_target -> Vec<u8>` path peaked at 2.8 GB RAM to capture a 6.1 GB target/debug into a 995 MiB compressed tar. Every byte crossed RAM before touching the network. * `capture_target_to_writer(target_dir, writer) -> u64` — new streaming variant. Walks the tree + writes tar+zstd straight into the caller's Writer via a small ByteCounter wrapper. Peak memory stays at ~zstd sliding window size (few MB). * `capture_target -> Vec<u8>` kept as a thin wrapper for the tests + smaller callers that don't care. * `cmd_build`: capture into a tempfile under `target/`, then open it with `tokio::fs::File` (AsyncRead + Unpin) and hand that to `call_blob_put_stream`. Same-filesystem tempfile means no cross- mount concerns; auto-unlinks on drop. +1 test: `capture_streaming_matches_buffered_and_restores_correctly` proves the streamed bytes match the buffered variant, the reported byte count agrees with the written length, and roundtrip restore from the streamed file works. Combined with PR #22 (QUIC idle timeout), this closes the two RAM/ timeout blockers surfaced by the clawverse pilot. Expected memory ceiling on a runner drops from GBs to MBs, unlocking small-runner deployments (the actual pitch use case). |
||
|
Omar Sobh
|
e70f5d74e0 |
Pilot findings: 5 real-world fixes from the 2026-07-12 deploy
Bundles the profile→dir bug (PR #20 supersede) with four new fixes discovered by running clawstor against itself + across the fabric: * target_subdir_for: `dev`/`test` → `debug/`, `release`/`bench` → `release/`, custom passes through. Was silently skipping upload. * rustc release via gossip: daemon probes `rustc --version` at start, publishes the release string as `clawstor.rustc.release`. PeerView carries it; `cluster-peer-status` prints it in a new column and emits a warning line when the fleet has mixed versions. Would have surfaced the tank/architect 1.96.1 vs 1.95.0 drift instantly. * prewarm publishes fingerprint→blob ref downstream: `pin` now writes a companion tag `<name>.fingerprint` holding the fingerprint bytes. `prewarm` reads the companion, PutTag's it downstream, then PutRef(fp→blob) so a subsequent fingerprint-based `build` HITS. Without this, prewarm was almost useless for the runner path (build always missed even with matching source + rustc). * streaming byte counters: BlobPutStream + BlobGetStream now record the transferred bytes via `record_blob_{put,get}_bytes`. Metric used to stay at 0 no matter how much you moved. * capture determinism: replaced `tar::Builder::append_dir_all` (uses `read_dir`'s native order) with `append_dir_sorted` that walks the tree recursively and sorts by filename bytes at every level. Two byte-identical trees now produce byte-identical tars regardless of filesystem ordering. +3 tests: - target_subdir_matches_cargo_layout (from #20) - fingerprint_companion_tag_uses_dotted_suffix - capture_is_order_independent_of_filesystem_readdir (guard against the exact bug we saw in the field) 252 tests pass (+1 from Phase 5h's 251). Pre-existing macOS `du -sb` failure unchanged. Supersedes #20 (also included here). Ready for re-deploy to tank + architect for the retest run. |
||
|
Omar Sobh
|
242ef527b4 |
Phase 5a: fingerprint + capture + restore for build-artifact cache
The substrate for the killer feature. Given a cargo workspace, compute
a deterministic 32-byte BLAKE3 fingerprint over the inputs that
determine what artifacts should be produced, then bundle the portable
subset of `target/<profile>/` into a zstd-compressed tarball ready to
hand to BlobStore.
## Module: cluster/build_cache.rs (661 lines)
Types:
- FingerprintInputs { cargo_lock, rustc_version_verbose, cargo_config,
rust_toolchain, profile, features, rustflags, target_triple }
- Fingerprint(32 bytes) — parallel shape to BlobId
Public API:
- FingerprintInputs::collect(workspace, profile, features) — reads
Cargo.lock, shells out to `rustc --version --verbose`, reads
optional config files, extracts host triple, sorts+dedups features
- FingerprintInputs::compute() → Fingerprint — domain-separated
BLAKE3 with per-field labels + null separators so field boundaries
can't collide
- capture_target(target_dir) → zstd-tarball bytes
- restore_target(bytes, target_dir) → unpacks
- capture_workspace(workspace, profile) — resolves target dir
- compute_workspace_fingerprint(workspace, profile, features) —
returns (inputs, fingerprint)
Captured: deps/, .fingerprint/, build/, examples/, plus small
top-level files (.cargo-lock, .rustc_info.json, CACHEDIR.TAG).
Explicitly NOT captured: incremental/ (per-machine, not portable —
tied to absolute paths + rustc state; restoring on another host
silently corrupts the build).
Determinism guarantees:
- HeaderMode::Deterministic on the tar builder — identical trees
produce byte-identical tarballs (proven by
`capture_yields_identical_bytes_for_identical_input`)
- follow_symlinks(false) — symlinks archived as symlinks, not their
targets, so the fingerprint doesn't drift with symlink destinations
- Features sorted + deduped so `["b","a"]` and `["a","b"]` hash the same
- Missing optional files treated as empty strings so `absent ==
empty` (add-then-remove doesn't churn the hash)
Deps added: tar 0.4, zstd 0.13.
## Tests (18 new, all real filesystem — no mocks)
Fingerprint (7):
- fingerprint_is_deterministic
- fingerprint_changes_when_cargo_lock_changes
- fingerprint_changes_when_profile_changes
- fingerprint_changes_when_features_change
- fingerprint_is_feature_order_independent (proves sort semantics)
- fingerprint_domain_separation_prevents_field_collision — swap
content between two string fields; naive concat hasher would
collide, ours doesn't
- fingerprint_hex_length_and_stability
Input collection (4):
- read_optional_returns_empty_for_missing
- read_optional_returns_content_for_existing
- collect_errors_when_cargo_lock_absent
- collect_reads_cargo_lock_and_computes — actually shells out to
`rustc --version --verbose`, verifies triple extraction
Capture/restore (5):
- capture_target_errors_when_dir_missing
- capture_and_restore_round_trip_preserves_files — full 6-file
tree including a captured example + an intentionally-excluded
incremental/ dir; verifies incremental/ is absent after restore
- capture_yields_identical_bytes_for_identical_input — byte-equal
tarballs across two identical source trees
- capture_skips_top_level_files_not_in_allowlist
- capture_workspace_resolves_profile_dir
End-to-end (1):
- **end_to_end_fingerprint_capture_blob_restore** — the full
workspace → fingerprint → capture → BlobStore::put_bytes →
BlobStore::get_bytes → restore_target loop. Proves the whole
round-trip lands byte-equal for both captured file trees, and
the BlobId is deterministic across runs. This is the primitive
the cargo-cache CLI sits on top of.
174 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (well under 1300-line ceiling):
- cluster/build_cache.rs: 661
- cluster.rs (submodule declarations): 279
## What this unlocks
Phase 5a is the substrate. The CLI wrapper (`claw-cargo build`) is
Phase 5b — thin glue that:
1. Runs compute_workspace_fingerprint()
2. Asks the peer BlobStat(fingerprint_as_blob_id)
3. Hit → BlobGetStream + restore_target + cargo build (just the
workspace's own crates, ~seconds)
4. Miss → cargo build (full), then capture_target +
push_blob_missing_chunks + PutManifest
Every piece of infrastructure that Phase 5b needs — content-addressed
blob store, streaming and partial-chunk RPC, mTLS transport, gossip-
driven peer discovery — is already merged. Phase 5b is CLI polish,
not new distributed-systems machinery.
## Follow-on
- Phase 5b: claw-cargo CLI wrapper (small — 200-400 lines)
- Phase 5c: pre-fetch on Gitea webhook (workflow triggers → daemon
pre-warms the fingerprint on the target runner)
- Phase 3 remains a parallel track — human-readable namespace layer
on top of raw fingerprints so operators can pin
`clawverse:main:latest-cache` instead of a hex string
|