Phase 5g: cache metrics + GetMetrics RPC + peer-metrics CLI

Every RPC handler that answers a hit-or-miss question now increments
lock-free atomic counters. The GetMetrics RPC (0x13) returns a JSON
snapshot of every counter; the new claw-cargo peer-metrics CLI
prints hit rates, byte volumes, and counter uptime.

Placement engines can now poll these across the fleet to bias runner
scheduling toward whichever node has the warmest cache for a given
repo/tag combination.

## New module: cluster/metrics.rs (268 lines)

Types:
- CacheMetrics — atomic counters, all AtomicU64, Relaxed ordering
  (metrics are advisory, not consistency-critical)
- MetricsReply — JSON snapshot returned by GetMetrics

Public API:
- CacheMetrics::new() — timestamped start, all counters at 0
- record_get_ref_hit / _miss
- record_get_tag_hit / _miss
- record_blob_get_bytes / record_blob_put_bytes
- record_get_chunk_hit / _miss
- record_has_chunk_hit / _miss
- snapshot() — atomic-load every field into a MetricsReply

MetricsReply derived helpers:
- get_ref_hit_rate() / get_tag_hit_rate() / has_chunk_hit_rate() —
  Option<f64> so 0/0 returns None instead of NaN

## RPC method

- GetMetrics (0x13): payload = empty; reply = JSON MetricsReply

Wire-level instrumentation added to RpcRouter dispatch:
- GetRef  → record_get_ref_hit / _miss
- GetTag  → record_get_tag_hit / _miss
- BlobGet → record_blob_get_bytes (on hit)
- BlobPut → record_blob_put_bytes
- HasChunk → record_has_chunk_hit / _miss
- GetChunk → record_get_chunk_hit + record_blob_get_bytes on hit
             / record_get_chunk_miss

RpcRouter grows Arc<CacheMetrics> unconditionally — every router has
metrics, so a fresh node with no traffic still returns a valid
snapshot with all zeros + started_unix.

Streaming variants (BlobPutStream / BlobGetStream) don't yet track
byte counts — they'd require plumbing the count out of put_stream /
stream_to. Follow-on if it turns out to matter for placement.

## Client helper + CLI

- call_get_metrics(&conn) → Result<MetricsReply>

- claw-cargo peer-metrics [--peer ...] [--peer-addr ...] [--tls-dir ...]
    Fetches + pretty-prints:
      counter uptime:      42s
      GetRef  hits/misses: 123 / 45
              hit rate:    73.21%
      GetTag  hits/misses: 8 / 2
              hit rate:    80.00%
      HasChunk hits/miss:  512 / 88
               hit rate:   85.33%
      GetChunk hits/miss:  47 / 12
      Blob GET bytes:      1.23 GiB
      Blob PUT bytes:      3.45 GiB

human_bytes() helper picks GiB / MiB / KiB / B based on magnitude.

Subcommand count now 9: build / prefetch / status / fingerprint /
pin / unpin / list-tags / prewarm / peer-metrics.

## Tests (13 new, all real filesystem / real QUIC — no mocks)

CacheMetrics (6):
- new_starts_all_counters_at_zero_except_timestamp
- recorders_increment_the_right_field (every recorder × 1-2 counts)
- hit_rates_none_when_zero_events (avoids 0/0 NaN)
- hit_rates_compute_correctly (3 hits / 1 miss → 75%)
- snapshot_round_trips_through_json
- snapshots_across_threads_are_consistent_up_to_relaxed_ordering
  (8 threads × 1000 increments → exactly 8000)

RPC integration (7):
- phase_5g_method_byte_encoding
- get_metrics_returns_empty_snapshot_before_any_activity
- get_ref_records_hit_and_miss_counters (2 hits + 1 miss)
- get_tag_records_hit_and_miss_counters
- blob_get_and_blob_put_record_byte_counts
- has_chunk_and_get_chunk_record_hit_miss_counters
- **end_to_end_get_metrics_over_real_quic** — seed activity locally,
  fire GetRef/GetTag/BlobGet dispatches to move the counters, then
  fetch metrics through real QUIC + mTLS and verify each field
  including the 0.5 hit rate calculation

238 tests pass. Pre-existing macOS-only failure unchanged.

File sizes (all under 1300-line ceiling):
- cluster/metrics.rs: 268
- cluster/rpc.rs: 818
- cluster/rpc/tests_phase5.rs: 905
- claw_cargo.rs: 894

## What this enables

Fleet-wide visibility into which peer is actually serving traffic:

  # From anywhere with connectivity + fleet mTLS
  claw-cargo peer-metrics --peer tank
  claw-cargo peer-metrics --peer architect
  claw-cargo peer-metrics --peer morpheus

Compare hit rates side by side to see which node's cache is warmest.
A placement engine can automate this — poll every 30s, feed the
scheduler.

## Follow-on

- 5h: streaming variant of prewarm (fixed-memory ceiling for many-GB
  blobs)
- 5i: metrics also published via gossip so PeerView carries hit rate
  without a per-peer GetMetrics roundtrip
- 5j: prometheus /metrics endpoint on the daemon for existing dash
  integrations
- 6: FUSE mount for warm-tier git worktrees
This commit is contained in:
Omar Sobh
2026-07-12 04:06:17 -07:00
parent db55903311
commit d36cec11a6
6 changed files with 589 additions and 7 deletions
+183
View File
@@ -290,6 +290,189 @@ async fn list_tags_returns_json_sorted() {
assert_eq!(decoded[0].decode_value().unwrap(), [1u8; 32]);
}
// ── Phase 5g: cache metrics RPC ──────────────────────────────────────
#[test]
fn phase_5g_method_byte_encoding() {
assert_eq!(Method::GetMetrics.as_byte(), 0x13);
assert_eq!(Method::from_byte(0x13), Some(Method::GetMetrics));
}
#[tokio::test]
async fn get_metrics_returns_empty_snapshot_before_any_activity() {
let (_tmp, router) = router_with_full_stack("solo", next_port()).await;
let reply = dispatch(&router, &[Method::GetMetrics.as_byte()]).await;
let snapshot: crate::cluster::metrics::MetricsReply =
serde_json::from_slice(&reply).unwrap();
assert!(snapshot.started_unix > 0);
assert_eq!(snapshot.get_ref_hits, 0);
assert_eq!(snapshot.get_ref_misses, 0);
assert_eq!(snapshot.blob_get_bytes, 0);
}
#[tokio::test]
async fn get_ref_records_hit_and_miss_counters() {
let (_tmp, router) = router_with_full_stack("solo", next_port()).await;
let store = router.ref_store().unwrap().clone();
let key = [0x11u8; 32];
store.put(&key, &[0x22u8; 32]).await.unwrap();
// Two hits.
for _ in 0..2 {
let mut req = vec![Method::GetRef.as_byte()];
req.extend_from_slice(&key);
let reply = dispatch(&router, &req).await;
assert_eq!(reply.len(), 32);
}
// One miss.
let mut req = vec![Method::GetRef.as_byte()];
req.extend_from_slice(&[0xffu8; 32]);
let reply = dispatch(&router, &req).await;
assert_eq!(reply, vec![ErrorCode::NotFound.as_byte()]);
let snapshot = router.metrics().snapshot();
assert_eq!(snapshot.get_ref_hits, 2);
assert_eq!(snapshot.get_ref_misses, 1);
}
#[tokio::test]
async fn get_tag_records_hit_and_miss_counters() {
let (_tmp, router) = router_with_full_stack("solo", next_port()).await;
let store = router.tag_store().unwrap().clone();
store.put("clawverse:main", &[0u8; 32]).await.unwrap();
let mut hit = vec![Method::GetTag.as_byte()];
hit.extend_from_slice(b"clawverse:main");
dispatch(&router, &hit).await;
let mut miss = vec![Method::GetTag.as_byte()];
miss.extend_from_slice(b"never-set");
dispatch(&router, &miss).await;
let snapshot = router.metrics().snapshot();
assert_eq!(snapshot.get_tag_hits, 1);
assert_eq!(snapshot.get_tag_misses, 1);
}
#[tokio::test]
async fn blob_get_and_blob_put_record_byte_counts() {
let (_tmp, router) = router_with_full_stack("solo", next_port()).await;
let payload = b"metrics witness";
// Put — should record blob_put_bytes.
let mut put_req = vec![Method::BlobPut.as_byte()];
put_req.extend_from_slice(payload);
let put_reply = dispatch(&router, &put_req).await;
assert_eq!(put_reply.len(), 32);
let mut id_bytes = [0u8; 32];
id_bytes.copy_from_slice(&put_reply);
// Get — should record blob_get_bytes.
let mut get_req = vec![Method::BlobGet.as_byte()];
get_req.extend_from_slice(&id_bytes);
let get_reply = dispatch(&router, &get_req).await;
assert_eq!(get_reply, payload);
let snapshot = router.metrics().snapshot();
assert_eq!(snapshot.blob_put_bytes, payload.len() as u64);
assert_eq!(snapshot.blob_get_bytes, payload.len() as u64);
}
#[tokio::test]
async fn has_chunk_and_get_chunk_record_hit_miss_counters() {
let (_tmp, router) = router_with_full_stack("solo", next_port()).await;
let store = router.blob_store().unwrap().clone();
let bytes = b"chunk-in-store";
let hash = crate::cluster::blob::ChunkHash::from_bytes(blake3::hash(bytes).into());
store.put_chunk(&hash, bytes).await.unwrap();
// HasChunk hit + miss.
let mut hit = vec![Method::HasChunk.as_byte()];
hit.extend_from_slice(hash.as_bytes());
dispatch(&router, &hit).await;
let mut miss = vec![Method::HasChunk.as_byte()];
miss.extend_from_slice(&[0u8; 32]);
dispatch(&router, &miss).await;
// GetChunk hit + miss.
let mut get_hit = vec![Method::GetChunk.as_byte()];
get_hit.extend_from_slice(hash.as_bytes());
dispatch(&router, &get_hit).await;
let mut get_miss = vec![Method::GetChunk.as_byte()];
get_miss.extend_from_slice(&[0u8; 32]);
dispatch(&router, &get_miss).await;
let snapshot = router.metrics().snapshot();
assert_eq!(snapshot.has_chunk_hits, 1);
assert_eq!(snapshot.has_chunk_misses, 1);
assert_eq!(snapshot.get_chunk_hits, 1);
assert_eq!(snapshot.get_chunk_misses, 1);
// Get_chunk hit also records blob_get_bytes.
assert_eq!(snapshot.blob_get_bytes, bytes.len() as u64);
}
#[tokio::test]
async fn end_to_end_get_metrics_over_real_quic() {
// Exercise every counter, then fetch the metrics reply through
// real QUIC and verify each field.
let (id_a, id_b) = NodeIdentity::generate_test_pair("a", "b").unwrap();
let (_tmp, router) = router_with_full_stack("a", next_port()).await;
// Seed some activity locally so the counters have real values.
let store = router.ref_store().unwrap().clone();
store.put(&[0x11u8; 32], &[0x22u8; 32]).await.unwrap();
let tag_store = router.tag_store().unwrap().clone();
tag_store.put("hit-me", &[0u8; 32]).await.unwrap();
let blob_store = router.blob_store().unwrap().clone();
let payload = vec![0x5au8; 4096];
let blob_id = blob_store.put_bytes(&payload).await.unwrap();
// Fire dispatches to move the counters.
let mut ref_hit = vec![Method::GetRef.as_byte()];
ref_hit.extend_from_slice(&[0x11u8; 32]);
dispatch(&router, &ref_hit).await;
let mut ref_miss = vec![Method::GetRef.as_byte()];
ref_miss.extend_from_slice(&[0x99u8; 32]);
dispatch(&router, &ref_miss).await;
let mut tag_hit = vec![Method::GetTag.as_byte()];
tag_hit.extend_from_slice(b"hit-me");
dispatch(&router, &tag_hit).await;
let mut blob_get = vec![Method::BlobGet.as_byte()];
blob_get.extend_from_slice(blob_id.as_bytes());
dispatch(&router, &blob_get).await;
// Now start the server + fetch metrics over the wire.
let server = QuicServer::bind(loopback(0), id_a).unwrap();
let server_addr = server.local_addr().unwrap();
let router_srv = router.clone();
let accept_task = tokio::spawn(async move {
if let Some(Ok(conn)) = server.accept().await {
let _ = serve_connection(conn, router_srv).await;
}
});
let client = QuicClient::new(loopback(0), id_b).unwrap();
let conn = client.connect(server_addr, "a").await.unwrap();
let m = call_get_metrics(&conn).await.unwrap();
assert_eq!(m.get_ref_hits, 1);
assert_eq!(m.get_ref_misses, 1);
assert_eq!(m.get_tag_hits, 1);
assert_eq!(m.get_tag_misses, 0);
assert_eq!(m.blob_get_bytes, payload.len() as u64);
assert_eq!(m.get_ref_hit_rate(), Some(0.5));
conn.close(quinn::VarInt::from_u32(0), b"done");
client.shutdown().await;
tokio::time::sleep(Duration::from_millis(50)).await;
accept_task.abort();
}
#[tokio::test]
async fn end_to_end_prewarm_copies_tagged_blob_between_two_peers() {
// Phase 5f: `claw-cargo prewarm --from A --to C --pin tag`.