Commit Graph
10 Commits
Author SHA1 Message Date
Omar Sobh 69a6e4e7f2 missions: sweeper + socket-proxy NETWORKS grant + mount ordering (C3 slice 4-5)
ci / gates (push) Successful in 6s
ci / rust (push) Failing after 10s
ci / frontend (push) Successful in 27s
ci / e2e (push) Skipped
ci / publish (push) Skipped
- mission_runtime::spawn_sweeper: force-removes runtime containers
  for missions terminal for >=30 min, clears runtime_endpoint. Wired
  into clawmates-server main().
- docker-compose socket-proxy: NETWORKS=1 so bollard.connect_network
  can attach containers to clawmates_edge for provider egress.
- phase_runner ordering: ensure_checkout BEFORE ensure_container so
  the mission dir exists before docker mounts it.
- provisioner: mkdir_p the mission dir defensively for research-only
  missions that skip checkout entirely.
2026-07-21 22:33:44 -07:00
Omar Sobh c0a4fbc943 compose: switch missions_workspaces to host bind-mount
Named docker volume required the separately-managed clawmates-runtime
container to know the volume's on-disk path (varies by docker root).
A predictable host path (/var/lib/clawmates-missions) means the
runtime's spawn command can bind-mount the same path directly.
2026-07-20 04:15:08 -07:00
Omar Sobh 214d0c5e9f task #25: per-mission repo checkout on mission launch
Closes the follow-up gap flagged when task #23 landed. security_scan
and benchmark_runner now exec against $CLAWMATES_MISSIONS_ROOT/
{mission_id}/repo — this commit is what actually puts a repo there.

  - crates/cm-api/src/mission_workspace.rs — new module.
    ensure_checkout(pool, workspace_id, mission_id):
      * mission with no repo_id → Ok(None), no-op
      * repo cloned into $ROOT/{id}/repo (--depth 1)
      * dir already a git repo → fetch + reset --hard origin/{branch}
        (idempotent — every launch brings the tree in sync with the
        remote default_branch)
    Auth uses the process's ambient git credential setup (SSH agent /
    .netrc / helper). Tokens deliberately not embedded in URLs.

  - crates/cm-api/src/mission_orchestrator.rs — on_launch calls
    ensure_checkout after team materialization + team_id bind.
    Non-fatal: clone failures log and continue so research_only
    missions (no repo needed) don't get blocked.

  - deploy/compose/docker-compose.yml — new named volume
    missions_workspaces mounted at /var/lib/clawmates-missions on
    both the server (writer) and where the clawmates-runtime
    container will mount it (reader for docker exec). CLAWMATES_
    MISSIONS_ROOT + CLAWMATES_RUNTIME_CONTAINER env vars set on
    the server so mission_workspace + exec_target read the same
    canonical values.

The scan/bench trigger buttons now actually produce findings once
you (a) run a mission whose repo_id is set, (b) have the
clawmates-runtime container bind-mounting missions_workspaces at
/var/lib/clawmates-missions.

Verified: SQLX_OFFLINE=true cargo check -p cm-api +
cargo test -p cm-api --test mission_orchestrator both green.
2026-07-20 04:04:36 -07:00
Omar Sobh b431d00f1a deploy: put the broker on both core + edge networks
ci / gates (push) Successful in 7s
ci / frontend (push) Successful in 25s
ci / rust (push) Successful in 4m5s
ci / e2e (push) Has been skipped
ci / publish (push) Successful in 21s
Broker acts as the §14 door for outbound provider fetches (Gitea +
GitHub + GitLab repo lists, Slack sends, OAuth token exchange). The
prior 'internal: true'-only core network gave the container no route
to the internet, so BrokerClient::fetch_authorized() failed at DNS
before it could reach api.github.com or a self-hosted Gitea.

Adding edge keeps the broker inbound-tight (still only listens on the
unix socket inside broker_run — no exposed TCP port) while granting
outbound. Postgres access via core is preserved.

Applied by hand to /opt/clawmates/docker-compose.yml on gw-04 to
unblock the current repo-connect flow; committing here so the next
re-provision doesn't regress it. The auto-deploy timer only pulls
images — compose file drift lives with us until a future
compose-sync step is added.
2026-07-07 17:03:53 -07:00
Omar SobhandClaude Opus 4.8 148705769f Reaping: sandbox orphan reaper + DB expiry/retention sweeper + volume-init retry
ci / gates (push) Has been cancelled
ci / rust (push) Has been cancelled
ci / sandbox-k8s (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / e2e (push) Has been cancelled
Closes the cleanup gaps found in review (latent today; bites under load/crashes).

Sandbox containers (cm-sandbox / cm-runtime):
- label every sandbox `clawmates.sandbox={agent|browser}` at create
- SandboxDriver::list_managed(kind) (Docker label filter + K8s label selector)
- SandboxManager::reconcile_orphans(ttl) + spawn_reaper: removes engine
  containers no live handle owns (ZERO = all)
- boot reconciliation (every pre-existing sandbox is an orphan from a dead
  process) + periodic reaper (5m interval / 10m TTL)
- SIGTERM graceful drain: serve().with_graceful_shutdown → shutdown() both
  managers so a redeploy can't leak; destroy errors now logged not swallowed

DB expiry/retention (cm-db cleanup.rs + cm-api cleanup_sweeper, hourly):
- expire auth_sessions + oauth_states past expires_at (security)
- prune sent outbox(7d), run_events(14d), routine_runs(30d), terminal
  topology_runs(90d), consumed execution_grants(7d)

Compose: volume-init restart "no" → on-failure:5 (retry instead of wedging boot).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-19 04:55:27 -07:00
Omar SobhandClaude Fable 5 447f7039d8 Compose: production README, env knobs, first-owner bootstrap — deployed & live
Made the Docker Compose route turn-key for a real self-host, then stood
the whole stack up and drove a live chat through it.

- First-owner bootstrap (cm-auth::bootstrap_owner): a fresh local-auth
  install has no users and no signup route, so the initial Owner +
  workspace are provisioned ONCE from CLAWMATES_BOOTSTRAP_* env on first
  boot — idempotent, never clobbers an existing install (keys on 'any
  workspace exists'). Two real-Postgres tests (creates + signs in;
  second call is a no-op). Wired into server boot, guarded on a
  non-empty password
- deploy/compose/README.md: full production bring-up — services, the
  security topology, every config knob, Anthropic vs local-LLM, the
  broker-key backup, ops, and TLS/SSE proxy notes
- .env.example fleshed out (bootstrap, LLM, auth mode, OTLP); compose
  uses optional env_file so only the knobs you set are injected (unset
  options never override clawmates.toml with empty strings)
- volume-init one-shot chowns the broker's named volumes so the non-root
  scratch broker can write its socket + generated master key

Deployed locally and verified end to end: all 5 containers healthy,
broker generated its key, server bootstrapped owner@…, login + /api/user/me
work, and a real message streamed a live Anthropic response through the
gateway. Captured screenshots of login, workspace home, chat, and the
Computer panel.

166 Rust tests (+2 bootstrap).

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-06-10 14:06:15 -05:00
Omar SobhandClaude Fable 5 add4f79fed Rebrand: TeamClaw -> Clawmates (clawmates.work)
Full-depth rename per the approved plan; the 'claw' product vocabulary
(claws, /claws routes, clawId, Claw Chat) stays — it is now the brand.

- Display brand: Clawmates (manifest, titles, hero, login/rail logo
  'clawmates'); default host app.clawmates.work; registry
  ghcr.io/clawmates
- Crates tc-* -> cm-* (16 crates + all imports); binaries
  clawmates-server/broker/bundler; images clawmates/*; env prefix
  CLAWMATES_* (+ CM_TEST_DATABASE_URL / CM_LIVE_LLM); config
  clawmates.toml; helm chart deploy/helm/clawmates with clawmates-*
  resources; db names clawmates*; sockets /run/clawmates; cookie
  cm_session; kind cluster clawmates-test; seccomp node profile
  clawmates-agent-profile.json
- All 9 Playwright brand assertions updated in lockstep; historical
  spec document left untouched as the only remaining 'TeamClaw'
- Local env migrated: dev pg clawmates-dev-pg/clawmates_dev, shared
  test server clawmates-test-pg, kind cluster recreated with image +
  profile, compose images rebuilt under clawmates/*

Verified end to end: 161 Rust + 68 frontend tests, 29 Playwright
journeys, 4 live kind tests, helm/install/LOC/placeholder gates, and
the clean-room install rehearsal serving the clawmates login page from
a signed bundle of the rebuilt images.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-06-10 12:31:25 -05:00
Omar SobhandClaude Fable 5 8046853feb Post-1.0: OTLP tracing, broker image + compose service, install rehearsal
- tc-telemetry: fmt subscriber always; with [telemetry] otlp_endpoint
  set, spans batch-export over OTLP/HTTP. Tested against a REAL OTLP
  receiver decoding the actual protobuf (official proto types): the
  emitted span and service.name arrive on the wire. No endpoint = no
  export = no network (air-gap stance). tower-http TraceLayer gives
  every API request a span
- The broker finally has its own image (images/broker.Dockerfile,
  9.5MB from scratch) — the Helm chart referenced one that never
  existed — and the compose deployment now RUNS the broker, sharing a
  socket volume with the server (the unix-socket equivalent of the K8s
  sidecar). Compose secret flows were silently dead before this
- server.Dockerfile fixes surfaced by the rehearsal: the workspace
  build needs tools/ (bundler joined the workspace) and
  images/seccomp/ (include_str! profile) in the build context
- scripts/rehearse-install.sh (plan: clean-VM rehearsal): assembles a
  REAL signed bundle from the built images (server/frontend/broker/
  postgres/socket-proxy), runs the customer path — offline verify,
  docker load, compose up — and asserts /healthz plus the served login
  page before teardown. Passing locally; wired as a release.yml step,
  which also builds/ships the broker + socket-proxy images now

161 Rust tests + 29 journeys; clean-room rehearsal green.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-06-10 11:35:32 -05:00
Omar SobhandClaude Fable 5 a26939d7bc Plan hardening: socket-proxy allowlist, gateway load test, offline verify
- The compose server NEVER sees the raw Docker socket (plan risk #5):
  tecnativa/docker-socket-proxy on an isolated engine_net with exactly
  CONTAINERS/POST/EXEC/DELETE/VERSION allowed; server reaches it via
  DOCKER_HOST. DockerDriver honors DOCKER_HOST (connect_to). Proven by a
  REAL proxy test: full sandbox lifecycle works through the allowlist
  while /networks, /secrets, and /images all 403 — the blast-radius cap
  if the server is ever owned. (This also fixes compose deployments,
  where sandbox provisioning previously had no engine access at all.)
- Gateway load test (plan P6): 40 concurrent SSE streams against one
  server — every run completes with the full §13 event vocabulary,
  every journal strictly monotonic, every resumeFrom=0 replay byte-equal
  to its live stream
- release.yml: SBOMs (syft, spdx-json) for all four images shipped
  INSIDE the signed bundle; final verification now runs in a
  --network none container — proving the customer's verify path needs
  no internet, not just claiming it

159 Rust tests.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-06-10 10:50:55 -05:00
Omar SobhandClaude Fable 5 fc173f170d P0 exit: e2e harness, Playwright shell journeys green, deploy skeleton
- teamclaw-server e2e mode (TEAMCLAW_MODE=e2e): idempotent deterministic
  seed through the real registration paths
- Playwright suite (6 journeys) against the real backend + prod Next build:
  login redirect, bad-password error, shell/roster/online-dot, team members,
  seeded credits, sign-out revocation — P0 exit criterion met
- Dockerfiles: musl-static server -> distroless, Next standalone -> distroless
  node (multi-arch via TARGETARCH)
- Air-gapped compose topology with edge/core/sandbox_net/secrets_net
  segmentation (engine-validated in CI), config-file + env-overlay pattern
- CI: compose validation + e2e job with trace upload

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-06-09 22:48:28 -05:00