This commit is contained in:
Omar Sobh
2026-06-15 21:51:23 -07:00
parent 1f8beb6fea
commit 3f8d2b9d51
73 changed files with 13369 additions and 0 deletions
+8
View File
@@ -0,0 +1,8 @@
/target
**/target
node_modules
dist
.env
*.log
/data
Generated
+4139
View File
File diff suppressed because it is too large Load Diff
+3
View File
@@ -0,0 +1,3 @@
[workspace]
members = ["review-api", "transcription-svc", "stream-node"]
resolver = "2"
+80
View File
@@ -0,0 +1,80 @@
# ClawReview (`mobifeedback`)
A pure-Rust real-device review platform: stream live Android handsets into the browser,
with side-by-side notes and voice-transcribed findings, for engineers and partners to
review RedClaw apps.
See [`SPEC.md`](./SPEC.md) for the full spec, PRD, and roadmap (Draft v1.1).
## Status
**Phase 0, Track 1 (review surface against a placeholder video source) — complete.**
The right-hand product and the full backend persistence path work with zero streaming
code: claim/release, debounced-autosave typed notes, voice capture with real
transcription, one-tap frame capture attached to findings, and the anchoring chain
(finding → session → device → build).
**Phase 0, Track 2 (single-device WebRTC spike) — in progress.**
- Input back-channel: scrcpy control-message encoding (touch/keycode) + normalized→device
coordinate mapping, tested against the exact scrcpy v4.0 byte layout (§4.3, §9 risk #2).
- scrcpy supervisor + H.264 demuxer (`scrcpy.rs`): pushes the server, reverse-tunnels, reads
live H.264 access units, injects taps/keys. **Validated on a real device** (LAGENIO A11
Pro) — the dumped config+keyframe decodes as valid Baseline H.264. Run:
`cargo run -p stream-node --bin scrcpy-probe -- <serial>`.
- webrtc-rs bridge (`webrtc_bridge.rs`): peer connection + H.264 track + SDP answer + access-unit
pump. Pinned to **webrtc 0.11** (the 0.20 line is an unusable sans-IO alpha — §9 #5).
- `streamd` (`src/bin/streamd.rs`): serves a WebRTC client page, brokers SDP (offer→answer), runs a
scrcpy session, pumps its H.264 into the peer connection, and forwards browser pointer/key events
to the device control socket. **The full phone→browser path is verified end to end** by
`tests/streamd_e2e.rs` (real RTP video from the device through streamd to a webrtc-rs receiver).
Run it: `cargo run -p stream-node --bin streamd` then open `http://<host>:8095/` and drive the phone.
**Phase 0 Track 2 (single-device WebRTC spike) is functionally complete** — the spike's goal (a real
device streaming to a browser with working input, no re-encode hop) is met and validated on hardware.
**Phase 1 (single-device vertical slice) — in progress.** The two Phase 0 halves are now joined: the
React review surface's left pane is a live WebRTC client (`frontend/src/components/LeftPane.tsx`) that
streams the real device from `streamd` with working touch/key control and frame capture, while the
right pane captures typed/voice findings anchored to the review-api session. The real device is
registered in the registry. Vite proxies `/api`→review-api and `/streamd`→streamd. Run all three
(review-api, streamd, `frontend` dev server) and open the app to drive the device and log findings.
Still ahead for Phase 1: user auth and the operator provisioning workflow.
## Layout
- `review-api/` — Rust / Axum / sqlx review API: registry, atomic claim/release,
findings CRUD, audio + frame upload, transcription handoff. Listens on `:8090`.
- `transcription-svc/` — self-hosted Whisper-class service (ffmpeg + whisper.cpp via
`whisper-rs`). Separate process so transcription never touches the real-time path.
Listens on `:8099`.
- `stream-node/` — per-device streaming plane (§6.1). Currently the input back-channel:
scrcpy control-protocol encoding + coordinate mapping. WebRTC/scrcpy wiring pending.
- `frontend/` — React + TypeScript + Tailwind v4 split-screen review surface.
- `docker-compose.yml` — Postgres 16 for local dev.
## Run it locally
```sh
docker compose up -d postgres # Postgres on :5432
./transcription-svc/scripts/fetch-model.sh tiny.en # download a ggml model (once)
WHISPER_MODEL=transcription-svc/models/ggml-tiny.en.bin cargo run -p transcription-svc # :8099
cargo run -p review-api # :8090 (auto-migrates + seeds a placeholder device)
cd frontend && npm install && npm run dev # Vite on :5173, proxies /api -> :8090
```
Open http://localhost:5173, claim the placeholder device, and capture findings. Voice
recording and frame capture need a real browser (mic + canvas permissions).
## Tests
```sh
export DATABASE_URL=postgres://clawreview:clawreview@localhost:5432/clawreview
cargo test --workspace # API + transcription HTTP contract (ephemeral DBs)
cargo test -p transcription-svc --test whisper -- --ignored # real whisper.cpp E2E (needs model + ffmpeg)
```
Config via env: `DATABASE_URL`, `BIND_ADDR`, `STORAGE_DIR` (local R2 stand-in, default
`./data`), `TRANSCRIBE_URL` (review-api → transcription-svc, default `http://127.0.0.1:8099/transcribe`),
`WHISPER_MODEL` (transcription-svc model path).
+374
View File
@@ -0,0 +1,374 @@
# ClawReview — Platform Plan (Revised v1.1)
**A pure-Rust real-device review platform: stream live Android handsets into the browser, with side-by-side notes and voice-transcribed findings, for engineers and partners to review RedClaw apps.**
- Document type: Combined Spec + PRD + Roadmap
- Owner: Omar Sobh, RedClaw Systems LLC
- Status: Draft v1.1
- Date: June 2026
## Changelog vs v1.0
- §3 / §10 Phase 3 / §11 — remote = software-only; on-site operator owns physical hardware events.
- §4.3 — input back-channel settled on WebSocket (was "data channel or WebSocket").
- §4.4 / §6.2 — atomic claim (conditional UPDATE), heartbeat, orphaned-claim timeout, stream-node reconciliation.
- §4.4 / §4.5 / §6.2 / §8 — device state reset on release.
- §4.5 / §7 — findings carry a captured frame **and** a session timestamp offset; optional short clip noted for temporal findings.
- Media transport stays pure-Rust webrtc-rs everywhere (v1.0 decision retained).
---
## 1. Executive Summary
ClawReview is an internal and partner-facing web platform that lets a reviewer drive a
real, physical Android device from a browser while capturing structured findings beside
it. The screen of each handset is streamed live into the left half of the browser. The
right half holds two stacked review surfaces: a text notes panel on top and a voice
recorder with automatic transcription beneath it. Reviewers claim a device, navigate the
pre-installed app, type or speak their findings, and every finding is automatically
anchored to the device, the build under review, the reviewer, a timestamp, and a captured
frame.
The platform exists because several RedClaw products (Digital Hallmark, CardClaws,
StackSticker) depend on native hardware behavior — NFC and UHF RFID — that no emulator can
reproduce. Reviews must run on real silicon near real readers. ClawReview turns a rack of
physical phones into a shared, remotely accessible review fleet, so engineers on-site and
partners across MENA, Southeast Asia, and the US can review the same handsets without
shipping hardware. On-site engineers drive both software and hardware flows directly;
remote partners drive software flows, with hardware events presented by an on-site
operator.
The system is built pure-Rust on the backend (Axum, Tokio, the `webrtc` crate) and React
with TailwindCSS and shadcn/ui on the frontend. Owning the full WebRTC media path in Rust
— rather than delegating it to a third-party media server — keeps the entire stack in one
language, keeps confidential partner builds on RedClaw infrastructure, and gives full
control over the latency-critical streaming and input back-channel.
---
## 2. Goals and Non-Goals
### 2.1 Goals
See and control a real Android device from any modern browser at responsive latency.
Support multiple physical devices per host, each independently claimable. Capture typed and
spoken findings, transcribe speech on RedClaw infrastructure, and anchor every finding to
device, build, reviewer, time, and on-screen frame. Work for on-site engineers (LAN) and
remote partners (relayed) through the same pipeline. Keep all confidential builds and all
reviewer findings inside RedClaw-controlled infrastructure at every step.
### 2.2 Non-Goals (v1)
No software emulators — real devices only. No iOS (cannot run on Linux hosts; a simulator
cannot exercise NFC/RFID). No app install at review time — builds are pre-installed out of
band by an operator. No automated test execution, scripting, or CI integration. No full
session video recording (frame captures, plus optional short clips, only). No app signing
or Play Store provisioning beyond the documented operator install step. **No remote
partner-driven physical hardware interaction** — remote partners review software flows;
physical NFC/RFID presentation is an on-site operator action.
---
## 3. Personas and Primary Use Cases
The **on-site engineer** sits on the same network as the rig. They claim a handset,
exercise a build end to end — including physically tapping NFC/RFID against a nearby reader
— and log defects as fast typed notes. Latency is near zero; their priority is throughput
and precise frame-anchored notes.
The **remote partner** (Jakarta, Istanbul, Muscat, …) reviews a build running on a phone
physically located in the US. Their stream is relayed through coturn. They drive the
software/UI flows and dictate findings by voice when typing in a second language is slower.
Because they cannot physically present a tag to the US-located reader, **any hardware step
in their review is triggered by an on-site operator**, with the partner watching the result
resolve on screen and logging the finding. Confidentiality matters most here: their findings
and the build they see must never leave RedClaw infrastructure.
The **review operator / admin** provisions the rig: attaches phones, installs and tags
builds, registers devices, monitors fleet health, and — for remote sessions — performs the
physical hardware presentation on request. They never author review findings themselves;
they keep the fleet ready and the hardware exercised.
**Defining use case (on-site):** an operator installs the latest Digital Hallmark build on
three handsets and registers them. An on-site engineer sees three available devices labeled
with the build version, claims one, drives the provenance flow, taps a real RFID tag against
the reader attached to that phone, watches the read resolve, and dictates a finding about
read latency. The finding is saved with the device serial, the build hash, the engineer's
identity, the timestamp, the session time-offset, and a captured frame (and, for a timing
finding, an optional short clip) of the result screen. The engineer releases the device; its
app state is reset and it returns to the available pool.
**Defining use case (remote):** a partner in Istanbul claims one of the same devices, drives
the on-screen provenance flow, and — at the read step — asks the on-site operator (in-session)
to present the tag. The operator taps; the partner watches the read resolve, captures the
frame, and dictates the latency finding. Same anchoring chain, same confidentiality guarantees.
---
## 4. System Architecture
### 4.1 Overview
Two backend planes, a frontend, a relay, and the physical rig.
- **Stream node** — latency-sensitive, per-device, co-located with the rig. Manages the
scrcpy lifecycle over ADB, reads each device's H.264 stream, packetizes into WebRTC, runs
per-device peer connections, and routes input back to the device via the scrcpy control
protocol. Home of the `webrtc` crate, H.264 handling, and Tokio per-device tasks.
- **Review API** — request/response plane, runs anywhere reachable by reviewers and the
stream node. Owns the device registry, claim/release orchestration, signaling brokerage,
auth, the findings store, audio upload, and the transcription handoff. Talks to PostgreSQL
and Cloudflare R2.
- **Frontend** — React/Tailwind/shadcn split-screen surface: WebRTC client + input-capture
on the left, notes/voice review on the right.
- **TURN relay** — existing RedClaw coturn (reused from LiveCast) for reviewers who can't
reach the stream node directly (most remote partners).
- **Device rig** — Android handsets on a powered USB hub on the stream-node host.
### 4.2 Media Path (device → browser)
Each device runs the scrcpy server (pushed via ADB), emitting an H.264 elementary stream on
a local socket plus a control socket for input injection. The stream node reads the H.264
stream and — because browsers reject H.264 with B-frames over WebRTC — ensures the encoder is
configured for a baseline-compatible, B-frame-free profile so the stream forwards without
re-encoding. Access units feed a per-device `webrtc`-crate peer connection handling RTP
packetization, SPS/PPS parameter-set handling, keyframe cadence, SRTP, and ICE. The browser
receives the track on a standard `<video>` element. Where a device model's encoder cannot
avoid B-frames, a per-stream re-encode hop is the documented fallback at a measured CPU cost;
the default path assumes no re-encode.
### 4.3 Input Path (browser → device)
The hardest custom component, with no off-the-shelf pure-Rust equivalent. The browser
captures pointer/touch/keyboard events over the video element and scales each event's
coordinates from the rendered element size to the device's real resolution, accounting for
current orientation. **These normalized events travel over the existing session WebSocket**
(ordered, reliable, trivially debuggable — chosen over a WebRTC data channel for simplicity;
the LAN latency delta is negligible and the control channel is low-bandwidth). The stream
node validates and translates them into scrcpy control-protocol messages written to the
device's control socket. Coordinate-scaling is mirrored: the frontend scales into device
space; the backend re-validates and forwards.
### 4.4 Signaling and Session Lifecycle
A reviewer claims a device through the review API. **The claim is atomic** — a conditional
`UPDATE devices SET status='claimed', … WHERE serial=$1 AND status='free' RETURNING …` so two
reviewers can never hold the same handset (no read-then-write race). On success the API
brokers the WebRTC handshake: SDP offer/answer and ICE candidates pass over a WebSocket
between browser and stream node, with the review API mediating authorization. ICE negotiates
the best path — direct on LAN, relayed through coturn for remote partners.
**A session heartbeat** (browser → API) keeps the claim alive. On explicit release,
disconnect, missed-heartbeat timeout, or admin override, the peer connection tears down, the
scrcpy session resets cleanly, **the app-under-review's state is reset** (clear app data /
relaunch, see §8), and the device returns to the available pool. The stream node reports
device health to the API, which **reconciles** registry status against that health (e.g. a
stream-node restart re-derives free/claimed/offline) so an orphaned claim never strands a
device permanently.
### 4.5 Review Surface (right pane)
The notes panel is a debounced-autosave text surface writing to the findings store, each note
carrying the active session's anchoring metadata. The voice panel captures microphone audio
via the MediaRecorder API, uploads the blob to the review API, which stores it in R2 and
submits it to a self-hosted Whisper-class transcription service on RedClaw infrastructure
(Hetzner/Vultr) so audio and transcripts never leave RedClaw control; the transcript renders
inline beneath the recording. A one-tap **capture-frame** control draws the current `<video>`
frame to a canvas, producing a still that anchors the finding visually **and records the
session time-offset** so the finding is locatable in time, not just in space. For temporal
findings (e.g. read latency), an **optional short clip** (a few seconds of rolling buffer
around the capture) may be retained — single stills cannot evidence timing, which is exactly
what several target findings are about. Full session recording remains a non-goal.
### 4.6 Why Pure Rust, and What That Commits Us To
Owning the WebRTC media plane in Rust via the `webrtc` crate keeps the backend in one language
and toolchain, keeps confidential builds and findings fully on RedClaw infrastructure, and
gives complete control over the latency-critical path. The commitment is real: the `webrtc`
crate is younger and less battle-hardened than the Go/C++ equivalents, and the
H.264-to-WebRTC packetization and parameter-set negotiation is the single most failure-prone
area of the system. This risk is concentrated, identified, and addressed first via a
single-device spike before any orchestration is built around it.
---
## 5. Technical Stack
Backend: Rust throughout. Axum on Tokio. WebRTC via the `webrtc` crate (peer connections,
RTP/SRTP, ICE). Device comms via ADB + the scrcpy server protocol (video and control).
PostgreSQL for registry, sessions, users, and findings metadata. Cloudflare R2 for audio
blobs, frame captures, and optional clips. Transcription: a self-hosted Whisper-class service
on Hetzner/Vultr, called over HTTP from the review API so it stays off the real-time path.
TURN relay: existing RedClaw coturn.
Frontend: React + TypeScript, TailwindCSS, shadcn/ui. Left pane uses native
`RTCPeerConnection` and `<video>`, the peer connection held in an explicit owned lifecycle
object (never tied to component render). Right pane uses MediaRecorder and shadcn primitives
(resizable panels, tabs, cards).
Infra follows RedClaw standards: Hetzner/Vultr compute, Cloudflare R2, coturn relay. The
stream node is co-located with the rig.
---
## 6. Component Specifications
### 6.1 Stream Node (Rust)
scrcpy supervisor (push server per device, open video/control sockets, detect drops/USB
disconnects, recover cleanly), H.264 reader + WebRTC bridge (read access units, packetize
into RTP, manage parameter sets and keyframe cadence, run the per-device peer connection),
and control receiver (validate normalized input, translate to scrcpy control messages, write
to the control socket). Exposes its half of the signaling handshake and reports device health
to the review API. Resilient by design: one device dropping must not disturb the others.
### 6.2 Review API (Rust / Axum)
Owns the device registry (serial → installed build/version → status: free/claimed/offline),
populated by operator provisioning and kept current by stream-node health signals. Owns
claim/release with the **atomic conditional-update claim model** and **heartbeat-based
orphan recovery** of §4.4, including **app-state reset on release**. Brokers signaling.
Authenticates users and gates partner access. Manages findings CRUD, audio upload to R2, and
the transcription handoff. The registry is the source of truth that auto-stamps every finding
with the correct build metadata.
### 6.3 Frontend (React / Tailwind / shadcn)
Split-screen shell with resizable panes. Left: WebRTC video consumer + input-capture and
coordinate-scaling layer. Right: debounced-autosave notes, MediaRecorder voice capture with
playback and inline transcript, one-tap frame capture (still + time-offset, optional clip).
Three state concerns kept deliberately separate: the WebRTC/peer-connection lifecycle, the
claim state, and the findings buffer. The peer connection is owned explicitly and never
re-created on render.
### 6.4 Transcription Service (self-hosted)
A Whisper-class model over HTTP on RedClaw infrastructure. Accepts an audio blob, returns a
transcript. Separate from the review API so transcription load never touches the real-time
path; on RedClaw infrastructure so partner findings never leave RedClaw control. Model size
chosen for multilingual accuracy (Turkish, Indonesian, Arabic) within latency budget.
### 6.5 Provisioning / Admin (out of band)
An operator workflow — not coupled to runtime sessions — to attach devices, install builds via
ADB, tag each device with build/version, and register/retire devices in the registry. Because
installs happen here, the runtime session layer never handles APK installs, signing, or install
latency. The operator also performs in-session physical hardware presentation for remote partners.
---
## 7. Data Model (high level)
- **device** — serial, model, status, attached stream-node, currently installed build reference.
- **build** — app identity, version, hash, install metadata.
- **session** — links a reviewer to a device for a time window; references the build installed
at claim time; carries a claim start time so findings can record a relative time-offset.
- **finding** — note text or audio reference + transcript; anchors to session, device serial,
build hash, reviewer, timestamp, **session time-offset**, and an optional captured-frame and
optional short-clip reference.
Audio blobs, frame captures, and clips live in R2; their references live in PostgreSQL. The
anchoring chain finding → session → device → build guarantees every finding is unambiguously
attributable.
---
## 8. Security, Privacy, and Access
All confidential builds remain on RedClaw infrastructure — installed on physically controlled
devices, never uploaded to third-party services. All reviewer findings (typed, spoken,
transcribed) stay on RedClaw infrastructure end to end, including transcription. Partner access
is authenticated and gated; partners see and claim only the devices and builds intended for
them. Remote streams relay through RedClaw-controlled coturn. The threat model treats partner
accounts as semi-trusted: scoped access, no fleet-wide visibility, clear separation of what
each partner may review.
**Cross-session device isolation:** because one physical phone is shared across partners and
builds, the app-under-review's data and on-screen state persist between claims unless cleared.
On every release the lifecycle **resets app state** (clear app data / relaunch) so one
reviewer's session — including any confidential interaction — does not leak into the next.
---
## 9. Key Risks and Mitigations
1. **H.264→WebRTC on the `webrtc` crate** (dominant) — parameter-set handling, profile
negotiation, packetization; a mismatch yields a black screen that is painful to debug.
Mitigated by isolating it in the Phase 0 single-device spike before anything depends on it,
and by configuring scrcpy for a B-frame-free, baseline-compatible profile.
2. **Input back-channel coordinate scaling** across rendered-vs-device resolution and
orientation; if wrong, taps land in the wrong place. Mitigated by mirroring scaling on both
ends and validating against multiple device models.
3. **Physical rig** — phones drop off USB under load. Mitigated with a powered USB hub,
per-device drop detection/recovery in the scrcpy supervisor, and treating one device's
failure as isolated.
4. **Remote stream quality** across long internet paths. Mitigated by the coturn relay,
TCP-transport fallback when UDP is blocked, and setting partner latency expectations.
5. **`webrtc`-crate maturity under multi-stream load.** Mitigated by validating the
single-device path thoroughly first and scaling stream count deliberately while watching for
instability, with the explicit option to reassess if the crate cannot carry target concurrency.
6. **Orphaned claims / stale registry** — a crashed browser or stream-node restart stranding a
device. Mitigated by the atomic claim, session heartbeat with timeout, and stream-node ↔
registry health reconciliation (§4.4).
---
## 10. Roadmap
### Phase 0 — Foundations and De-Risking
Two parallel tracks (no shared dependencies):
- **Review surface end to end against a placeholder video source** — split-screen shell, notes
with autosave, voice capture with MediaRecorder, the self-hosted transcription service, R2
audio storage, the findings data model, and the anchoring chain — all without streaming code.
Validates the entire right-hand product and the full persistence path at zero streaming risk.
- **Single-device WebRTC spike** — one phone, scrcpy video into the `webrtc` crate, into one
browser tab, with working taps over the WebSocket back-channel. No orchestration, no TURN, no
multi-device. Proves/disproves the riskiest decision cheaply. Also determines, per target
device model, whether the encoder needs the baseline re-encode hop, setting the per-stream CPU
budget and therefore device density per host.
**Exit:** review surface works against a placeholder; one phone streams to one browser with
working input.
### Phase 1 — Single-Device Vertical Slice
Join the two tracks: a real device streams into the real review surface; a reviewer drives it
and logs typed and voice findings; every finding is correctly anchored (device, build, reviewer,
time, time-offset, frame). Add user authentication and the operator provisioning workflow (install,
tag, register). **Include a minimal on-site NFC/RFID smoke test** — one hardware flow on one
device — so the premise that justified real devices is validated before the fleet is built, not
deferred to Phase 4.
**Exit:** an on-site engineer reviews a real pre-installed build on one device with full finding
capture, including at least one physical hardware flow.
### Phase 2 — Multi-Device Fleet and Orchestration
Scale to the rig: device registry as live source of truth, the atomic claim/release model with
heartbeat and reconciliation, fleet-health monitoring, and the stream node's capacity for multiple
concurrent per-device peer connections and scrcpy sessions. Validate device density per host
against the Phase 0 CPU budget. Harden the USB rig and the scrcpy supervisor's drop-recovery.
**Exit:** multiple on-site engineers concurrently review different devices in the same fleet.
### Phase 3 — Remote Partners and Relay
Bring in the coturn-relayed path so remote partners review US-located devices. Validate ICE path
selection (direct on LAN, relayed for remote), TCP-transport fallback for restrictive firewalls,
and partner-scoped auth/access. **Implement the operator-assisted hardware flow** for remote
sessions: an in-session channel by which a remote partner requests a physical tag presentation and
the on-site operator performs it while the partner watches and logs the finding. Tune for
cross-continental latency and set partner expectations. Confirm end to end that partner builds and
findings never leave RedClaw infrastructure.
**Exit:** a partner abroad reviews a US-located device's software flows, with operator-assisted
hardware steps, confidentiality preserved end to end.
### Phase 4 — Hardening and Native-Flow Coverage
Validate the full NFC/UHF RFID review workflows against physical readers on specific handsets for
Digital Hallmark, CardClaws, and StackSticker, on-site and operator-assisted-remote. Strengthen
resilience, observability, and recovery across the fleet. Refine the review surface from real
feedback (frame-anchoring ergonomics, temporal-clip ergonomics, transcript editing).
**Exit:** the platform reliably supports the NFC/RFID review workflows that motivated it, for
on-site and (operator-assisted) remote reviewers.
### Future Considerations (post-v1)
iOS via Mac hosts and real iPhones. Full session recording/playback. Multiple stream-node hosts and
geographic rig distribution. Automated/scripted review flows. An automated tag-presentation rig so
remote partners can trigger hardware reads without an on-site operator. Deeper integration with
RedClaw build pipelines (EAS/Expo build → rig).
---
## 11. Success Criteria
The platform succeeds when an on-site engineer or remote partner can, without shipping hardware,
claim a real device from a browser, drive a pre-installed RedClaw build — its full NFC/RFID flows
on-site, its software flows with operator-assisted hardware steps remotely — and capture typed and
spoken findings automatically and unambiguously anchored to what they reviewed (device, build,
reviewer, time, time-offset, frame), with all builds and findings remaining entirely on RedClaw
infrastructure — on-site at near-zero latency and remotely over a relayed but usable stream.
+19
View File
@@ -0,0 +1,19 @@
services:
postgres:
image: postgres:16
environment:
POSTGRES_USER: clawreview
POSTGRES_PASSWORD: clawreview
POSTGRES_DB: clawreview
ports:
- "5432:5432"
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U clawreview"]
interval: 2s
timeout: 3s
retries: 30
volumes:
pgdata:
+1
View File
@@ -0,0 +1 @@
certs/
+12
View File
@@ -0,0 +1,12 @@
<!doctype html>
<html lang="en" class="dark">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>ClawReview</title>
</head>
<body>
<div id="root"></div>
<script type="module" src="/src/main.tsx"></script>
</body>
</html>
+3462
View File
File diff suppressed because it is too large Load Diff
+39
View File
@@ -0,0 +1,39 @@
{
"name": "clawreview-frontend",
"private": true,
"version": "0.1.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"preview": "vite preview"
},
"dependencies": {
"@radix-ui/react-dialog": "^1.1.17",
"@radix-ui/react-dropdown-menu": "^2.1.18",
"@radix-ui/react-scroll-area": "^1.2.12",
"@radix-ui/react-separator": "^1.1.10",
"@radix-ui/react-slot": "^1.3.0",
"@radix-ui/react-tabs": "^1.1.15",
"@radix-ui/react-tooltip": "^1.2.10",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"cmdk": "^1.1.1",
"lucide-react": "^1.18.0",
"react": "^18.3.1",
"react-dom": "^18.3.1",
"react-resizable-panels": "^2.1.7",
"sonner": "^2.0.7",
"tailwind-merge": "^3.6.0"
},
"devDependencies": {
"@tailwindcss/vite": "^4.0.0",
"@types/node": "^25.9.3",
"@types/react": "^18.3.12",
"@types/react-dom": "^18.3.1",
"@vitejs/plugin-react": "^4.3.4",
"tailwindcss": "^4.0.0",
"typescript": "^5.6.3",
"vite": "^6.0.5"
}
}
+215
View File
@@ -0,0 +1,215 @@
import { useEffect, useMemo, useRef, useState } from "react";
import { toast } from "sonner";
import { Settings as SettingsIcon, MonitorSmartphone } from "lucide-react";
import { api, type Device, type Session } from "./api";
import { ReviewShell } from "./components/ReviewShell";
import { SettingsDialog } from "./components/SettingsDialog";
import { CommandPalette, type CommandAction } from "./components/CommandPalette";
import { DeviceCard } from "./components/DeviceCard";
import { Button } from "@/components/ui/button";
// Top-level owns the *claim state* only — one of the three deliberately separate
// state concerns from §6.3 (claim / findings buffer / peer-connection lifecycle).
export default function App() {
const [devices, setDevices] = useState<Device[]>([]);
const [session, setSession] = useState<Session | null>(null);
const [error, setError] = useState<string | null>(null);
const [settingsOpen, setSettingsOpen] = useState(false);
const [paletteOpen, setPaletteOpen] = useState(false);
// Bump every 5 minutes to refresh device thumbnails.
const [thumbTick, setThumbTick] = useState(0);
useEffect(() => {
const t = window.setInterval(() => setThumbTick((n) => n + 1), 5 * 60 * 1000);
return () => window.clearInterval(t);
}, []);
const refreshDevices = () =>
api.listDevices().then(setDevices).catch((e) => setError(String(e)));
useEffect(() => {
refreshDevices();
}, []);
// Live registry/session events (Workstream 2): update the device list in place and
// surface toasts, replacing the poll-only model.
useEffect(() => {
const es = new EventSource("/api/events");
es.onmessage = (e) => {
const ev = JSON.parse(e.data);
if (ev.type === "device_status") {
setDevices((prev) =>
prev.map((d) => (d.serial === ev.serial ? { ...d, status: ev.status } : d)),
);
if (ev.status === "offline") toast.warning(`${ev.serial} went offline`);
} else if (ev.type === "session_ended") {
setSession((cur) => {
if (cur && cur.id === ev.session_id) {
toast.warning("Session ended — device released or taken over");
refreshDevices();
return null;
}
return cur;
});
}
};
return () => es.close();
}, []);
// Keep the claim alive while a session is held. The heartbeat reports whether the
// reviewer interacted since the last beat, so a dormant (no-interaction) session is
// auto-reclaimed even with the tab open; a closed tab releases via the unload beacon.
const activeSinceBeat = useRef(false);
useEffect(() => {
if (!session) return;
const id = session.id;
const markActive = () => {
activeSinceBeat.current = true;
};
window.addEventListener("pointerdown", markActive);
window.addEventListener("keydown", markActive);
window.addEventListener("wheel", markActive, { passive: true });
const beat = () => {
api.heartbeat(id, activeSinceBeat.current);
activeSinceBeat.current = false;
};
beat();
const timer = window.setInterval(beat, 10_000);
const onUnload = () => navigator.sendBeacon(`/api/sessions/${id}/release`);
window.addEventListener("pagehide", onUnload);
return () => {
window.clearInterval(timer);
window.removeEventListener("pagehide", onUnload);
window.removeEventListener("pointerdown", markActive);
window.removeEventListener("keydown", markActive);
window.removeEventListener("wheel", markActive);
};
}, [session]);
async function claim(serial: string) {
setError(null);
try {
const s = await api.createSession(serial);
setSession(s);
const model = devices.find((d) => d.serial === serial)?.model ?? serial;
toast.success(`Claimed ${model}`);
} catch (e) {
const msg = String(e);
setError(msg);
toast.error(msg.replace(/^Error:\s*/, ""));
refreshDevices();
}
}
async function release() {
if (!session) return;
try {
await api.releaseSession(session.id);
toast("Device released");
} finally {
setSession(null);
refreshDevices();
}
}
async function switchDevice(serial: string) {
if (session) {
try {
await api.releaseSession(session.id);
} catch {
/* ignore */
}
setSession(null);
}
await claim(serial);
}
const actions = useMemo<CommandAction[]>(() => {
const list: CommandAction[] = [];
for (const d of devices) {
if (d.status === "offline") continue;
const claimed = d.serial === session?.device_serial;
if (claimed) continue;
list.push({
id: `device-${d.serial}`,
group: session ? "Switch device" : "Devices",
label: `${d.status === "claimed" ? "Take over" : "Claim"} ${d.model ?? d.serial}`,
icon: <MonitorSmartphone />,
onSelect: () => switchDevice(d.serial),
});
}
if (session) {
list.push({ id: "release", group: "Session", label: "Release device", onSelect: release });
}
list.push({
id: "settings",
group: "App",
label: "Open settings",
icon: <SettingsIcon />,
onSelect: () => setSettingsOpen(true),
});
return list;
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [devices, session]);
const overlays = (
<>
<SettingsDialog open={settingsOpen} onOpenChange={setSettingsOpen} />
<CommandPalette open={paletteOpen} onOpenChange={setPaletteOpen} actions={actions} />
</>
);
if (session) {
const device = devices.find((d) => d.serial === session.device_serial) ?? null;
return (
<>
<ReviewShell
session={session}
device={device}
devices={devices}
onRelease={release}
onSwitch={switchDevice}
onOpenSettings={() => setSettingsOpen(true)}
onOpenPalette={() => setPaletteOpen(true)}
/>
{overlays}
</>
);
}
return (
<div className="mx-auto max-w-3xl px-6 py-10">
<header className="mb-8 flex items-start justify-between">
<div>
<h1 className="text-2xl font-semibold tracking-tight">ClawReview</h1>
<p className="mt-1 text-sm text-muted-foreground">
Claim a device to start a review session.{" "}
<button onClick={() => setPaletteOpen(true)} className="underline-offset-2 hover:underline">
⌘K
</button>{" "}
for commands.
</p>
</div>
<Button variant="ghost" size="icon" onClick={() => setSettingsOpen(true)} aria-label="Settings">
<SettingsIcon />
</Button>
</header>
{error && (
<div className="mb-4 rounded-md border border-destructive/40 bg-destructive/10 px-4 py-2 text-sm text-destructive">
{error}
</div>
)}
<div className="grid grid-cols-1 gap-4 sm:grid-cols-2 lg:grid-cols-3">
{devices.map((d) => (
<DeviceCard key={d.serial} device={d} thumbTick={thumbTick} onClaim={() => claim(d.serial)} />
))}
{devices.length === 0 && !error && (
<p className="text-sm text-muted-foreground">No devices registered.</p>
)}
</div>
{overlays}
</div>
);
}
+122
View File
@@ -0,0 +1,122 @@
// Thin client over the review-api (§6.2). Types mirror the Rust models.
export interface Device {
serial: string;
model: string | null;
status: "free" | "claimed" | "offline";
stream_node: string | null;
current_build_id: string | null;
updated_at: string;
}
export interface Session {
id: string;
reviewer_id: string;
device_serial: string;
build_id: string | null;
started_at: string;
ended_at: string | null;
}
export interface Finding {
id: string;
session_id: string;
device_serial: string;
build_hash: string | null;
reviewer_id: string;
kind: "note" | "voice";
note_text: string | null;
audio_ref: string | null;
transcript: string | null;
frame_ref: string | null;
clip_ref: string | null;
time_offset_ms: number | null;
created_at: string;
}
async function json<T>(res: Response): Promise<T> {
if (!res.ok) {
const body = await res.json().catch(() => ({}));
throw new Error(body.error ?? `${res.status} ${res.statusText}`);
}
return res.json() as Promise<T>;
}
export const api = {
listDevices: () => fetch("/api/devices").then(json<Device[]>),
createSession: (deviceSerial: string) =>
fetch("/api/sessions", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ device_serial: deviceSerial }),
}).then(json<Session>),
releaseSession: (sessionId: string) =>
fetch(`/api/sessions/${sessionId}/release`, { method: "POST" }).then((r) => {
if (!r.ok) throw new Error("release failed");
}),
// `active` marks recent interaction since the last beat; a dormant (no-interaction)
// session ages toward auto-reclaim even while its tab stays open.
heartbeat: (sessionId: string, active = false) =>
fetch(`/api/sessions/${sessionId}/heartbeat?active=${active}`, { method: "POST" })
.then((r) => r.ok)
.catch(() => false),
listFindings: (sessionId: string) =>
fetch(`/api/sessions/${sessionId}/findings`).then(json<Finding[]>),
createNote: (sessionId: string, noteText: string) =>
fetch(`/api/sessions/${sessionId}/findings`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ kind: "note", note_text: noteText }),
}).then(json<Finding>),
updateNote: (findingId: string, noteText: string) =>
fetch(`/api/findings/${findingId}`, {
method: "PATCH",
headers: { "content-type": "application/json" },
body: JSON.stringify({ note_text: noteText }),
}).then(json<Finding>),
uploadAudio: (sessionId: string, blob: Blob) => {
const form = new FormData();
form.append("audio", blob, "recording.webm");
return fetch(`/api/sessions/${sessionId}/findings/audio`, {
method: "POST",
body: form,
}).then(json<Finding>);
},
uploadFrame: (findingId: string, blob: Blob) => {
const form = new FormData();
form.append("frame", blob, "frame.png");
return fetch(`/api/findings/${findingId}/frame`, {
method: "POST",
body: form,
}).then(json<Finding>);
},
// Transcribe audio without persisting — for live, interim dictation.
transcribeLive: (blob: Blob) => {
const form = new FormData();
form.append("audio", blob, "live.webm");
return fetch("/api/transcribe", { method: "POST", body: form }).then(
json<{ text: string }>,
);
},
// Create a standalone screen-snapshot finding from a captured frame.
createFrameFinding: (sessionId: string, blob: Blob) => {
const form = new FormData();
form.append("frame", blob, "frame.png");
return fetch(`/api/sessions/${sessionId}/findings/frame`, {
method: "POST",
body: form,
}).then(json<Finding>);
},
frameUrl: (findingId: string) => `/api/findings/${findingId}/frame`,
};
@@ -0,0 +1,68 @@
import { useEffect, type ReactNode } from "react";
import {
CommandDialog,
CommandEmpty,
CommandGroup,
CommandInput,
CommandItem,
CommandList,
} from "@/components/ui/command";
export interface CommandAction {
id: string;
label: string;
group: string;
icon?: ReactNode;
onSelect: () => void;
}
export function CommandPalette({
open,
onOpenChange,
actions,
}: {
open: boolean;
onOpenChange: (open: boolean) => void;
actions: CommandAction[];
}) {
// Global ⌘K / Ctrl-K toggle.
useEffect(() => {
const onKey = (e: KeyboardEvent) => {
if (e.key === "k" && (e.metaKey || e.ctrlKey)) {
e.preventDefault();
onOpenChange(!open);
}
};
window.addEventListener("keydown", onKey);
return () => window.removeEventListener("keydown", onKey);
}, [open, onOpenChange]);
const groups = Array.from(new Set(actions.map((a) => a.group)));
return (
<CommandDialog open={open} onOpenChange={onOpenChange}>
<CommandInput placeholder="Type a command or search…" />
<CommandList>
<CommandEmpty>No matching commands.</CommandEmpty>
{groups.map((group) => (
<CommandGroup key={group} heading={group}>
{actions
.filter((a) => a.group === group)
.map((a) => (
<CommandItem
key={a.id}
onSelect={() => {
onOpenChange(false);
a.onSelect();
}}
>
{a.icon}
{a.label}
</CommandItem>
))}
</CommandGroup>
))}
</CommandList>
</CommandDialog>
);
}
+60
View File
@@ -0,0 +1,60 @@
import { useEffect, useState } from "react";
import { MonitorSmartphone } from "lucide-react";
import { Button } from "@/components/ui/button";
import { Badge } from "@/components/ui/badge";
import { Card } from "@/components/ui/card";
import type { Device } from "../api";
// A device tile: title, a live-ish screen thumbnail (captured by streamd via adb
// screencap, refreshed on each `thumbTick`), status, and the claim action.
export function DeviceCard({
device,
thumbTick,
onClaim,
}: {
device: Device;
thumbTick: number;
onClaim: () => void;
}) {
const [errored, setErrored] = useState(false);
useEffect(() => setErrored(false), [thumbTick]); // retry on each refresh
const variant =
device.status === "free" ? "success" : device.status === "claimed" ? "warning" : "muted";
const src = `/streamd/thumbnail?serial=${encodeURIComponent(device.serial)}&t=${thumbTick}`;
const showImage = !errored && device.status !== "offline";
return (
<Card className="flex flex-col overflow-hidden">
<div className="flex items-center justify-between gap-2 px-3 py-2">
<div className="truncate font-medium">{device.model ?? device.serial}</div>
<Badge variant={variant}>{device.status}</Badge>
</div>
<div className="flex aspect-[9/16] items-center justify-center border-y border-border bg-black">
{showImage ? (
<img
src={src}
alt={`${device.model ?? device.serial} screen`}
onError={() => setErrored(true)}
className="h-full w-full object-contain"
/>
) : (
<MonitorSmartphone className="size-10 text-muted-foreground/40" />
)}
</div>
<div className="flex items-center justify-between gap-2 p-3">
<span className="truncate text-xs text-muted-foreground">{device.serial}</span>
<Button
size="sm"
disabled={device.status === "offline"}
variant={device.status === "claimed" ? "secondary" : "default"}
onClick={onClaim}
>
{device.status === "claimed" ? "Take over" : "Claim"}
</Button>
</div>
</Card>
);
}
+38
View File
@@ -0,0 +1,38 @@
import { api, type Finding } from "../api";
// Read-only view of the session's findings buffer, each showing the anchoring that
// makes it attributable (§7): build hash + session time-offset. Frame snapshots show
// a thumbnail served by the review API.
export function FindingsList({ findings }: { findings: Finding[] }) {
return (
<div className="flex h-full flex-col p-3">
<h2 className="mb-2 text-sm font-semibold text-foreground">
Findings <span className="text-muted-foreground/60">({findings.length})</span>
</h2>
<div className="flex-1 space-y-2 overflow-y-auto">
{findings.map((f) => (
<div key={f.id} className="rounded border border-border bg-card/40 p-2">
<div className="mb-1 flex items-center gap-2 text-xs text-muted-foreground">
<span className="rounded bg-muted px-1.5 py-0.5 uppercase">{f.kind}</span>
<span>build {f.build_hash ?? "—"}</span>
{f.time_offset_ms != null && <span>· t+{(f.time_offset_ms / 1000).toFixed(1)}s</span>}
</div>
{f.frame_ref && (
<img
src={api.frameUrl(f.id)}
alt="screen capture"
className="mb-1 max-h-40 w-auto rounded border border-border"
/>
)}
{f.kind === "voice" ? (
<div className="text-sm italic text-muted-foreground">{f.transcript ?? "(no transcript)"}</div>
) : (
f.note_text && <div className="text-sm text-foreground">{f.note_text}</div>
)}
</div>
))}
{findings.length === 0 && <p className="text-sm text-muted-foreground/60">No findings yet.</p>}
</div>
</div>
);
}
+155
View File
@@ -0,0 +1,155 @@
import { useEffect, useRef, useState } from "react";
// Live WebRTC client for the device feed (§4.2/§4.3). Connects to streamd: POSTs an
// SDP offer, renders the H.264 track in <video>, and forwards pointer/key events over
// the input WebSocket. The peer connection is owned in a ref and never recreated on
// render (§6.3). Frame capture draws the real <video> to a canvas.
const OFFER_URL = "/streamd/offer";
function inputWsUrl(): string {
const proto = location.protocol === "https:" ? "wss" : "ws";
return `${proto}://${location.host}/streamd/input`;
}
function iceComplete(pc: RTCPeerConnection): Promise<void> {
return new Promise((resolve) => {
if (pc.iceGatheringState === "complete") return resolve();
pc.addEventListener("icegatheringstatechange", () => {
if (pc.iceGatheringState === "complete") resolve();
});
});
}
export function LeftPane({
label,
onCapture,
onStatus,
}: {
label: string;
onCapture: (dataUrl: string) => void;
onStatus?: (status: string) => void;
}) {
const videoRef = useRef<HTMLVideoElement>(null);
const pcRef = useRef<RTCPeerConnection | null>(null);
const wsRef = useRef<WebSocket | null>(null);
const dragging = useRef(false);
const [status, setStatusRaw] = useState("connecting…");
const setStatus = (s: string) => {
setStatusRaw(s);
onStatus?.(s);
};
useEffect(() => {
let cancelled = false;
const pc = new RTCPeerConnection();
pcRef.current = pc;
pc.addTransceiver("video", { direction: "recvonly" });
pc.ontrack = (e) => {
if (videoRef.current) {
videoRef.current.srcObject = e.streams[0];
setStatus("streaming");
}
};
pc.oniceconnectionstatechange = () => setStatus(`ice: ${pc.iceConnectionState}`);
(async () => {
await pc.setLocalDescription(await pc.createOffer());
await iceComplete(pc);
if (cancelled) return;
const res = await fetch(OFFER_URL, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ sdp: pc.localDescription!.sdp }),
});
if (!res.ok) {
setStatus(`offer failed: ${await res.text()}`);
return;
}
const { sdp } = await res.json();
await pc.setRemoteDescription({ type: "answer", sdp });
if (cancelled) return;
wsRef.current = new WebSocket(inputWsUrl());
})().catch((e) => setStatus(`error: ${e}`));
return () => {
cancelled = true;
wsRef.current?.close();
pcRef.current?.close();
pcRef.current = null;
wsRef.current = null;
};
}, []);
function sendTouch(action: "down" | "move" | "up", e: React.PointerEvent<HTMLVideoElement>) {
const ws = wsRef.current;
const video = videoRef.current;
if (!ws || ws.readyState !== WebSocket.OPEN || !video) return;
const r = video.getBoundingClientRect();
const x = (e.clientX - r.left) / r.width;
const y = (e.clientY - r.top) / r.height;
if (x < 0 || x > 1 || y < 0 || y > 1) return;
ws.send(JSON.stringify({ kind: "touch", action, x, y }));
}
function sendKey(keycode: number) {
const ws = wsRef.current;
if (!ws || ws.readyState !== WebSocket.OPEN) return;
ws.send(JSON.stringify({ kind: "key", action: "down", keycode }));
ws.send(JSON.stringify({ kind: "key", action: "up", keycode }));
}
function capture() {
const video = videoRef.current;
if (!video || !video.videoWidth) return;
const canvas = document.createElement("canvas");
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
const ctx = canvas.getContext("2d");
if (!ctx) return;
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
onCapture(canvas.toDataURL("image/png"));
}
const keyBtn = "rounded-md border border-input bg-card/80 px-3 py-1.5 text-xs hover:bg-accent";
return (
<div className="relative flex h-full items-center justify-center gap-3 bg-black">
<video
ref={videoRef}
autoPlay
playsInline
muted
className="h-full w-auto max-w-full rounded-lg"
style={{ touchAction: "none" }}
onPointerDown={(e) => {
dragging.current = true;
e.currentTarget.setPointerCapture(e.pointerId);
sendTouch("down", e);
}}
onPointerMove={(e) => {
if (dragging.current) sendTouch("move", e);
}}
onPointerUp={(e) => {
dragging.current = false;
sendTouch("up", e);
}}
onPointerCancel={() => {
dragging.current = false;
}}
/>
<div className="absolute left-3 top-3 rounded bg-card/80 px-2 py-1 text-xs text-muted-foreground">
{label} · {status}
</div>
<div className="flex flex-col gap-2">
<button onClick={() => sendKey(3)} className={keyBtn}>HOME</button>
<button onClick={() => sendKey(4)} className={keyBtn}>BACK</button>
<button onClick={() => sendKey(187)} className={keyBtn}>RECENTS</button>
<button onClick={capture} className="mt-2 rounded-md bg-primary px-3 py-1.5 text-xs font-medium text-primary-foreground hover:bg-primary/90">
Capture frame
</button>
</div>
</div>
);
}
+83
View File
@@ -0,0 +1,83 @@
import { useEffect, useRef, useState } from "react";
import { api } from "../api";
type SaveState = "idle" | "saving" | "saved";
// Debounced-autosave notes surface (§4.5), controlled by the parent so voice
// dictation can flow into the same note. The first non-empty value creates a note
// finding; subsequent changes PATCH it. `interim` is the live (not-yet-committed)
// dictation transcript, shown as a preview while recording.
export function NotesPanel({
sessionId,
value,
onChange,
interim,
onSaved,
}: {
sessionId: string;
value: string;
onChange: (text: string) => void;
interim: string;
onSaved: () => void;
}) {
const [findingId, setFindingId] = useState<string | null>(null);
const [saveState, setSaveState] = useState<SaveState>("idle");
const timer = useRef<number | undefined>(undefined);
useEffect(() => {
if (value.trim() === "") return;
setSaveState("saving");
window.clearTimeout(timer.current);
timer.current = window.setTimeout(async () => {
try {
if (findingId) {
await api.updateNote(findingId, value);
} else {
const created = await api.createNote(sessionId, value);
setFindingId(created.id);
}
setSaveState("saved");
onSaved();
} catch {
setSaveState("idle");
}
}, 600);
return () => window.clearTimeout(timer.current);
}, [value, findingId, sessionId, onSaved]);
function newNote() {
onChange("");
setFindingId(null);
setSaveState("idle");
}
return (
<div className="flex h-full flex-col p-3">
<div className="mb-2 flex items-center justify-between">
<h2 className="text-sm font-semibold text-foreground">Notes</h2>
<div className="flex items-center gap-3 text-xs text-muted-foreground">
<span>
{saveState === "saving" ? "saving…" : saveState === "saved" ? "saved" : ""}
</span>
<button onClick={newNote} className="rounded border border-input px-2 py-0.5 hover:bg-accent">
New note
</button>
</div>
</div>
{interim && (
<div className="mb-2 rounded border border-success/30 bg-success/10 px-2 py-1.5 text-sm text-success">
<span className="mr-1 inline-block h-2 w-2 animate-pulse rounded-full bg-success align-middle" />
{interim}
</div>
)}
<textarea
value={value}
onChange={(e) => onChange(e.target.value)}
placeholder="Type a finding, or dictate with the recorder below… (autosaves)"
className="flex-1 resize-none rounded-md border border-border bg-background p-2.5 text-sm text-foreground outline-none focus:border-ring"
/>
</div>
);
}
+110
View File
@@ -0,0 +1,110 @@
import { useCallback, useEffect, useState } from "react";
import { Panel, PanelGroup, PanelResizeHandle } from "react-resizable-panels";
import { toast } from "sonner";
import { TopBar } from "./TopBar";
import { api, type Device, type Finding, type Session } from "../api";
import { LeftPane } from "./LeftPane";
import { NotesPanel } from "./NotesPanel";
import { VoicePanel } from "./VoicePanel";
import { FindingsList } from "./FindingsList";
// The split-screen review surface (§4.5 / §6.3): live device left, stacked notes +
// voice review surfaces right. Owns the *findings buffer* state concern.
export function ReviewShell({
session,
device,
devices,
onRelease,
onSwitch,
onOpenSettings,
onOpenPalette,
}: {
session: Session;
device: Device | null;
devices: Device[];
onRelease: () => void;
onSwitch: (serial: string) => void;
onOpenSettings: () => void;
onOpenPalette: () => void;
}) {
const [findings, setFindings] = useState<Finding[]>([]);
// Note draft + live dictation are owned here so voice dictation can flow into notes.
const [noteText, setNoteText] = useState("");
const [interim, setInterim] = useState("");
const [connection, setConnection] = useState("connecting…");
const refreshFindings = useCallback(() => {
api.listFindings(session.id).then(setFindings).catch(() => {});
}, [session.id]);
useEffect(() => {
refreshFindings();
}, [refreshFindings]);
// One-tap capture → a standalone snapshot finding of the current screen.
async function captureFrame(dataUrl: string) {
const blob = await (await fetch(dataUrl)).blob();
try {
await api.createFrameFinding(session.id, blob);
refreshFindings();
toast.success("Screenshot saved");
} catch {
toast.error("Failed to save screenshot");
}
}
// Final dictation gets appended to the note text; the interim preview clears.
function commitDictation(text: string) {
setNoteText((prev) => (prev.trim() ? `${prev.trimEnd()} ${text}` : text));
setInterim("");
}
return (
<div className="flex h-screen flex-col">
<TopBar
session={session}
device={device}
devices={devices}
connection={connection}
onSwitch={onSwitch}
onRelease={onRelease}
onOpenSettings={onOpenSettings}
onOpenPalette={onOpenPalette}
/>
<PanelGroup direction="horizontal" className="flex-1">
<Panel defaultSize={62} minSize={35}>
<LeftPane
label={device?.model ?? session.device_serial}
onCapture={captureFrame}
onStatus={setConnection}
/>
</Panel>
<PanelResizeHandle className="w-1 bg-border transition-colors hover:bg-primary" />
<Panel defaultSize={38} minSize={25}>
<PanelGroup direction="vertical">
<Panel defaultSize={45} minSize={20}>
<NotesPanel
sessionId={session.id}
value={noteText}
onChange={setNoteText}
interim={interim}
onSaved={refreshFindings}
/>
</Panel>
<PanelResizeHandle className="h-1 bg-border transition-colors hover:bg-primary" />
<Panel defaultSize={30} minSize={15}>
<VoicePanel onInterim={setInterim} onFinal={commitDictation} />
</Panel>
<PanelResizeHandle className="h-1 bg-border transition-colors hover:bg-primary" />
<Panel defaultSize={25} minSize={10}>
<FindingsList findings={findings} />
</Panel>
</PanelGroup>
</Panel>
</PanelGroup>
</div>
);
}
+105
View File
@@ -0,0 +1,105 @@
import {
Dialog,
DialogContent,
DialogDescription,
DialogHeader,
DialogTitle,
} from "@/components/ui/dialog";
import { Button } from "@/components/ui/button";
import { useSettings, type Accent, type Quality } from "@/lib/settings";
import { cn } from "@/lib/utils";
const LANGUAGES: { value: string; label: string }[] = [
{ value: "auto", label: "Auto-detect" },
{ value: "en", label: "English" },
{ value: "tr", label: "Turkish" },
{ value: "id", label: "Indonesian" },
{ value: "ar", label: "Arabic" },
{ value: "es", label: "Spanish" },
{ value: "fr", label: "French" },
{ value: "zh", label: "Chinese" },
];
const QUALITIES: Quality[] = ["high", "balanced", "low"];
const ACCENTS: Accent[] = ["emerald", "sky", "violet", "amber"];
const ACCENT_SWATCH: Record<Accent, string> = {
emerald: "#10b981",
sky: "#0ea5e9",
violet: "#8b5cf6",
amber: "#f59e0b",
};
function Row({ label, children }: { label: string; children: React.ReactNode }) {
return (
<div className="flex items-center justify-between gap-4 py-2">
<span className="text-sm text-muted-foreground">{label}</span>
<div className="flex items-center gap-1.5">{children}</div>
</div>
);
}
export function SettingsDialog({
open,
onOpenChange,
}: {
open: boolean;
onOpenChange: (open: boolean) => void;
}) {
const { settings, update } = useSettings();
return (
<Dialog open={open} onOpenChange={onOpenChange}>
<DialogContent>
<DialogHeader>
<DialogTitle>Settings</DialogTitle>
<DialogDescription>Preferences are saved in this browser.</DialogDescription>
</DialogHeader>
<div className="divide-y divide-border">
<Row label="Accent">
{ACCENTS.map((a) => (
<button
key={a}
onClick={() => update({ accent: a })}
aria-label={a}
className={cn(
"size-6 rounded-full border-2 transition-transform hover:scale-110",
settings.accent === a ? "border-foreground" : "border-transparent",
)}
style={{ backgroundColor: ACCENT_SWATCH[a] }}
/>
))}
</Row>
<Row label="Transcription language">
<select
value={settings.language}
onChange={(e) => update({ language: e.target.value })}
className="h-8 rounded-md border border-input bg-background px-2 text-sm text-foreground outline-none focus-visible:ring-2 focus-visible:ring-ring"
>
{LANGUAGES.map((l) => (
<option key={l.value} value={l.value}>
{l.label}
</option>
))}
</select>
</Row>
<Row label="Stream quality">
{QUALITIES.map((q) => (
<Button
key={q}
size="sm"
variant={settings.quality === q ? "default" : "outline"}
onClick={() => update({ quality: q })}
className="capitalize"
>
{q}
</Button>
))}
</Row>
</div>
</DialogContent>
</Dialog>
);
}
+100
View File
@@ -0,0 +1,100 @@
import { useEffect, useState } from "react";
import { ChevronDown, Settings, Command as CommandIcon, LogOut } from "lucide-react";
import { Button } from "@/components/ui/button";
import { Badge } from "@/components/ui/badge";
import {
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuLabel,
DropdownMenuSeparator,
DropdownMenuTrigger,
} from "@/components/ui/dropdown-menu";
import type { Device, Session } from "../api";
function elapsed(since: string): string {
const secs = Math.max(0, Math.floor((Date.now() - new Date(since).getTime()) / 1000));
const m = Math.floor(secs / 60);
const s = secs % 60;
return `${m}:${s.toString().padStart(2, "0")}`;
}
export function TopBar({
session,
device,
devices,
connection,
onSwitch,
onRelease,
onOpenSettings,
onOpenPalette,
}: {
session: Session;
device: Device | null;
devices: Device[];
connection: string;
onSwitch: (serial: string) => void;
onRelease: () => void;
onOpenSettings: () => void;
onOpenPalette: () => void;
}) {
const [, tick] = useState(0);
useEffect(() => {
const t = window.setInterval(() => tick((n) => n + 1), 1000);
return () => window.clearInterval(t);
}, []);
const live = connection === "streaming" || connection === "connected" || connection === "ice: connected";
return (
<header className="flex items-center justify-between border-b border-border px-3 py-2">
<div className="flex items-center gap-3">
<span className="font-semibold">ClawReview</span>
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button variant="outline" size="sm" className="gap-1.5">
{device?.model ?? session.device_serial}
<ChevronDown />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="start">
<DropdownMenuLabel>Switch device</DropdownMenuLabel>
{devices.map((d) => (
<DropdownMenuItem
key={d.serial}
disabled={d.serial === session.device_serial || d.status === "offline"}
onSelect={() => onSwitch(d.serial)}
>
<span className="flex-1">{d.model ?? d.serial}</span>
<Badge variant={d.status === "free" ? "success" : d.status === "claimed" ? "warning" : "muted"}>
{d.status}
</Badge>
</DropdownMenuItem>
))}
<DropdownMenuSeparator />
<DropdownMenuItem onSelect={onRelease}>
<LogOut /> Release device
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
<span className="text-xs text-muted-foreground">session {session.id.slice(0, 8)}</span>
<span className="font-mono text-xs text-muted-foreground tabular-nums">{elapsed(session.started_at)}</span>
<Badge variant={live ? "success" : "muted"} className="gap-1.5">
<span className={`size-1.5 rounded-full ${live ? "bg-success" : "bg-muted-foreground"}`} />
{connection}
</Badge>
</div>
<div className="flex items-center gap-2">
<Button variant="ghost" size="sm" onClick={onOpenPalette} className="gap-1.5 text-muted-foreground">
<CommandIcon /> <span className="hidden sm:inline">⌘K</span>
</Button>
<Button variant="ghost" size="icon" onClick={onOpenSettings} aria-label="Settings">
<Settings />
</Button>
</div>
</header>
);
}
+112
View File
@@ -0,0 +1,112 @@
import { useRef, useState } from "react";
import { api } from "../api";
type Phase = "idle" | "recording" | "finalizing";
// Voice dictation (§4.5): captures mic audio with MediaRecorder and transcribes it
// live on RedClaw infrastructure (the self-hosted Whisper service via /api/transcribe).
// The growing audio is re-transcribed every few seconds for an interim preview; on
// stop, the final transcript is committed into the notes above.
const INTERIM_MS = 3000;
export function VoicePanel({
onInterim,
onFinal,
}: {
onInterim: (text: string) => void;
onFinal: (text: string) => void;
}) {
const [phase, setPhase] = useState<Phase>("idle");
const [error, setError] = useState<string | null>(null);
const recorder = useRef<MediaRecorder | null>(null);
const chunks = useRef<Blob[]>([]);
const interval = useRef<number | undefined>(undefined);
async function transcribe(): Promise<string> {
const blob = new Blob(chunks.current, { type: "audio/webm" });
const { text } = await api.transcribeLive(blob);
return text.trim();
}
async function start() {
setError(null);
// getUserMedia only exists in a secure context (HTTPS or http://localhost). Over
// plain HTTP on a LAN/Tailscale IP the browser withholds navigator.mediaDevices.
if (!navigator.mediaDevices?.getUserMedia) {
setError(
"Microphone needs a secure context. Open the app over HTTPS (https://<host>.ts.net) or http://localhost.",
);
return;
}
try {
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const rec = new MediaRecorder(stream, { mimeType: "audio/webm" });
chunks.current = [];
rec.ondataavailable = (e) => {
if (e.data.size > 0) chunks.current.push(e.data);
};
rec.onstop = async () => {
window.clearInterval(interval.current);
stream.getTracks().forEach((t) => t.stop());
setPhase("finalizing");
try {
const text = await transcribe();
onInterim("");
if (text) onFinal(text);
} catch (e) {
setError(String(e));
onInterim("");
}
setPhase("idle");
};
rec.start(1000); // 1s timeslice so chunks accumulate for interim passes
recorder.current = rec;
setPhase("recording");
interval.current = window.setInterval(async () => {
if (chunks.current.length === 0) return;
try {
onInterim(await transcribe());
} catch {
/* a partial buffer may fail to decode; the next pass retries */
}
}, INTERIM_MS);
} catch (e) {
setError(`mic unavailable: ${e}`);
}
}
function stop() {
recorder.current?.stop();
}
return (
<div className="flex h-full flex-col p-3">
<h2 className="mb-2 text-sm font-semibold text-foreground">
Voice <span className="text-muted-foreground/60">— dictate into notes</span>
</h2>
<div className="flex items-center gap-3">
{phase === "idle" && (
<button onClick={start} className="rounded-md bg-destructive px-3 py-1.5 text-sm font-medium text-white hover:bg-destructive/90">
● Record
</button>
)}
{phase === "recording" && (
<>
<button onClick={stop} className="rounded-md bg-secondary px-3 py-1.5 text-sm font-medium text-secondary-foreground hover:bg-secondary/80">
■ Stop
</button>
<span className="flex items-center gap-1.5 text-sm text-destructive">
<span className="h-2 w-2 animate-pulse rounded-full bg-destructive" /> recording — words appear in Notes
</span>
</>
)}
{phase === "finalizing" && <span className="text-sm text-muted-foreground">finalizing transcript…</span>}
</div>
{error && <div className="mt-2 text-xs text-destructive">{error}</div>}
</div>
);
}
+31
View File
@@ -0,0 +1,31 @@
import * as React from "react";
import { cva, type VariantProps } from "class-variance-authority";
import { cn } from "@/lib/utils";
const badgeVariants = cva(
"inline-flex items-center rounded-full border px-2 py-0.5 text-xs font-medium transition-colors",
{
variants: {
variant: {
default: "border-transparent bg-primary/15 text-primary",
secondary: "border-transparent bg-secondary text-secondary-foreground",
outline: "border-border text-foreground",
success: "border-success/30 bg-success/10 text-success",
warning: "border-warning/30 bg-warning/10 text-warning",
destructive: "border-destructive/30 bg-destructive/10 text-destructive",
muted: "border-transparent bg-muted text-muted-foreground",
},
},
defaultVariants: { variant: "default" },
},
);
export interface BadgeProps
extends React.HTMLAttributes<HTMLSpanElement>,
VariantProps<typeof badgeVariants> {}
export function Badge({ className, variant, ...props }: BadgeProps) {
return <span className={cn(badgeVariants({ variant }), className)} {...props} />;
}
export { badgeVariants };
+43
View File
@@ -0,0 +1,43 @@
import * as React from "react";
import { Slot } from "@radix-ui/react-slot";
import { cva, type VariantProps } from "class-variance-authority";
import { cn } from "@/lib/utils";
const buttonVariants = cva(
"inline-flex items-center justify-center gap-2 whitespace-nowrap rounded-md text-sm font-medium transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring disabled:pointer-events-none disabled:opacity-50 [&_svg]:size-4 [&_svg]:shrink-0",
{
variants: {
variant: {
default: "bg-primary text-primary-foreground hover:bg-primary/90",
destructive: "bg-destructive text-destructive-foreground hover:bg-destructive/90",
outline: "border border-input bg-transparent hover:bg-accent hover:text-accent-foreground",
secondary: "bg-secondary text-secondary-foreground hover:bg-secondary/80",
ghost: "hover:bg-accent hover:text-accent-foreground",
link: "text-primary underline-offset-4 hover:underline",
},
size: {
default: "h-9 px-4 py-2",
sm: "h-8 rounded-md px-3 text-xs",
lg: "h-10 rounded-md px-6",
icon: "h-9 w-9",
},
},
defaultVariants: { variant: "default", size: "default" },
},
);
export interface ButtonProps
extends React.ButtonHTMLAttributes<HTMLButtonElement>,
VariantProps<typeof buttonVariants> {
asChild?: boolean;
}
export const Button = React.forwardRef<HTMLButtonElement, ButtonProps>(
({ className, variant, size, asChild = false, ...props }, ref) => {
const Comp = asChild ? Slot : "button";
return <Comp ref={ref} className={cn(buttonVariants({ variant, size, className }))} {...props} />;
},
);
Button.displayName = "Button";
export { buttonVariants };
+13
View File
@@ -0,0 +1,13 @@
import * as React from "react";
import { cn } from "@/lib/utils";
export const Card = React.forwardRef<HTMLDivElement, React.HTMLAttributes<HTMLDivElement>>(
({ className, ...props }, ref) => (
<div
ref={ref}
className={cn("rounded-lg border border-border bg-card text-card-foreground shadow-sm", className)}
{...props}
/>
),
);
Card.displayName = "Card";
+100
View File
@@ -0,0 +1,100 @@
import * as React from "react";
import { Command as CommandPrimitive } from "cmdk";
import { Search } from "lucide-react";
import { Dialog, DialogContent } from "@/components/ui/dialog";
import { cn } from "@/lib/utils";
export const Command = React.forwardRef<
React.ElementRef<typeof CommandPrimitive>,
React.ComponentPropsWithoutRef<typeof CommandPrimitive>
>(({ className, ...props }, ref) => (
<CommandPrimitive
ref={ref}
className={cn(
"flex h-full w-full flex-col overflow-hidden rounded-md bg-popover text-popover-foreground",
className,
)}
{...props}
/>
));
Command.displayName = CommandPrimitive.displayName;
export function CommandDialog({
children,
...props
}: React.ComponentProps<typeof Dialog>) {
return (
<Dialog {...props}>
<DialogContent hideClose className="overflow-hidden p-0">
<Command className="[&_[cmdk-group-heading]]:px-2 [&_[cmdk-group-heading]]:py-1.5 [&_[cmdk-group-heading]]:text-xs [&_[cmdk-group-heading]]:font-medium [&_[cmdk-group-heading]]:text-muted-foreground [&_[cmdk-item]]:px-2 [&_[cmdk-item]]:py-2">
{children}
</Command>
</DialogContent>
</Dialog>
);
}
export const CommandInput = React.forwardRef<
React.ElementRef<typeof CommandPrimitive.Input>,
React.ComponentPropsWithoutRef<typeof CommandPrimitive.Input>
>(({ className, ...props }, ref) => (
<div className="flex items-center border-b border-border px-3">
<Search className="mr-2 size-4 shrink-0 text-muted-foreground" />
<CommandPrimitive.Input
ref={ref}
className={cn(
"flex h-11 w-full bg-transparent py-3 text-sm text-foreground outline-none placeholder:text-muted-foreground disabled:cursor-not-allowed disabled:opacity-50",
className,
)}
{...props}
/>
</div>
));
CommandInput.displayName = CommandPrimitive.Input.displayName;
export const CommandList = React.forwardRef<
React.ElementRef<typeof CommandPrimitive.List>,
React.ComponentPropsWithoutRef<typeof CommandPrimitive.List>
>(({ className, ...props }, ref) => (
<CommandPrimitive.List
ref={ref}
className={cn("max-h-80 overflow-y-auto overflow-x-hidden p-1", className)}
{...props}
/>
));
CommandList.displayName = CommandPrimitive.List.displayName;
export const CommandEmpty = React.forwardRef<
React.ElementRef<typeof CommandPrimitive.Empty>,
React.ComponentPropsWithoutRef<typeof CommandPrimitive.Empty>
>((props, ref) => (
<CommandPrimitive.Empty
ref={ref}
className="py-6 text-center text-sm text-muted-foreground"
{...props}
/>
));
CommandEmpty.displayName = CommandPrimitive.Empty.displayName;
export const CommandGroup = React.forwardRef<
React.ElementRef<typeof CommandPrimitive.Group>,
React.ComponentPropsWithoutRef<typeof CommandPrimitive.Group>
>(({ className, ...props }, ref) => (
<CommandPrimitive.Group ref={ref} className={cn("overflow-hidden p-1", className)} {...props} />
));
CommandGroup.displayName = CommandPrimitive.Group.displayName;
export const CommandItem = React.forwardRef<
React.ElementRef<typeof CommandPrimitive.Item>,
React.ComponentPropsWithoutRef<typeof CommandPrimitive.Item>
>(({ className, ...props }, ref) => (
<CommandPrimitive.Item
ref={ref}
className={cn(
"relative flex cursor-pointer select-none items-center gap-2 rounded-sm px-2 py-2 text-sm text-foreground outline-none data-[selected=true]:bg-accent data-[selected=true]:text-accent-foreground data-[disabled=true]:pointer-events-none data-[disabled=true]:opacity-50 [&_svg]:size-4 [&_svg]:text-muted-foreground",
className,
)}
{...props}
/>
));
CommandItem.displayName = CommandPrimitive.Item.displayName;
+77
View File
@@ -0,0 +1,77 @@
import * as React from "react";
import * as DialogPrimitive from "@radix-ui/react-dialog";
import { X } from "lucide-react";
import { cn } from "@/lib/utils";
export const Dialog = DialogPrimitive.Root;
export const DialogTrigger = DialogPrimitive.Trigger;
export const DialogClose = DialogPrimitive.Close;
const DialogOverlay = React.forwardRef<
React.ElementRef<typeof DialogPrimitive.Overlay>,
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Overlay>
>(({ className, ...props }, ref) => (
<DialogPrimitive.Overlay
ref={ref}
className={cn(
"fixed inset-0 z-50 bg-black/70 backdrop-blur-sm animate-in fade-in-0",
className,
)}
{...props}
/>
));
DialogOverlay.displayName = DialogPrimitive.Overlay.displayName;
export const DialogContent = React.forwardRef<
React.ElementRef<typeof DialogPrimitive.Content>,
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Content> & { hideClose?: boolean }
>(({ className, children, hideClose, ...props }, ref) => (
<DialogPrimitive.Portal>
<DialogOverlay />
<DialogPrimitive.Content
ref={ref}
className={cn(
"fixed left-1/2 top-1/2 z-50 grid w-full max-w-lg -translate-x-1/2 -translate-y-1/2 gap-4 rounded-lg border border-border bg-card p-6 shadow-lg animate-in fade-in-0 zoom-in-95",
className,
)}
{...props}
>
{children}
{!hideClose && (
<DialogPrimitive.Close className="absolute right-4 top-4 rounded-sm text-muted-foreground opacity-70 transition-opacity hover:opacity-100 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring">
<X className="size-4" />
<span className="sr-only">Close</span>
</DialogPrimitive.Close>
)}
</DialogPrimitive.Content>
</DialogPrimitive.Portal>
));
DialogContent.displayName = DialogPrimitive.Content.displayName;
export function DialogHeader({ className, ...props }: React.HTMLAttributes<HTMLDivElement>) {
return <div className={cn("flex flex-col gap-1.5", className)} {...props} />;
}
export const DialogTitle = React.forwardRef<
React.ElementRef<typeof DialogPrimitive.Title>,
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Title>
>(({ className, ...props }, ref) => (
<DialogPrimitive.Title
ref={ref}
className={cn("text-lg font-semibold text-foreground", className)}
{...props}
/>
));
DialogTitle.displayName = DialogPrimitive.Title.displayName;
export const DialogDescription = React.forwardRef<
React.ElementRef<typeof DialogPrimitive.Description>,
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Description>
>(({ className, ...props }, ref) => (
<DialogPrimitive.Description
ref={ref}
className={cn("text-sm text-muted-foreground", className)}
{...props}
/>
));
DialogDescription.displayName = DialogPrimitive.Description.displayName;
@@ -0,0 +1,85 @@
import * as React from "react";
import * as DropdownMenuPrimitive from "@radix-ui/react-dropdown-menu";
import { Check } from "lucide-react";
import { cn } from "@/lib/utils";
export const DropdownMenu = DropdownMenuPrimitive.Root;
export const DropdownMenuTrigger = DropdownMenuPrimitive.Trigger;
export const DropdownMenuContent = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.Content>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Content>
>(({ className, sideOffset = 4, ...props }, ref) => (
<DropdownMenuPrimitive.Portal>
<DropdownMenuPrimitive.Content
ref={ref}
sideOffset={sideOffset}
className={cn(
"z-50 min-w-[12rem] overflow-hidden rounded-md border border-border bg-popover p-1 text-popover-foreground shadow-md animate-in fade-in-0 zoom-in-95",
className,
)}
{...props}
/>
</DropdownMenuPrimitive.Portal>
));
DropdownMenuContent.displayName = DropdownMenuPrimitive.Content.displayName;
export const DropdownMenuItem = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.Item>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Item> & { inset?: boolean }
>(({ className, inset, ...props }, ref) => (
<DropdownMenuPrimitive.Item
ref={ref}
className={cn(
"relative flex cursor-pointer select-none items-center gap-2 rounded-sm px-2 py-1.5 text-sm outline-none transition-colors focus:bg-accent focus:text-accent-foreground data-[disabled]:pointer-events-none data-[disabled]:opacity-50 [&_svg]:size-4 [&_svg]:text-muted-foreground",
inset && "pl-8",
className,
)}
{...props}
/>
));
DropdownMenuItem.displayName = DropdownMenuPrimitive.Item.displayName;
export const DropdownMenuCheckboxItem = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.CheckboxItem>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.CheckboxItem>
>(({ className, children, checked, ...props }, ref) => (
<DropdownMenuPrimitive.CheckboxItem
ref={ref}
checked={checked}
className={cn(
"relative flex cursor-pointer select-none items-center rounded-sm py-1.5 pl-8 pr-2 text-sm outline-none transition-colors focus:bg-accent focus:text-accent-foreground",
className,
)}
{...props}
>
<span className="absolute left-2 flex size-4 items-center justify-center">
<DropdownMenuPrimitive.ItemIndicator>
<Check className="size-4" />
</DropdownMenuPrimitive.ItemIndicator>
</span>
{children}
</DropdownMenuPrimitive.CheckboxItem>
));
DropdownMenuCheckboxItem.displayName = DropdownMenuPrimitive.CheckboxItem.displayName;
export function DropdownMenuLabel({
className,
...props
}: React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Label>) {
return (
<DropdownMenuPrimitive.Label
className={cn("px-2 py-1.5 text-xs font-medium text-muted-foreground", className)}
{...props}
/>
);
}
export function DropdownMenuSeparator({
className,
...props
}: React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Separator>) {
return (
<DropdownMenuPrimitive.Separator className={cn("-mx-1 my-1 h-px bg-border", className)} {...props} />
);
}
+17
View File
@@ -0,0 +1,17 @@
import * as React from "react";
import { cn } from "@/lib/utils";
export const Input = React.forwardRef<HTMLInputElement, React.InputHTMLAttributes<HTMLInputElement>>(
({ className, type, ...props }, ref) => (
<input
type={type}
ref={ref}
className={cn(
"flex h-9 w-full rounded-md border border-input bg-transparent px-3 py-1 text-sm text-foreground shadow-sm transition-colors placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring disabled:cursor-not-allowed disabled:opacity-50",
className,
)}
{...props}
/>
),
);
Input.displayName = "Input";
+21
View File
@@ -0,0 +1,21 @@
import * as React from "react";
import * as SeparatorPrimitive from "@radix-ui/react-separator";
import { cn } from "@/lib/utils";
export const Separator = React.forwardRef<
React.ElementRef<typeof SeparatorPrimitive.Root>,
React.ComponentPropsWithoutRef<typeof SeparatorPrimitive.Root>
>(({ className, orientation = "horizontal", decorative = true, ...props }, ref) => (
<SeparatorPrimitive.Root
ref={ref}
decorative={decorative}
orientation={orientation}
className={cn(
"shrink-0 bg-border",
orientation === "horizontal" ? "h-px w-full" : "h-full w-px",
className,
)}
{...props}
/>
));
Separator.displayName = SeparatorPrimitive.Root.displayName;
+19
View File
@@ -0,0 +1,19 @@
import { Toaster as Sonner, type ToasterProps } from "sonner";
// App-wide toast host (Workstream 2). Themed to our tokens.
export function Toaster(props: ToasterProps) {
return (
<Sonner
theme="dark"
position="bottom-right"
toastOptions={{
style: {
background: "var(--popover)",
color: "var(--popover-foreground)",
border: "1px solid var(--border)",
},
}}
{...props}
/>
);
}
+17
View File
@@ -0,0 +1,17 @@
import * as React from "react";
import { cn } from "@/lib/utils";
export const Textarea = React.forwardRef<
HTMLTextAreaElement,
React.TextareaHTMLAttributes<HTMLTextAreaElement>
>(({ className, ...props }, ref) => (
<textarea
ref={ref}
className={cn(
"flex w-full rounded-md border border-input bg-transparent px-3 py-2 text-sm text-foreground shadow-sm transition-colors placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring disabled:cursor-not-allowed disabled:opacity-50",
className,
)}
{...props}
/>
));
Textarea.displayName = "Textarea";
+25
View File
@@ -0,0 +1,25 @@
import * as React from "react";
import * as TooltipPrimitive from "@radix-ui/react-tooltip";
import { cn } from "@/lib/utils";
export const TooltipProvider = TooltipPrimitive.Provider;
export const Tooltip = TooltipPrimitive.Root;
export const TooltipTrigger = TooltipPrimitive.Trigger;
export const TooltipContent = React.forwardRef<
React.ElementRef<typeof TooltipPrimitive.Content>,
React.ComponentPropsWithoutRef<typeof TooltipPrimitive.Content>
>(({ className, sideOffset = 4, ...props }, ref) => (
<TooltipPrimitive.Portal>
<TooltipPrimitive.Content
ref={ref}
sideOffset={sideOffset}
className={cn(
"z-50 overflow-hidden rounded-md border border-border bg-popover px-2.5 py-1.5 text-xs text-popover-foreground shadow-md animate-in fade-in-0 zoom-in-95",
className,
)}
{...props}
/>
</TooltipPrimitive.Portal>
));
TooltipContent.displayName = TooltipPrimitive.Content.displayName;
+115
View File
@@ -0,0 +1,115 @@
@import "tailwindcss";
/* Dark theme is applied via a `.dark` class on <html>. */
@custom-variant dark (&:is(.dark *));
/*
* Design tokens (Workstream 1). Semantic CSS variables, not hardcoded colors, so
* the whole UI is themeable. Dark values avoid pure black/white to prevent halation
* (WCAG dark-mode guidance); text/controls clear 4.5:1 / 3:1.
*/
:root {
--radius: 0.625rem;
/* Light (kept minimal for now — the app ships dark-first). */
--background: #fafafa;
--foreground: #18181b;
--card: #ffffff;
--card-foreground: #18181b;
--popover: #ffffff;
--popover-foreground: #18181b;
--primary: #059669;
--primary-foreground: #fafafa;
--secondary: #f4f4f5;
--secondary-foreground: #18181b;
--muted: #f4f4f5;
--muted-foreground: #52525b;
--accent: #f4f4f5;
--accent-foreground: #18181b;
--destructive: #dc2626;
--destructive-foreground: #fafafa;
--warning: #d97706;
--warning-foreground: #fafafa;
--success: #059669;
--border: #e4e4e7;
--input: #e4e4e7;
--ring: #10b981;
}
.dark {
--background: #09090b; /* zinc-950, not #000 */
--foreground: #e4e4e7; /* zinc-200, not #fff */
--card: #18181b;
--card-foreground: #e4e4e7;
--popover: #18181b;
--popover-foreground: #e4e4e7;
--primary: #10b981; /* emerald-500 */
--primary-foreground: #052e1f;
--secondary: #27272a;
--secondary-foreground: #e4e4e7;
--muted: #27272a;
--muted-foreground: #a1a1aa; /* zinc-400 */
--accent: #27272a;
--accent-foreground: #e4e4e7;
--destructive: #ef4444; /* red-500 */
--destructive-foreground: #fef2f2;
--warning: #f59e0b; /* amber-500 */
--warning-foreground: #1c1917;
--success: #34d399; /* emerald-400 */
--border: #27272a; /* zinc-800 */
--input: #3f3f46; /* zinc-700 */
--ring: #10b981;
}
/* Map tokens onto Tailwind color utilities (bg-background, text-muted-foreground, …). */
@theme inline {
--color-background: var(--background);
--color-foreground: var(--foreground);
--color-card: var(--card);
--color-card-foreground: var(--card-foreground);
--color-popover: var(--popover);
--color-popover-foreground: var(--popover-foreground);
--color-primary: var(--primary);
--color-primary-foreground: var(--primary-foreground);
--color-secondary: var(--secondary);
--color-secondary-foreground: var(--secondary-foreground);
--color-muted: var(--muted);
--color-muted-foreground: var(--muted-foreground);
--color-accent: var(--accent);
--color-accent-foreground: var(--accent-foreground);
--color-destructive: var(--destructive);
--color-destructive-foreground: var(--destructive-foreground);
--color-warning: var(--warning);
--color-warning-foreground: var(--warning-foreground);
--color-success: var(--success);
--color-border: var(--border);
--color-input: var(--input);
--color-ring: var(--ring);
--radius-sm: calc(var(--radius) - 4px);
--radius-md: calc(var(--radius) - 2px);
--radius-lg: var(--radius);
--radius-xl: calc(var(--radius) + 4px);
}
@layer base {
* {
border-color: var(--border);
}
html,
body,
#root {
height: 100%;
}
body {
margin: 0;
background-color: var(--background);
color: var(--foreground);
font-family: ui-sans-serif, system-ui, -apple-system, "Segoe UI", sans-serif;
}
/* Visible focus ring for keyboard users (accessibility). */
:focus-visible {
outline: 2px solid var(--ring);
outline-offset: 2px;
}
}
+66
View File
@@ -0,0 +1,66 @@
import { createContext, useContext, useEffect, useState, type ReactNode } from "react";
export type Quality = "high" | "balanced" | "low";
export type Accent = "emerald" | "sky" | "violet" | "amber";
export interface Settings {
/** Transcription language ("auto" lets the model detect). Feeds W4. */
language: string;
/** Stream quality preset. Feeds W4. */
quality: Quality;
/** Primary accent color. */
accent: Accent;
}
const DEFAULTS: Settings = { language: "auto", quality: "balanced", accent: "emerald" };
const ACCENTS: Record<Accent, { primary: string; ring: string; fg: string }> = {
emerald: { primary: "#10b981", ring: "#10b981", fg: "#052e1f" },
sky: { primary: "#0ea5e9", ring: "#0ea5e9", fg: "#082f49" },
violet: { primary: "#8b5cf6", ring: "#8b5cf6", fg: "#1e1b4b" },
amber: { primary: "#f59e0b", ring: "#f59e0b", fg: "#1c1917" },
};
const STORAGE_KEY = "clawreview.settings";
interface SettingsContextValue {
settings: Settings;
update: (patch: Partial<Settings>) => void;
}
const SettingsContext = createContext<SettingsContextValue | null>(null);
function load(): Settings {
try {
return { ...DEFAULTS, ...JSON.parse(localStorage.getItem(STORAGE_KEY) ?? "{}") };
} catch {
return DEFAULTS;
}
}
function applyAccent(accent: Accent) {
const a = ACCENTS[accent];
const root = document.documentElement;
root.style.setProperty("--primary", a.primary);
root.style.setProperty("--ring", a.ring);
root.style.setProperty("--primary-foreground", a.fg);
}
export function SettingsProvider({ children }: { children: ReactNode }) {
const [settings, setSettings] = useState<Settings>(load);
useEffect(() => {
localStorage.setItem(STORAGE_KEY, JSON.stringify(settings));
applyAccent(settings.accent);
}, [settings]);
const update = (patch: Partial<Settings>) => setSettings((s) => ({ ...s, ...patch }));
return <SettingsContext.Provider value={{ settings, update }}>{children}</SettingsContext.Provider>;
}
export function useSettings(): SettingsContextValue {
const ctx = useContext(SettingsContext);
if (!ctx) throw new Error("useSettings must be used within SettingsProvider");
return ctx;
}
+7
View File
@@ -0,0 +1,7 @@
import { clsx, type ClassValue } from "clsx";
import { twMerge } from "tailwind-merge";
/** Merge conditional + conflicting Tailwind classes (shadcn convention). */
export function cn(...inputs: ClassValue[]) {
return twMerge(clsx(inputs));
}
+18
View File
@@ -0,0 +1,18 @@
import React from "react";
import ReactDOM from "react-dom/client";
import { TooltipProvider } from "@/components/ui/tooltip";
import { Toaster } from "@/components/ui/sonner";
import { SettingsProvider } from "@/lib/settings";
import App from "./App";
import "./index.css";
ReactDOM.createRoot(document.getElementById("root")!).render(
<React.StrictMode>
<SettingsProvider>
<TooltipProvider delayDuration={300}>
<App />
<Toaster />
</TooltipProvider>
</SettingsProvider>
</React.StrictMode>,
);
+22
View File
@@ -0,0 +1,22 @@
{
"compilerOptions": {
"target": "ES2020",
"useDefineForClassFields": true,
"lib": ["ES2020", "DOM", "DOM.Iterable"],
"module": "ESNext",
"skipLibCheck": true,
"moduleResolution": "bundler",
"allowImportingTsExtensions": true,
"resolveJsonModule": true,
"isolatedModules": true,
"moduleDetection": "force",
"noEmit": true,
"jsx": "react-jsx",
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"baseUrl": ".",
"paths": { "@/*": ["./src/*"] }
},
"include": ["src", "vite.config.ts"]
}
+43
View File
@@ -0,0 +1,43 @@
import { defineConfig } from "vite";
import { fileURLToPath } from "node:url";
import { existsSync, readFileSync } from "node:fs";
import react from "@vitejs/plugin-react";
import tailwindcss from "@tailwindcss/vite";
// Serve over HTTPS when a Tailscale cert is present (mint with
// `tailscale cert --cert-file certs/tailnet.crt --key-file certs/tailnet.key <host>.ts.net`).
// A secure context is required for the microphone (getUserMedia) — plain HTTP on a
// LAN/Tailscale IP withholds navigator.mediaDevices. Access via the cert's ts.net host.
const keyPath = fileURLToPath(new URL("./certs/tailnet.key", import.meta.url));
const crtPath = fileURLToPath(new URL("./certs/tailnet.crt", import.meta.url));
const https =
existsSync(keyPath) && existsSync(crtPath)
? { key: readFileSync(keyPath), cert: readFileSync(crtPath) }
: undefined;
// /api is proxied to the review-api (§6.2) so the dev server and API share an origin.
export default defineConfig({
plugins: [react(), tailwindcss()],
resolve: {
alias: { "@": fileURLToPath(new URL("./src", import.meta.url)) },
},
server: {
port: 5173,
// Listen on all interfaces so the dev server is reachable over the Tailscale IP
// (mirrors the LAN/relay access model — on-site engineers reach it directly).
host: true,
https,
// Allow the tailnet hostnames (Vite blocks unknown Host headers by default).
allowedHosts: [".ts.net", "localhost"],
proxy: {
"/api": "http://localhost:8090",
// streamd (stream node): SDP signaling + input WebSocket. Rewrite strips the
// /streamd prefix; ws:true so the input socket is proxied too.
"/streamd": {
target: "http://localhost:8095",
ws: true,
rewrite: (p) => p.replace(/^\/streamd/, ""),
},
},
},
});
+19
View File
@@ -0,0 +1,19 @@
[package]
name = "review-api"
version = "0.1.0"
edition = "2021"
[dependencies]
axum = { version = "0.7", features = ["multipart"] }
tokio = { version = "1", features = ["full"] }
tokio-stream = { version = "0.1", features = ["sync"] }
tower-http = { version = "0.6", features = ["cors", "trace"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
sqlx = { version = "0.8", default-features = false, features = ["runtime-tokio", "tls-rustls", "postgres", "uuid", "chrono", "macros", "migrate"] }
uuid = { version = "1", features = ["v4", "serde"] }
chrono = { version = "0.4", features = ["serde"] }
anyhow = "1"
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json", "multipart"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
+59
View File
@@ -0,0 +1,59 @@
-- ClawReview core data model (§7 of SPEC.md).
-- The anchoring chain finding -> session -> device -> build is what guarantees
-- every finding is unambiguously attributable. device_serial / build_hash /
-- reviewer_id are denormalized onto findings so attribution is immutable even if
-- a device is later re-flashed or re-registered.
CREATE TABLE users (
id UUID PRIMARY KEY,
email TEXT NOT NULL UNIQUE,
display_name TEXT NOT NULL,
role TEXT NOT NULL CHECK (role IN ('engineer','partner','operator','admin')),
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE builds (
id UUID PRIMARY KEY,
app_identity TEXT NOT NULL,
version TEXT NOT NULL,
hash TEXT NOT NULL,
install_metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE devices (
serial TEXT PRIMARY KEY,
model TEXT,
status TEXT NOT NULL DEFAULT 'offline' CHECK (status IN ('free','claimed','offline')),
stream_node TEXT,
current_build_id UUID REFERENCES builds(id),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE sessions (
id UUID PRIMARY KEY,
reviewer_id UUID NOT NULL REFERENCES users(id),
device_serial TEXT NOT NULL REFERENCES devices(serial),
build_id UUID REFERENCES builds(id),
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
ended_at TIMESTAMPTZ
);
CREATE TABLE findings (
id UUID PRIMARY KEY,
session_id UUID NOT NULL REFERENCES sessions(id),
device_serial TEXT NOT NULL,
build_hash TEXT,
reviewer_id UUID NOT NULL REFERENCES users(id),
kind TEXT NOT NULL DEFAULT 'note' CHECK (kind IN ('note','voice')),
note_text TEXT,
audio_ref TEXT,
transcript TEXT,
frame_ref TEXT,
clip_ref TEXT,
time_offset_ms BIGINT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_findings_session ON findings(session_id);
CREATE INDEX idx_sessions_device ON sessions(device_serial);
@@ -0,0 +1,3 @@
-- Allow a dedicated 'frame' finding kind for one-tap screen snapshots (§4.5).
ALTER TABLE findings DROP CONSTRAINT findings_kind_check;
ALTER TABLE findings ADD CONSTRAINT findings_kind_check CHECK (kind IN ('note', 'voice', 'frame'));
@@ -0,0 +1,4 @@
-- Session heartbeat (§4.4): a claim is kept alive by periodic heartbeats. A claim
-- whose heartbeat has gone stale can be reclaimed, so a closed/refreshed browser
-- never strands a device.
ALTER TABLE sessions ADD COLUMN heartbeat_at TIMESTAMPTZ NOT NULL DEFAULT now();
@@ -0,0 +1,5 @@
-- Activity tracking for dormancy-based auto-reclaim (§4.4, sharing scenario).
-- heartbeat_at = connection liveness (browser still open); last_activity_at = the
-- last real interaction (tap/key/scroll/note/voice). A session is reclaimable when
-- it disconnects (stale heartbeat) OR goes dormant (no activity for the idle window).
ALTER TABLE sessions ADD COLUMN last_activity_at TIMESTAMPTZ NOT NULL DEFAULT now();
+42
View File
@@ -0,0 +1,42 @@
use axum::{
http::StatusCode,
response::{IntoResponse, Response},
Json,
};
use serde_json::json;
/// Application error mapped to an HTTP response. Any error that converts into
/// `anyhow::Error` (sqlx, etc.) becomes a 500 via the blanket `From` below.
pub enum AppError {
NotFound(String),
Conflict(String),
BadRequest(String),
Internal(anyhow::Error),
}
impl IntoResponse for AppError {
fn into_response(self) -> Response {
let (status, message) = match self {
AppError::NotFound(m) => (StatusCode::NOT_FOUND, m),
AppError::Conflict(m) => (StatusCode::CONFLICT, m),
AppError::BadRequest(m) => (StatusCode::BAD_REQUEST, m),
AppError::Internal(e) => {
tracing::error!(error = ?e, "internal error");
(
StatusCode::INTERNAL_SERVER_ERROR,
"internal server error".to_string(),
)
}
};
(status, Json(json!({ "error": message }))).into_response()
}
}
impl<E> From<E> for AppError
where
E: Into<anyhow::Error>,
{
fn from(err: E) -> Self {
AppError::Internal(err.into())
}
}
+42
View File
@@ -0,0 +1,42 @@
//! Server-sent events (Workstream 2): a broadcast channel of registry/session events
//! the browser subscribes to for live fleet status and push notifications, replacing
//! the previous poll-only model.
use std::convert::Infallible;
use axum::extract::State;
use axum::response::sse::{Event, KeepAlive, Sse};
use serde::Serialize;
use tokio::sync::broadcast;
use tokio_stream::wrappers::BroadcastStream;
use tokio_stream::{Stream, StreamExt};
use uuid::Uuid;
use crate::AppState;
#[derive(Clone, Debug, Serialize)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum AppEvent {
/// A device changed status (free/claimed/offline) — drives the switcher + toasts.
DeviceStatus { serial: String, status: String },
/// An active session ended (released or reaped by a takeover) — the holder's
/// browser can surface a toast and return to the device list.
SessionEnded { session_id: Uuid },
}
pub fn channel() -> broadcast::Sender<AppEvent> {
broadcast::channel(128).0
}
/// Publish an event to all subscribers; a no-op when nobody is listening.
pub fn emit(state: &AppState, event: AppEvent) {
let _ = state.events.send(event);
}
pub async fn events_handler(
State(state): State<AppState>,
) -> Sse<impl Stream<Item = Result<Event, Infallible>>> {
let stream = BroadcastStream::new(state.events.subscribe())
.filter_map(|res| res.ok().map(|ev| Ok(Event::default().json_data(ev).unwrap())));
Sse::new(stream).keep_alive(KeepAlive::default())
}
+491
View File
@@ -0,0 +1,491 @@
use axum::{
extract::{Multipart, Path, State},
http::StatusCode,
response::IntoResponse,
Json,
};
use chrono::Utc;
use serde_json::{json, Value};
use sqlx::PgPool;
use uuid::Uuid;
use crate::error::AppError;
use crate::events::{emit, AppEvent};
use crate::models::*;
use crate::AppState;
// Fixed seed identifiers. Auth (real users) lands in Phase 1; for the Track-1
// placeholder surface every session is authored by this dev reviewer.
const DEV_USER_ID: Uuid = Uuid::from_u128(0x0000_0000_0000_0000_0000_0000_0000_0001);
const PLACEHOLDER_BUILD_ID: Uuid = Uuid::from_u128(0x0000_0000_0000_0000_0000_0000_0000_0002);
pub async fn health() -> Json<Value> {
Json(json!({ "status": "ok" }))
}
pub async fn list_devices(State(st): State<AppState>) -> Result<Json<Vec<Device>>, AppError> {
let devices = sqlx::query_as::<_, Device>("SELECT * FROM devices ORDER BY serial")
.fetch_all(&st.db)
.await?;
Ok(Json(devices))
}
pub async fn create_session(
State(st): State<AppState>,
Json(body): Json<CreateSession>,
) -> Result<Json<Session>, AppError> {
let reviewer_id = DEV_USER_ID; // real reviewer identity arrives with auth (Phase 1).
let mut tx = st.db.begin().await?;
// Lock the device row so concurrent claims on the same device serialize.
let build_id: Option<Uuid> =
sqlx::query_scalar("SELECT current_build_id FROM devices WHERE serial=$1 FOR UPDATE")
.bind(&body.device_serial)
.fetch_optional(&mut *tx)
.await?
.ok_or_else(|| AppError::NotFound(format!("device {} not found", body.device_serial)))?;
// Block only if someone *else* holds a claim that is both connected (fresh
// heartbeat) and active (recent interaction). Our own claims (a refresh),
// disconnected claims, and dormant claims are all reclaimable.
let blocked: Option<i32> = sqlx::query_scalar(
"SELECT 1 FROM sessions \
WHERE device_serial=$1 AND ended_at IS NULL AND reviewer_id <> $2 \
AND heartbeat_at > now() - make_interval(secs => $3) \
AND last_activity_at > now() - make_interval(secs => $4) LIMIT 1",
)
.bind(&body.device_serial)
.bind(reviewer_id)
.bind(st.disconnect_timeout_secs as f64)
.bind(st.idle_timeout_secs as f64)
.fetch_optional(&mut *tx)
.await?;
if blocked.is_some() {
return Err(AppError::Conflict(format!(
"device {} is in use by another reviewer",
body.device_serial
)));
}
// Reap every reclaimable active session for this device, then take it.
let reaped: Vec<Uuid> = sqlx::query_scalar(
"UPDATE sessions SET ended_at=now() WHERE device_serial=$1 AND ended_at IS NULL RETURNING id",
)
.bind(&body.device_serial)
.fetch_all(&mut *tx)
.await?;
sqlx::query("UPDATE devices SET status='claimed', updated_at=now() WHERE serial=$1")
.bind(&body.device_serial)
.execute(&mut *tx)
.await?;
let session = sqlx::query_as::<_, Session>(
"INSERT INTO sessions (id, reviewer_id, device_serial, build_id, started_at, heartbeat_at) \
VALUES ($1, $2, $3, $4, now(), now()) RETURNING *",
)
.bind(Uuid::new_v4())
.bind(reviewer_id)
.bind(&body.device_serial)
.bind(build_id)
.fetch_one(&mut *tx)
.await?;
tx.commit().await?;
for session_id in reaped {
emit(&st, AppEvent::SessionEnded { session_id });
}
emit(
&st,
AppEvent::DeviceStatus {
serial: body.device_serial.clone(),
status: "claimed".into(),
},
);
Ok(Json(session))
}
#[derive(Debug, serde::Deserialize)]
pub struct HeartbeatParams {
/// True when the reviewer interacted since the last heartbeat. Bumps the
/// dormancy clock; a connected-but-idle session still ages toward reclaim.
#[serde(default)]
pub active: bool,
}
/// Keep a claim alive. The browser pings this periodically (connection liveness);
/// `?active=true` also marks recent interaction so a dormant session can be reaped
/// even while its tab stays open (§4.4).
pub async fn heartbeat(
State(st): State<AppState>,
Path(id): Path<Uuid>,
axum::extract::Query(params): axum::extract::Query<HeartbeatParams>,
) -> Result<StatusCode, AppError> {
let sql = if params.active {
"UPDATE sessions SET heartbeat_at=now(), last_activity_at=now() WHERE id=$1 AND ended_at IS NULL"
} else {
"UPDATE sessions SET heartbeat_at=now() WHERE id=$1 AND ended_at IS NULL"
};
let updated = sqlx::query(sql).bind(id).execute(&st.db).await?;
if updated.rows_affected() == 0 {
return Err(AppError::NotFound("session not active".into()));
}
Ok(StatusCode::NO_CONTENT)
}
pub async fn release_session(
State(st): State<AppState>,
Path(id): Path<Uuid>,
) -> Result<StatusCode, AppError> {
let session = sqlx::query_as::<_, Session>(
"UPDATE sessions SET ended_at=now() WHERE id=$1 AND ended_at IS NULL RETURNING *",
)
.bind(id)
.fetch_optional(&st.db)
.await?
.ok_or_else(|| AppError::NotFound("session not found or already ended".into()))?;
// Return the device to the available pool. App-state reset on release (§8)
// belongs to the stream node and arrives with Phase 1.
sqlx::query("UPDATE devices SET status='free', updated_at=now() WHERE serial=$1")
.bind(&session.device_serial)
.execute(&st.db)
.await?;
emit(
&st,
AppEvent::DeviceStatus {
serial: session.device_serial.clone(),
status: "free".into(),
},
);
emit(&st, AppEvent::SessionEnded { session_id: session.id });
Ok(StatusCode::NO_CONTENT)
}
pub async fn list_findings(
State(st): State<AppState>,
Path(session_id): Path<Uuid>,
) -> Result<Json<Vec<Finding>>, AppError> {
let findings = sqlx::query_as::<_, Finding>(
"SELECT * FROM findings WHERE session_id=$1 ORDER BY created_at",
)
.bind(session_id)
.fetch_all(&st.db)
.await?;
Ok(Json(findings))
}
pub async fn create_finding(
State(st): State<AppState>,
Path(session_id): Path<Uuid>,
Json(body): Json<CreateFinding>,
) -> Result<Json<Finding>, AppError> {
let session = load_session(&st.db, session_id).await?;
// Auto-stamp the anchoring metadata from the session's source of truth.
let build_hash = build_hash_for(&st.db, &session).await?;
let time_offset_ms = (Utc::now() - session.started_at).num_milliseconds();
let finding = sqlx::query_as::<_, Finding>(
"INSERT INTO findings \
(id, session_id, device_serial, build_hash, reviewer_id, kind, note_text, time_offset_ms) \
VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING *",
)
.bind(Uuid::new_v4())
.bind(session_id)
.bind(&session.device_serial)
.bind(build_hash)
.bind(session.reviewer_id)
.bind(body.kind.as_deref().unwrap_or("note"))
.bind(body.note_text)
.bind(time_offset_ms)
.fetch_one(&st.db)
.await?;
Ok(Json(finding))
}
pub async fn update_finding(
State(st): State<AppState>,
Path(id): Path<Uuid>,
Json(body): Json<UpdateFinding>,
) -> Result<Json<Finding>, AppError> {
// Debounced-autosave target: patch note text, leave anchoring untouched.
let finding = sqlx::query_as::<_, Finding>(
"UPDATE findings SET note_text=COALESCE($2, note_text) WHERE id=$1 RETURNING *",
)
.bind(id)
.bind(body.note_text)
.fetch_optional(&st.db)
.await?
.ok_or_else(|| AppError::NotFound("finding not found".into()))?;
Ok(Json(finding))
}
/// Attach a captured `<video>` frame to an existing finding (§4.5). The still is
/// stored and its reference recorded; the rest of the finding is untouched.
pub async fn attach_frame(
State(st): State<AppState>,
Path(finding_id): Path<Uuid>,
mut multipart: Multipart,
) -> Result<Json<Finding>, AppError> {
let (content_type, data) = read_upload(&mut multipart, "frame", "image/png").await?;
let exists: Option<Uuid> = sqlx::query_scalar("SELECT id FROM findings WHERE id=$1")
.bind(finding_id)
.fetch_optional(&st.db)
.await?;
if exists.is_none() {
return Err(AppError::NotFound("finding not found".into()));
}
let frame_ref = crate::storage::put(
&st.storage_dir,
"frames",
&format!("{finding_id}.{}", image_ext(&content_type)),
&data,
)
.await?;
let finding = sqlx::query_as::<_, Finding>(
"UPDATE findings SET frame_ref=$2 WHERE id=$1 RETURNING *",
)
.bind(finding_id)
.bind(frame_ref)
.fetch_one(&st.db)
.await?;
Ok(Json(finding))
}
pub async fn create_audio_finding(
State(st): State<AppState>,
Path(session_id): Path<Uuid>,
mut multipart: Multipart,
) -> Result<Json<Finding>, AppError> {
let session = load_session(&st.db, session_id).await?;
let (content_type, data) = read_upload(&mut multipart, "audio", "application/octet-stream").await?;
let finding_id = Uuid::new_v4();
let ext = match content_type.as_str() {
"audio/webm" => "webm",
"audio/ogg" => "ogg",
"audio/mpeg" => "mp3",
"audio/mp4" | "audio/aac" => "m4a",
"audio/wav" | "audio/x-wav" => "wav",
_ => "bin",
};
let audio_ref =
crate::storage::put(&st.storage_dir, "audio", &format!("{finding_id}.{ext}"), &data)
.await?;
// Transcription handoff to the separate service (§6.4). A transcription failure
// must not lose the recording: save the finding with a null transcript and log.
let transcript = match crate::transcription::transcribe(
st.transcribe_url.as_deref(),
&data,
&content_type,
)
.await
{
Ok(t) => t,
Err(e) => {
tracing::warn!(error = ?e, "transcription failed; saving finding without transcript");
None
}
};
let build_hash = build_hash_for(&st.db, &session).await?;
let time_offset_ms = (Utc::now() - session.started_at).num_milliseconds();
let finding = sqlx::query_as::<_, Finding>(
"INSERT INTO findings \
(id, session_id, device_serial, build_hash, reviewer_id, kind, audio_ref, transcript, time_offset_ms) \
VALUES ($1, $2, $3, $4, $5, 'voice', $6, $7, $8) RETURNING *",
)
.bind(finding_id)
.bind(session_id)
.bind(&session.device_serial)
.bind(build_hash)
.bind(session.reviewer_id)
.bind(audio_ref)
.bind(transcript)
.bind(time_offset_ms)
.fetch_one(&st.db)
.await?;
Ok(Json(finding))
}
/// Advance a multipart stream, mapping parse errors (malformed/empty body, a
/// truncated upload) to 400 rather than 500 — they are client-supplied.
async fn next_field(
mp: &mut Multipart,
) -> Result<Option<axum::extract::multipart::Field<'_>>, AppError> {
mp.next_field()
.await
.map_err(|e| AppError::BadRequest(format!("invalid multipart upload: {e}")))
}
/// Read a single named file part from a multipart upload, returning its
/// content-type (or `default_ct`) and bytes.
async fn read_upload(
multipart: &mut Multipart,
field_name: &str,
default_ct: &str,
) -> Result<(String, axum::body::Bytes), AppError> {
let mut content_type = default_ct.to_string();
let mut data: Option<axum::body::Bytes> = None;
while let Some(field) = next_field(multipart).await? {
if field.name() == Some(field_name) {
if let Some(ct) = field.content_type() {
content_type = ct.to_string();
}
data = Some(
field
.bytes()
.await
.map_err(|e| AppError::BadRequest(format!("invalid multipart upload: {e}")))?,
);
}
}
let data = data.ok_or_else(|| AppError::BadRequest(format!("missing '{field_name}' field")))?;
Ok((content_type, data))
}
fn image_ext(content_type: &str) -> &'static str {
match content_type {
"image/png" => "png",
"image/jpeg" => "jpg",
"image/webp" => "webp",
_ => "bin",
}
}
/// Transcribe audio without persisting anything — used for live, interim dictation
/// while the reviewer is still speaking (§4.5). Returns `{ "text": ... }`.
pub async fn transcribe_audio(
State(st): State<AppState>,
mut multipart: Multipart,
) -> Result<Json<Value>, AppError> {
let (content_type, data) = read_upload(&mut multipart, "audio", "audio/webm").await?;
let text = crate::transcription::transcribe(st.transcribe_url.as_deref(), &data, &content_type)
.await?
.unwrap_or_default();
Ok(Json(json!({ "text": text })))
}
/// Create a finding from a one-tap screen snapshot (§4.5): stores the frame and
/// anchors a `frame`-kind finding to the session.
pub async fn create_frame_finding(
State(st): State<AppState>,
Path(session_id): Path<Uuid>,
mut multipart: Multipart,
) -> Result<Json<Finding>, AppError> {
let session = load_session(&st.db, session_id).await?;
let (content_type, data) = read_upload(&mut multipart, "frame", "image/png").await?;
let finding_id = Uuid::new_v4();
let frame_ref = crate::storage::put(
&st.storage_dir,
"frames",
&format!("{finding_id}.{}", image_ext(&content_type)),
&data,
)
.await?;
let build_hash = build_hash_for(&st.db, &session).await?;
let time_offset_ms = (Utc::now() - session.started_at).num_milliseconds();
let finding = sqlx::query_as::<_, Finding>(
"INSERT INTO findings \
(id, session_id, device_serial, build_hash, reviewer_id, kind, frame_ref, time_offset_ms) \
VALUES ($1, $2, $3, $4, $5, 'frame', $6, $7) RETURNING *",
)
.bind(finding_id)
.bind(session_id)
.bind(&session.device_serial)
.bind(build_hash)
.bind(session.reviewer_id)
.bind(frame_ref)
.bind(time_offset_ms)
.fetch_one(&st.db)
.await?;
Ok(Json(finding))
}
/// Serve a finding's stored frame image for display in the review surface.
pub async fn get_frame(
State(st): State<AppState>,
Path(id): Path<Uuid>,
) -> Result<axum::response::Response, AppError> {
let reference: Option<String> = sqlx::query_scalar("SELECT frame_ref FROM findings WHERE id=$1")
.bind(id)
.fetch_optional(&st.db)
.await?
.flatten();
let reference = reference.ok_or_else(|| AppError::NotFound("no frame for finding".into()))?;
let bytes = crate::storage::get(&st.storage_dir, &reference).await?;
let content_type = crate::storage::image_content_type(&reference);
Ok(([(axum::http::header::CONTENT_TYPE, content_type)], bytes).into_response())
}
async fn load_session(db: &PgPool, id: Uuid) -> Result<Session, AppError> {
sqlx::query_as::<_, Session>("SELECT * FROM sessions WHERE id=$1")
.bind(id)
.fetch_optional(db)
.await?
.ok_or_else(|| AppError::NotFound("session not found".into()))
}
async fn build_hash_for(db: &PgPool, session: &Session) -> Result<Option<String>, AppError> {
match session.build_id {
Some(bid) => Ok(sqlx::query_scalar("SELECT hash FROM builds WHERE id=$1")
.bind(bid)
.fetch_optional(db)
.await?),
None => Ok(None),
}
}
/// Seed the dev reviewer plus one placeholder build and device so the Track-1
/// review surface has something to claim without any streaming code.
pub async fn seed_placeholder(db: &PgPool) -> anyhow::Result<()> {
sqlx::query(
"INSERT INTO users (id, email, display_name, role) VALUES ($1, $2, $3, 'engineer') \
ON CONFLICT (email) DO NOTHING",
)
.bind(DEV_USER_ID)
.bind("[email protected]")
.bind("Dev Reviewer")
.execute(db)
.await?;
sqlx::query(
"INSERT INTO builds (id, app_identity, version, hash) VALUES ($1, $2, $3, $4) \
ON CONFLICT (id) DO NOTHING",
)
.bind(PLACEHOLDER_BUILD_ID)
.bind("com.redclaw.digitalhallmark")
.bind("0.0.0-placeholder")
.bind("deadbeefcafe")
.execute(db)
.await?;
// The real rig device (Phase 1: single physical device streamed by streamd).
sqlx::query(
"INSERT INTO devices (serial, model, status, stream_node, current_build_id) \
VALUES ($1, $2, 'free', 'local-streamd', $3) ON CONFLICT (serial) DO NOTHING",
)
.bind("A11PRO052500527")
.bind("LAGENIO A11 Pro")
.bind(PLACEHOLDER_BUILD_ID)
.execute(db)
.await?;
Ok(())
}
+63
View File
@@ -0,0 +1,63 @@
pub mod error;
pub mod events;
pub mod handlers;
pub mod models;
pub mod reaper;
pub mod storage;
pub mod transcription;
use std::path::PathBuf;
use axum::{
routing::{get, patch, post},
Router,
};
use tower_http::cors::CorsLayer;
use crate::events::AppEvent;
/// Shared application state. Storage location and the transcription endpoint live
/// here (not read from env inside handlers) so tests can inject isolated values
/// and run in parallel without clobbering a process-global.
#[derive(Clone)]
pub struct AppState {
pub db: sqlx::PgPool,
pub storage_dir: PathBuf,
pub transcribe_url: Option<String>,
pub events: tokio::sync::broadcast::Sender<AppEvent>,
/// A claim with no heartbeat for this long is treated as disconnected.
pub disconnect_timeout_secs: i64,
/// A claim with no interaction for this long is treated as dormant (§4.4).
pub idle_timeout_secs: i64,
}
pub fn build_router(state: AppState) -> Router {
Router::new()
.route("/health", get(handlers::health))
.route("/api/events", get(events::events_handler))
.route("/api/devices", get(handlers::list_devices))
.route("/api/sessions", post(handlers::create_session))
.route("/api/sessions/:id/release", post(handlers::release_session))
.route("/api/sessions/:id/heartbeat", post(handlers::heartbeat))
.route(
"/api/sessions/:id/findings",
get(handlers::list_findings).post(handlers::create_finding),
)
.route(
"/api/sessions/:id/findings/audio",
post(handlers::create_audio_finding),
)
.route(
"/api/sessions/:id/findings/frame",
post(handlers::create_frame_finding),
)
.route("/api/transcribe", post(handlers::transcribe_audio))
.route("/api/findings/:id", patch(handlers::update_finding))
.route(
"/api/findings/:id/frame",
post(handlers::attach_frame).get(handlers::get_frame),
)
// Permissive CORS so the Vite dev server (different origin) can call the API.
.layer(CorsLayer::permissive())
.with_state(state)
}
+64
View File
@@ -0,0 +1,64 @@
use anyhow::Context;
use review_api::{build_router, handlers, AppState};
use sqlx::postgres::PgPoolOptions;
use std::time::Duration;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
fn env_secs(key: &str, default: i64) -> i64 {
std::env::var(key).ok().and_then(|v| v.parse().ok()).unwrap_or(default)
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
tracing_subscriber::registry()
.with(
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| "review_api=debug,tower_http=info,info".into()),
)
.with(tracing_subscriber::fmt::layer())
.init();
let database_url = std::env::var("DATABASE_URL").unwrap_or_else(|_| {
"postgres://clawreview:clawreview@localhost:5432/clawreview".to_string()
});
let db = PgPoolOptions::new()
.max_connections(10)
.acquire_timeout(Duration::from_secs(5))
.connect(&database_url)
.await
.context("failed to connect to Postgres")?;
sqlx::migrate!()
.run(&db)
.await
.context("database migrations failed")?;
handlers::seed_placeholder(&db)
.await
.context("seeding placeholder fixtures failed")?;
let state = AppState {
db,
storage_dir: std::env::var("STORAGE_DIR")
.unwrap_or_else(|_| "./data".to_string())
.into(),
transcribe_url: Some(
std::env::var("TRANSCRIBE_URL")
.unwrap_or_else(|_| "http://127.0.0.1:8099/transcribe".to_string()),
),
events: review_api::events::channel(),
disconnect_timeout_secs: env_secs("DISCONNECT_TIMEOUT_SECS", 45),
idle_timeout_secs: env_secs("IDLE_TIMEOUT_SECS", 600),
};
// Background dormancy reaper: auto-frees disconnected/idle devices (§4.4).
review_api::reaper::spawn(state.clone());
let app = build_router(state);
let addr = std::env::var("BIND_ADDR").unwrap_or_else(|_| "0.0.0.0:8090".to_string());
let listener = tokio::net::TcpListener::bind(&addr).await?;
tracing::info!("review-api listening on http://{addr}");
axum::serve(listener, app).await?;
Ok(())
}
+64
View File
@@ -0,0 +1,64 @@
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use sqlx::FromRow;
use uuid::Uuid;
#[derive(Debug, Serialize, FromRow)]
pub struct Device {
pub serial: String,
pub model: Option<String>,
pub status: String,
pub stream_node: Option<String>,
pub current_build_id: Option<Uuid>,
pub updated_at: DateTime<Utc>,
}
#[derive(Debug, Serialize, FromRow)]
pub struct Session {
pub id: Uuid,
pub reviewer_id: Uuid,
pub device_serial: String,
pub build_id: Option<Uuid>,
pub started_at: DateTime<Utc>,
pub ended_at: Option<DateTime<Utc>>,
pub heartbeat_at: DateTime<Utc>,
pub last_activity_at: DateTime<Utc>,
}
#[derive(Debug, Serialize, FromRow)]
pub struct Finding {
pub id: Uuid,
pub session_id: Uuid,
pub device_serial: String,
pub build_hash: Option<String>,
pub reviewer_id: Uuid,
pub kind: String,
pub note_text: Option<String>,
pub audio_ref: Option<String>,
pub transcript: Option<String>,
pub frame_ref: Option<String>,
pub clip_ref: Option<String>,
pub time_offset_ms: Option<i64>,
pub created_at: DateTime<Utc>,
}
// ---- request bodies ----
#[derive(Debug, Deserialize)]
pub struct CreateSession {
pub device_serial: String,
}
#[derive(Debug, Deserialize)]
pub struct CreateFinding {
#[serde(default)]
pub kind: Option<String>,
#[serde(default)]
pub note_text: Option<String>,
}
#[derive(Debug, Deserialize)]
pub struct UpdateFinding {
#[serde(default)]
pub note_text: Option<String>,
}
+83
View File
@@ -0,0 +1,83 @@
//! Background dormancy reaper (§4.4, device-sharing scenario). Periodically frees
//! devices whose active session has either disconnected (stale heartbeat) or gone
//! dormant (no interaction for the idle window), and pushes the change to everyone
//! over SSE. This makes reclamation automatic rather than only-on-next-claim.
use std::collections::HashSet;
use std::time::Duration;
use sqlx::PgPool;
use tokio::sync::broadcast::Sender;
use uuid::Uuid;
use crate::events::AppEvent;
use crate::AppState;
/// End every reclaimable session and free its device, returning the freed serials.
/// A session is reclaimable when `heartbeat_at` is older than `disconnect_secs`
/// (browser gone) OR `last_activity_at` is older than `idle_secs` (dormant).
pub async fn reap_dormant(
db: &PgPool,
events: &Sender<AppEvent>,
disconnect_secs: i64,
idle_secs: i64,
) -> anyhow::Result<Vec<String>> {
let reaped: Vec<(Uuid, String)> = sqlx::query_as(
"UPDATE sessions SET ended_at=now() \
WHERE ended_at IS NULL \
AND (heartbeat_at < now() - make_interval(secs => $1) \
OR last_activity_at < now() - make_interval(secs => $2)) \
RETURNING id, device_serial",
)
.bind(disconnect_secs as f64)
.bind(idle_secs as f64)
.fetch_all(db)
.await?;
// Free each affected device that now has no active session, and announce it.
let mut freed = Vec::new();
let serials: HashSet<String> = reaped.iter().map(|(_, s)| s.clone()).collect();
for serial in serials {
let updated = sqlx::query(
"UPDATE devices SET status='free', updated_at=now() \
WHERE serial=$1 AND status='claimed' \
AND NOT EXISTS (SELECT 1 FROM sessions WHERE device_serial=$1 AND ended_at IS NULL)",
)
.bind(&serial)
.execute(db)
.await?;
if updated.rows_affected() > 0 {
let _ = events.send(AppEvent::DeviceStatus {
serial: serial.clone(),
status: "free".into(),
});
freed.push(serial);
}
}
for (session_id, _) in reaped {
let _ = events.send(AppEvent::SessionEnded { session_id });
}
Ok(freed)
}
/// Spawn the periodic reaper loop. Runs for the life of the process.
pub fn spawn(state: AppState) {
tokio::spawn(async move {
let mut ticker = tokio::time::interval(Duration::from_secs(15));
loop {
ticker.tick().await;
if let Err(e) = reap_dormant(
&state.db,
&state.events,
state.disconnect_timeout_secs,
state.idle_timeout_secs,
)
.await
{
tracing::warn!(error = ?e, "dormancy reaper failed");
}
}
});
}
+38
View File
@@ -0,0 +1,38 @@
use std::path::Path;
/// Persist a blob under `base_dir/category/key` and return its storage reference.
/// The local filesystem stands in for Cloudflare R2 (§5) in dev; swap this impl
/// for an R2 client without touching callers — the returned reference string is
/// what lives in Postgres.
pub async fn put(
base_dir: &Path,
category: &str,
key: &str,
bytes: &[u8],
) -> anyhow::Result<String> {
let dir = base_dir.join(category);
tokio::fs::create_dir_all(&dir).await?;
tokio::fs::write(dir.join(key), bytes).await?;
Ok(format!("local://{category}/{key}"))
}
/// Read back a blob previously stored with `put`, given its `local://…` reference.
pub async fn get(base_dir: &Path, reference: &str) -> anyhow::Result<Vec<u8>> {
let rel = reference
.strip_prefix("local://")
.ok_or_else(|| anyhow::anyhow!("not a local reference: {reference}"))?;
Ok(tokio::fs::read(base_dir.join(rel)).await?)
}
/// Guess an image content-type from a stored reference's extension.
pub fn image_content_type(reference: &str) -> &'static str {
if reference.ends_with(".png") {
"image/png"
} else if reference.ends_with(".jpg") || reference.ends_with(".jpeg") {
"image/jpeg"
} else if reference.ends_with(".webp") {
"image/webp"
} else {
"application/octet-stream"
}
}
+31
View File
@@ -0,0 +1,31 @@
use serde_json::Value;
/// Hand audio to the self-hosted Whisper-class service (§6.4) over HTTP, kept off
/// the real-time streaming path. Returns `Ok(None)` when no endpoint is configured
/// so a finding can still be saved without a transcript; returns `Ok(Some(text))`
/// when the service responds with `{ "text": ... }`.
pub async fn transcribe(
endpoint: Option<&str>,
bytes: &[u8],
content_type: &str,
) -> anyhow::Result<Option<String>> {
let Some(url) = endpoint else {
return Ok(None);
};
let resp = reqwest::Client::new()
.post(url)
.header("content-type", content_type)
.body(bytes.to_vec())
.send()
.await?
.error_for_status()?;
let body: Value = resp.json().await?;
Ok(Some(
body.get("text")
.and_then(Value::as_str)
.unwrap_or_default()
.to_string(),
))
}
+525
View File
@@ -0,0 +1,525 @@
//! Integration tests for the review API. Each test runs against its own ephemeral
//! Postgres database (`#[sqlx::test]`) and drives the real Axum router over HTTP.
use review_api::{build_router, handlers, AppState};
use serde_json::Value;
use sqlx::PgPool;
use std::path::PathBuf;
use uuid::Uuid;
const DEVICE: &str = "A11PRO052500527";
struct TestApp {
base: String,
client: reqwest::Client,
storage_dir: PathBuf,
pool: PgPool,
events: tokio::sync::broadcast::Sender<review_api::events::AppEvent>,
}
async fn spawn(pool: PgPool, transcribe_url: Option<String>) -> TestApp {
handlers::seed_placeholder(&pool).await.unwrap();
let storage_dir = std::env::temp_dir().join(format!("clawreview-test-{}", Uuid::new_v4()));
let events = review_api::events::channel();
let state = AppState {
db: pool.clone(),
storage_dir: storage_dir.clone(),
transcribe_url,
events: events.clone(),
disconnect_timeout_secs: 45,
idle_timeout_secs: 600,
};
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, build_router(state)).await.unwrap();
});
TestApp {
base: format!("http://{addr}"),
client: reqwest::Client::new(),
storage_dir,
pool,
events,
}
}
/// A throwaway server that mimics the transcription service (§6.4).
async fn spawn_fake_transcriber(text: &'static str) -> String {
let app = axum::Router::new().route(
"/transcribe",
axum::routing::post(move || async move {
axum::Json(serde_json::json!({ "text": text }))
}),
);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
format!("http://{addr}/transcribe")
}
impl TestApp {
async fn claim(&self) -> String {
let res = self
.client
.post(format!("{}/api/sessions", self.base))
.json(&serde_json::json!({ "device_serial": DEVICE }))
.send()
.await
.unwrap();
assert_eq!(res.status(), 200, "claim should succeed");
res.json::<Value>().await.unwrap()["id"]
.as_str()
.unwrap()
.to_string()
}
async fn try_claim(&self) -> reqwest::StatusCode {
self.client
.post(format!("{}/api/sessions", self.base))
.json(&serde_json::json!({ "device_serial": DEVICE }))
.send()
.await
.unwrap()
.status()
}
async fn create_note(&self, session: &str, text: &str) -> Value {
self.client
.post(format!("{}/api/sessions/{session}/findings", self.base))
.json(&serde_json::json!({ "kind": "note", "note_text": text }))
.send()
.await
.unwrap()
.json::<Value>()
.await
.unwrap()
}
/// Directly insert an active claim held by a *different* reviewer, with a given
/// heartbeat age in seconds, and mark the device claimed — to exercise the reap.
async fn inject_foreign_claim(&self, heartbeat_age_secs: i64) {
let other = Uuid::new_v4();
sqlx::query("INSERT INTO users (id, email, display_name, role) VALUES ($1, $2, 'Other', 'engineer')")
.bind(other)
.bind(format!("other-{other}@redclaw.local"))
.execute(&self.pool)
.await
.unwrap();
sqlx::query(
"INSERT INTO sessions (id, reviewer_id, device_serial, started_at, heartbeat_at) \
VALUES ($1, $2, $3, now(), now() - make_interval(secs => $4))",
)
.bind(Uuid::new_v4())
.bind(other)
.bind(DEVICE)
.bind(heartbeat_age_secs as f64)
.execute(&self.pool)
.await
.unwrap();
sqlx::query("UPDATE devices SET status='claimed' WHERE serial=$1")
.bind(DEVICE)
.execute(&self.pool)
.await
.unwrap();
}
}
#[sqlx::test]
async fn same_reviewer_reclaims_on_refresh(pool: PgPool) {
let app = spawn(pool, None).await;
let first = app.claim().await;
// A refresh re-claims as the same reviewer: it takes over rather than conflicting.
assert_eq!(app.try_claim().await, 200, "same reviewer should reclaim");
// Exactly one active session remains for the device.
let active: i64 =
sqlx::query_scalar("SELECT count(*) FROM sessions WHERE device_serial=$1 AND ended_at IS NULL")
.bind(DEVICE)
.fetch_one(&app.pool)
.await
.unwrap();
assert_eq!(active, 1);
// The original session was ended by the takeover.
let ended: bool =
sqlx::query_scalar("SELECT ended_at IS NOT NULL FROM sessions WHERE id=$1::uuid")
.bind(&first)
.fetch_one(&app.pool)
.await
.unwrap();
assert!(ended, "the prior session should be ended after takeover");
}
#[sqlx::test]
async fn reaper_frees_dormant_devices(pool: PgPool) {
use review_api::events::AppEvent;
let app = spawn(pool, None).await;
let _ = app.claim().await; // fresh claim
// Simulate dormancy: no interaction for 15 minutes (idle window is 10).
sqlx::query(
"UPDATE sessions SET last_activity_at = now() - make_interval(secs => 900) \
WHERE device_serial=$1 AND ended_at IS NULL",
)
.bind(DEVICE)
.execute(&app.pool)
.await
.unwrap();
let mut rx = app.events.subscribe();
let freed = review_api::reaper::reap_dormant(&app.pool, &app.events, 45, 600)
.await
.unwrap();
assert_eq!(freed, vec![DEVICE.to_string()]);
let status: String = sqlx::query_scalar("SELECT status FROM devices WHERE serial=$1")
.bind(DEVICE)
.fetch_one(&app.pool)
.await
.unwrap();
assert_eq!(status, "free", "dormant device should be auto-freed");
// A "device free" event is broadcast to all subscribers.
let mut saw_free = false;
while let Ok(ev) = rx.try_recv() {
if matches!(ev, AppEvent::DeviceStatus { ref status, .. } if status == "free") {
saw_free = true;
}
}
assert!(saw_free, "freeing a dormant device should broadcast device_status free");
}
#[sqlx::test]
async fn reaper_leaves_active_sessions(pool: PgPool) {
let app = spawn(pool, None).await;
let _ = app.claim().await; // fresh heartbeat + activity
let freed = review_api::reaper::reap_dormant(&app.pool, &app.events, 45, 600)
.await
.unwrap();
assert!(freed.is_empty(), "an active session must not be reaped");
let status: String = sqlx::query_scalar("SELECT status FROM devices WHERE serial=$1")
.bind(DEVICE)
.fetch_one(&app.pool)
.await
.unwrap();
assert_eq!(status, "claimed");
}
#[sqlx::test]
async fn active_heartbeat_resets_the_dormancy_clock(pool: PgPool) {
let app = spawn(pool, None).await;
let session = app.claim().await;
// Go dormant…
sqlx::query(
"UPDATE sessions SET last_activity_at = now() - make_interval(secs => 900) WHERE id=$1::uuid",
)
.bind(&session)
.execute(&app.pool)
.await
.unwrap();
// …then interact: an active heartbeat bumps last_activity_at.
let res = app
.client
.post(format!("{}/api/sessions/{session}/heartbeat?active=true", app.base))
.send()
.await
.unwrap();
assert_eq!(res.status(), 204);
let freed = review_api::reaper::reap_dormant(&app.pool, &app.events, 45, 600)
.await
.unwrap();
assert!(freed.is_empty(), "a freshly-active session must not be reaped");
}
#[sqlx::test]
async fn another_reviewers_live_claim_blocks(pool: PgPool) {
let app = spawn(pool, None).await;
app.inject_foreign_claim(0).await; // fresh heartbeat → genuinely in use
assert_eq!(app.try_claim().await, 409, "a live foreign claim must block");
}
#[sqlx::test]
async fn stale_claim_is_reclaimable(pool: PgPool) {
let app = spawn(pool, None).await;
app.inject_foreign_claim(120).await; // heartbeat 2min old → abandoned
assert_eq!(app.try_claim().await, 200, "a stale claim must be reclaimable");
}
#[sqlx::test]
async fn heartbeat_keeps_session_then_404s_after_release(pool: PgPool) {
let app = spawn(pool, None).await;
let session = app.claim().await;
let beat = app
.client
.post(format!("{}/api/sessions/{session}/heartbeat", app.base))
.send()
.await
.unwrap();
assert_eq!(beat.status(), 204);
app.client
.post(format!("{}/api/sessions/{session}/release", app.base))
.send()
.await
.unwrap();
// After release the session is no longer active, so heartbeats 404.
let beat_after = app
.client
.post(format!("{}/api/sessions/{session}/heartbeat", app.base))
.send()
.await
.unwrap();
assert_eq!(beat_after.status(), 404);
}
#[sqlx::test]
async fn claim_broadcasts_a_device_status_event(pool: PgPool) {
use review_api::events::AppEvent;
let app = spawn(pool, None).await;
let mut rx = app.events.subscribe(); // subscribe before claiming
let _ = app.claim().await;
let ev = tokio::time::timeout(std::time::Duration::from_secs(2), rx.recv())
.await
.expect("an event should be broadcast")
.expect("recv ok");
match ev {
AppEvent::DeviceStatus { serial, status } => {
assert_eq!(serial, DEVICE);
assert_eq!(status, "claimed");
}
other => panic!("expected DeviceStatus claimed, got {other:?}"),
}
}
#[sqlx::test]
async fn note_is_anchored_and_autosaves(pool: PgPool) {
let app = spawn(pool, None).await;
let session = app.claim().await;
let finding = app.create_note(&session, "draft").await;
let fid = finding["id"].as_str().unwrap();
assert_eq!(finding["device_serial"], DEVICE);
assert_eq!(finding["build_hash"], "deadbeefcafe");
assert!(finding["time_offset_ms"].as_i64().unwrap() >= 0);
// PATCH (autosave) updates the text, leaving anchoring intact.
let patched: Value = app
.client
.patch(format!("{}/api/findings/{fid}", app.base))
.json(&serde_json::json!({ "note_text": "tap latency ~1.8s" }))
.send()
.await
.unwrap()
.json()
.await
.unwrap();
assert_eq!(patched["note_text"], "tap latency ~1.8s");
assert_eq!(patched["build_hash"], "deadbeefcafe");
}
#[sqlx::test]
async fn frame_attaches_to_finding_and_persists(pool: PgPool) {
let app = spawn(pool, None).await;
let session = app.claim().await;
let finding = app.create_note(&session, "result screen").await;
let fid = finding["id"].as_str().unwrap().to_string();
let png = b"\x89PNG\r\n\x1a\n-fake-frame-bytes".to_vec();
let form = reqwest::multipart::Form::new().part(
"frame",
reqwest::multipart::Part::bytes(png.clone())
.file_name("frame.png")
.mime_str("image/png")
.unwrap(),
);
let res = app
.client
.post(format!("{}/api/findings/{fid}/frame", app.base))
.multipart(form)
.send()
.await
.unwrap();
assert_eq!(res.status(), 200);
let body: Value = res.json().await.unwrap();
assert_eq!(body["frame_ref"], format!("local://frames/{fid}.png"));
// The blob is actually on disk.
let path = app.storage_dir.join("frames").join(format!("{fid}.png"));
let stored = std::fs::read(&path).expect("frame file should exist");
assert_eq!(stored, png);
}
#[sqlx::test]
async fn frame_on_missing_finding_is_404(pool: PgPool) {
let app = spawn(pool, None).await;
let form = reqwest::multipart::Form::new().part(
"frame",
reqwest::multipart::Part::bytes(b"x".to_vec())
.file_name("f.png")
.mime_str("image/png")
.unwrap(),
);
let res = app
.client
.post(format!("{}/api/findings/{}/frame", app.base, Uuid::new_v4()))
.multipart(form)
.send()
.await
.unwrap();
assert_eq!(res.status(), 404);
}
#[sqlx::test]
async fn frame_without_field_is_400(pool: PgPool) {
let app = spawn(pool, None).await;
let session = app.claim().await;
let finding = app.create_note(&session, "n").await;
let fid = finding["id"].as_str().unwrap();
let res = app
.client
.post(format!("{}/api/findings/{fid}/frame", app.base))
.multipart(reqwest::multipart::Form::new()) // no "frame" part
.send()
.await
.unwrap();
assert_eq!(res.status(), 400);
}
#[sqlx::test]
async fn live_transcribe_returns_text_without_persisting(pool: PgPool) {
let transcribe_url = spawn_fake_transcriber("live dictation text").await;
let app = spawn(pool, Some(transcribe_url)).await;
let session = app.claim().await;
let form = reqwest::multipart::Form::new().part(
"audio",
reqwest::multipart::Part::bytes(b"-partial-audio-".to_vec())
.file_name("live.webm")
.mime_str("audio/webm")
.unwrap(),
);
let body: Value = app
.client
.post(format!("{}/api/transcribe", app.base))
.multipart(form)
.send()
.await
.unwrap()
.json()
.await
.unwrap();
assert_eq!(body["text"], "live dictation text");
// Nothing was persisted as a finding.
let findings: Value = app
.client
.get(format!("{}/api/sessions/{session}/findings", app.base))
.send()
.await
.unwrap()
.json()
.await
.unwrap();
assert_eq!(findings.as_array().unwrap().len(), 0);
}
#[sqlx::test]
async fn frame_snapshot_creates_finding_and_serves_image(pool: PgPool) {
let app = spawn(pool, None).await;
let session = app.claim().await;
let png = b"\x89PNG\r\n\x1a\n-screen-snapshot".to_vec();
let form = reqwest::multipart::Form::new().part(
"frame",
reqwest::multipart::Part::bytes(png.clone())
.file_name("screen.png")
.mime_str("image/png")
.unwrap(),
);
let finding: Value = app
.client
.post(format!("{}/api/sessions/{session}/findings/frame", app.base))
.multipart(form)
.send()
.await
.unwrap()
.json()
.await
.unwrap();
assert_eq!(finding["kind"], "frame");
assert_eq!(finding["device_serial"], DEVICE);
let fid = finding["id"].as_str().unwrap();
// The frame is served back with an image content-type and the exact bytes.
let res = app
.client
.get(format!("{}/api/findings/{fid}/frame", app.base))
.send()
.await
.unwrap();
assert_eq!(res.status(), 200);
assert_eq!(res.headers()["content-type"], "image/png");
assert_eq!(res.bytes().await.unwrap().to_vec(), png);
}
#[sqlx::test]
async fn get_frame_404_when_finding_has_none(pool: PgPool) {
let app = spawn(pool, None).await;
let session = app.claim().await;
let note = app.create_note(&session, "no frame here").await;
let fid = note["id"].as_str().unwrap();
let res = app
.client
.get(format!("{}/api/findings/{fid}/frame", app.base))
.send()
.await
.unwrap();
assert_eq!(res.status(), 404);
}
#[sqlx::test]
async fn voice_finding_stores_audio_and_transcript(pool: PgPool) {
let transcribe_url = spawn_fake_transcriber("nfc read resolved in about two seconds").await;
let app = spawn(pool, Some(transcribe_url)).await;
let session = app.claim().await;
let audio = b"-fake-webm-opus-bytes-".to_vec();
let form = reqwest::multipart::Form::new().part(
"audio",
reqwest::multipart::Part::bytes(audio.clone())
.file_name("rec.webm")
.mime_str("audio/webm")
.unwrap(),
);
let res = app
.client
.post(format!("{}/api/sessions/{session}/findings/audio", app.base))
.multipart(form)
.send()
.await
.unwrap();
assert_eq!(res.status(), 200);
let body: Value = res.json().await.unwrap();
assert_eq!(body["kind"], "voice");
assert_eq!(body["transcript"], "nfc read resolved in about two seconds");
assert_eq!(body["audio_ref"], format!("local://audio/{}.webm", body["id"].as_str().unwrap()));
// Audio blob persisted.
let path = app
.storage_dir
.join("audio")
.join(format!("{}.webm", body["id"].as_str().unwrap()));
assert_eq!(std::fs::read(&path).unwrap(), audio);
}
+28
View File
@@ -0,0 +1,28 @@
[package]
name = "stream-node"
version = "0.1.0"
edition = "2021"
[dependencies]
tokio = { version = "1", features = ["rt-multi-thread", "macros", "net", "io-util", "process", "time", "sync"] }
anyhow = "1"
bytes = "1"
# 0.20.x is an incomplete sans-IO rewrite; 0.11 is the last stable classic API
# (MediaEngine / TrackLocalStaticSample / write_sample). See §9 risk #5.
webrtc = "0.11"
axum = { version = "0.7", features = ["ws"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
[[bin]]
name = "scrcpy-probe"
path = "src/bin/probe.rs"
[[bin]]
name = "streamd"
path = "src/bin/streamd.rs"
[dev-dependencies]
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json"] }
+105
View File
@@ -0,0 +1,105 @@
//! Single-device spike probe (§10 Phase 0, Track 2): start a real scrcpy session,
//! read the H.264 stream off the device, dump the first access unit for validation,
//! and exercise the control socket with a tap. Run with a device attached:
//! cargo run -p stream-node --bin scrcpy-probe -- [serial] [server-jar]
use std::time::Duration;
use anyhow::Result;
use stream_node::control::{KeyAction, MotionAction};
use stream_node::coords::VideoSize;
use stream_node::scrcpy::{StreamSession, CODEC_ID_H264};
use stream_node::{key_message, touch_message, PointerEvent};
#[tokio::main]
async fn main() -> Result<()> {
let mut args = std::env::args().skip(1);
let serial = args.next().unwrap_or_else(|| "A11PRO052500527".to_string());
let jar = args
.next()
.unwrap_or_else(|| "/opt/homebrew/share/scrcpy/scrcpy-server".to_string());
println!("starting scrcpy session on {serial} (jar {jar})");
let mut session = StreamSession::start(&serial, &jar, 0).await?;
println!("device name : {}", session.device_name);
println!(
"codec : 0x{:08x} ({})",
session.codec_id,
if session.codec_id == CODEC_ID_H264 { "h264" } else { "?" }
);
let mut config_unit: Vec<u8> = Vec::new();
let mut first_keyframe: Vec<u8> = Vec::new();
let (mut n_config, mut n_key, mut n_other, mut bytes) = (0u32, 0u32, 0u32, 0usize);
// A static screen emits few frames, so cap the wait per packet rather than
// requiring a fixed count. Stop once we have the config unit + a keyframe.
for _ in 0..200 {
let pkt = match tokio::time::timeout(Duration::from_secs(3), session.next_packet()).await {
Ok(pkt) => pkt?,
Err(_) => {
println!("(no new frame for 3s — stopping read)");
break;
}
};
bytes += pkt.data.len();
if pkt.header.config {
n_config += 1;
if config_unit.is_empty() {
config_unit = pkt.data.clone();
}
} else if pkt.header.keyframe {
n_key += 1;
if first_keyframe.is_empty() {
first_keyframe = pkt.data.clone();
}
} else {
n_other += 1;
}
if n_config >= 1 && n_key >= 1 && n_other >= 3 {
break;
}
}
println!("video size : {}x{} (from session packet)", session.width(), session.height());
println!(
"packets : {n_config} config, {n_key} keyframe, {n_other} other ({bytes} bytes)"
);
println!(
"config unit : {} bytes (SPS/PPS), first keyframe: {} bytes",
config_unit.len(),
first_keyframe.len()
);
let video = VideoSize::new(session.width().max(1), session.height().max(1));
// Write config + first keyframe as a standalone Annex-B access unit to validate.
let mut au = config_unit.clone();
au.extend_from_slice(&first_keyframe);
std::fs::write("/tmp/au.h264", &au)?;
println!("wrote /tmp/au.h264 ({} bytes) — validate with ffprobe", au.len());
// Exercise the control socket: a center tap (down then up) via the input encoder.
println!("sending a center tap over the control socket…");
let down = touch_message(
&PointerEvent { action: MotionAction::Down, norm_x: 0.5, norm_y: 0.5 },
video,
);
session.send_control(&down).await?;
tokio::time::sleep(Duration::from_millis(60)).await;
let up = touch_message(
&PointerEvent { action: MotionAction::Up, norm_x: 0.5, norm_y: 0.5 },
video,
);
session.send_control(&up).await?;
// And a HOME key press to show key injection works.
session.send_control(&key_message(KeyAction::Down, 3, 0)).await?; // KEYCODE_HOME = 3
session.send_control(&key_message(KeyAction::Up, 3, 0)).await?;
println!("control messages sent without error");
session.shutdown().await?;
println!("session shut down cleanly");
Ok(())
}
+215
View File
@@ -0,0 +1,215 @@
//! streamd: the browser-facing edge of the single-device spike (§4.4 signaling).
//! Serves the WebRTC client, brokers SDP (offer → answer), runs a scrcpy session,
//! pumps its H.264 into the peer connection, and forwards browser input events to
//! the device's control socket. One viewer / one session — enough for the spike.
use std::net::SocketAddr;
use std::sync::atomic::{AtomicU32, Ordering};
use std::sync::Arc;
use anyhow::Result;
use axum::{
extract::{
ws::{Message, WebSocket, WebSocketUpgrade},
Query, State,
},
http::{header, StatusCode},
response::{Html, IntoResponse, Response},
routing::{get, post},
Json, Router,
};
use serde::Deserialize;
use serde_json::json;
use tokio::io::AsyncWriteExt;
use tokio::net::TcpStream;
use tokio::sync::Mutex;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
use stream_node::control::{KeyAction, MotionAction};
use stream_node::coords::VideoSize;
use stream_node::scrcpy::{ServerHandle, StreamSession};
use stream_node::webrtc_bridge::{pump_video, SharedSize, VideoBridge};
use stream_node::{key_message, touch_message, PointerEvent};
const PAGE: &str = include_str!("streamd_page.html");
struct AppState {
serial: String,
jar: String,
control: Mutex<Option<TcpStream>>,
size: SharedSize,
guard: Mutex<Option<ServerHandle>>,
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::registry()
.with(
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| "stream_node=info,info".into()),
)
.with(tracing_subscriber::fmt::layer())
.init();
let state = Arc::new(AppState {
serial: std::env::var("DEVICE_SERIAL").unwrap_or_else(|_| "A11PRO052500527".into()),
jar: std::env::var("SCRCPY_SERVER")
.unwrap_or_else(|_| "/opt/homebrew/share/scrcpy/scrcpy-server".into()),
control: Mutex::new(None),
size: Arc::new((AtomicU32::new(720), AtomicU32::new(1600))),
guard: Mutex::new(None),
});
let app = Router::new()
.route("/", get(|| async { Html(PAGE) }))
.route("/offer", post(offer))
.route("/input", get(input_ws))
.route("/thumbnail", get(thumbnail))
.with_state(state);
let addr: SocketAddr = std::env::var("BIND_ADDR")
.unwrap_or_else(|_| "0.0.0.0:8095".into())
.parse()?;
let listener = tokio::net::TcpListener::bind(addr).await?;
tracing::info!("streamd listening on http://{addr}");
axum::serve(listener, app).await?;
Ok(())
}
#[derive(Deserialize)]
struct OfferBody {
sdp: String,
}
#[derive(Deserialize)]
struct ThumbQuery {
serial: String,
}
/// Capture a still of a device's current screen via `adb exec-out screencap -p`.
/// Works whether or not the device is being streamed, so the device gallery can show
/// a live-ish thumbnail. The serial is passed as an argv (no shell), so it can't inject.
async fn thumbnail(Query(q): Query<ThumbQuery>) -> Response {
let out = tokio::process::Command::new("adb")
.args(["-s", &q.serial, "exec-out", "screencap", "-p"])
.output()
.await;
match out {
Ok(o) if o.status.success() && !o.stdout.is_empty() => (
[
(header::CONTENT_TYPE, "image/png"),
(header::CACHE_CONTROL, "no-store"),
],
o.stdout,
)
.into_response(),
Ok(o) => (
StatusCode::SERVICE_UNAVAILABLE,
format!("screencap failed: {}", String::from_utf8_lossy(&o.stderr)),
)
.into_response(),
Err(e) => (StatusCode::SERVICE_UNAVAILABLE, format!("adb error: {e}")).into_response(),
}
}
async fn offer(State(st): State<Arc<AppState>>, Json(body): Json<OfferBody>) -> impl IntoResponse {
match start_stream(&st, body.sdp).await {
Ok(answer) => Json(json!({ "sdp": answer })).into_response(),
Err(e) => {
tracing::error!(error = ?e, "failed to start stream");
(StatusCode::INTERNAL_SERVER_ERROR, format!("{e:#}")).into_response()
}
}
}
async fn start_stream(st: &Arc<AppState>, offer_sdp: String) -> Result<String> {
// Build the peer connection and answer first, so a bad offer never spins up a
// scrcpy session on the device.
let bridge = VideoBridge::new(&["stun:stun.l.google.com:19302".to_string()]).await?;
let answer = bridge.answer(offer_sdp).await?;
// Tear down any previous session before starting a new one.
if let Some(prev) = st.guard.lock().await.take() {
let _ = prev.shutdown().await;
}
let session = StreamSession::start(&st.serial, &st.jar, 0).await?;
let (mut video, control, guard) = session.split();
*st.control.lock().await = Some(control);
*st.guard.lock().await = Some(guard);
let track = bridge.track();
let size = st.size.clone();
tokio::spawn(async move {
// Hold the bridge so its peer connection outlives this scope.
let _bridge = bridge;
if let Err(e) = pump_video(&mut video, track, size).await {
tracing::warn!(error = ?e, "video pump ended");
}
});
Ok(answer)
}
async fn input_ws(State(st): State<Arc<AppState>>, ws: WebSocketUpgrade) -> impl IntoResponse {
ws.on_upgrade(move |socket| handle_input(st, socket))
}
#[derive(Deserialize)]
#[serde(tag = "kind", rename_all = "lowercase")]
enum InputEvent {
Touch { action: String, x: f64, y: f64 },
Key { action: String, keycode: u32 },
}
async fn handle_input(st: Arc<AppState>, mut socket: WebSocket) {
while let Some(Ok(msg)) = socket.recv().await {
let text = match msg {
Message::Text(t) => t,
Message::Close(_) => break,
_ => continue,
};
let Ok(event) = serde_json::from_str::<InputEvent>(&text) else {
continue;
};
let Some(bytes) = build_control_message(&st, &event) else {
continue;
};
let mut guard = st.control.lock().await;
if let Some(control) = guard.as_mut() {
let _ = control.write_all(&bytes).await;
}
}
}
fn build_control_message(st: &AppState, event: &InputEvent) -> Option<Vec<u8>> {
match event {
InputEvent::Touch { action, x, y } => {
let action = match action.as_str() {
"down" => MotionAction::Down,
"up" => MotionAction::Up,
"move" => MotionAction::Move,
_ => return None,
};
let w = st.size.0.load(Ordering::Relaxed).max(1);
let h = st.size.1.load(Ordering::Relaxed).max(1);
Some(touch_message(
&PointerEvent {
action,
norm_x: *x,
norm_y: *y,
},
VideoSize::new(w, h),
))
}
InputEvent::Key { action, keycode } => {
let action = match action.as_str() {
"down" => KeyAction::Down,
"up" => KeyAction::Up,
_ => return None,
};
Some(key_message(action, *keycode, 0))
}
}
}
+95
View File
@@ -0,0 +1,95 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>ClawReview — live device</title>
<style>
body { margin: 0; background: #09090b; color: #e4e4e7; font-family: system-ui, sans-serif;
display: flex; flex-direction: column; align-items: center; height: 100vh; }
#status { padding: 6px 12px; font-size: 13px; color: #a1a1aa; }
#stage { flex: 1; display: flex; align-items: center; gap: 12px; }
video { height: 88vh; background: #000; border-radius: 10px; touch-action: none; cursor: crosshair; }
.keys { display: flex; flex-direction: column; gap: 8px; }
button { background: #18181b; color: #e4e4e7; border: 1px solid #3f3f46; border-radius: 8px;
padding: 8px 12px; font-size: 13px; cursor: pointer; }
button:hover { background: #27272a; }
</style>
</head>
<body>
<div id="status">connecting…</div>
<div id="stage">
<video id="v" autoplay playsinline muted></video>
<div class="keys">
<button data-key="3">HOME</button>
<button data-key="4">BACK</button>
<button data-key="187">RECENTS</button>
</div>
</div>
<script>
const statusEl = document.getElementById('status');
const video = document.getElementById('v');
let ws, down = false;
function setStatus(t) { statusEl.textContent = t; }
function iceComplete(pc) {
return new Promise((res) => {
if (pc.iceGatheringState === 'complete') return res();
pc.addEventListener('icegatheringstatechange', () => {
if (pc.iceGatheringState === 'complete') res();
});
});
}
async function start() {
const pc = new RTCPeerConnection();
pc.addTransceiver('video', { direction: 'recvonly' });
pc.ontrack = (e) => { video.srcObject = e.streams[0]; setStatus('streaming'); };
pc.oniceconnectionstatechange = () => setStatus('ice: ' + pc.iceConnectionState);
await pc.setLocalDescription(await pc.createOffer());
await iceComplete(pc);
const res = await fetch('/offer', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ sdp: pc.localDescription.sdp }),
});
if (!res.ok) { setStatus('offer failed: ' + (await res.text())); return; }
const { sdp } = await res.json();
await pc.setRemoteDescription({ type: 'answer', sdp });
const proto = location.protocol === 'https:' ? 'wss' : 'ws';
ws = new WebSocket(`${proto}://${location.host}/input`);
wireInput();
}
function norm(e) {
const r = video.getBoundingClientRect();
return { x: (e.clientX - r.left) / r.width, y: (e.clientY - r.top) / r.height };
}
function sendTouch(action, e) {
if (!ws || ws.readyState !== 1) return;
const { x, y } = norm(e);
if (x < 0 || x > 1 || y < 0 || y > 1) return;
ws.send(JSON.stringify({ kind: 'touch', action, x, y }));
}
function sendKey(keycode) {
if (!ws || ws.readyState !== 1) return;
ws.send(JSON.stringify({ kind: 'key', action: 'down', keycode }));
ws.send(JSON.stringify({ kind: 'key', action: 'up', keycode }));
}
function wireInput() {
video.addEventListener('pointerdown', (e) => { down = true; video.setPointerCapture(e.pointerId); sendTouch('down', e); });
video.addEventListener('pointermove', (e) => { if (down) sendTouch('move', e); });
video.addEventListener('pointerup', (e) => { down = false; sendTouch('up', e); });
video.addEventListener('pointercancel', () => { down = false; });
document.querySelectorAll('button[data-key]').forEach((b) =>
b.addEventListener('click', () => sendKey(parseInt(b.dataset.key, 10))));
}
start().catch((e) => setStatus('error: ' + e));
</script>
</body>
</html>
+171
View File
@@ -0,0 +1,171 @@
//! scrcpy control-protocol message encoding (§4.3, §6.1). Byte layouts match the
//! scrcpy v4.0 server's `ControlMessageReader` (client serializer `control_msg.c`):
//! all multi-byte integers are big-endian.
/// Control message type tags, in the scrcpy v4.0 enum order.
#[repr(u8)]
#[derive(Clone, Copy)]
pub enum MsgType {
InjectKeycode = 0,
InjectTouchEvent = 2,
BackOrScreenOn = 4,
}
/// Android `KeyEvent` actions.
#[repr(u8)]
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum KeyAction {
Down = 0,
Up = 1,
}
/// Android `MotionEvent` actions (the subset the input path uses).
#[repr(u8)]
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum MotionAction {
Down = 0,
Up = 1,
Move = 2,
}
/// scrcpy's synthetic pointer ids (Android `int64`, so two's-complement of -1/-2).
pub const POINTER_ID_MOUSE: u64 = u64::MAX; // -1
pub const POINTER_ID_GENERIC_FINGER: u64 = u64::MAX - 1; // -2
/// scrcpy `sc_float_to_u16fp`: map a [0,1] float to a u16 fixed-point value.
fn float_to_u16fp(value: f32) -> u16 {
debug_assert!((0.0..=1.0).contains(&value));
let u = (value * 65536.0) as u32; // f * 2^16
if u >= 0xffff {
0xffff
} else {
u as u16
}
}
/// INJECT_TOUCH_EVENT — 32 bytes.
#[allow(clippy::too_many_arguments)]
pub fn encode_touch(
action: MotionAction,
pointer_id: u64,
x: i32,
y: i32,
screen_width: u16,
screen_height: u16,
pressure: f32,
action_button: u32,
buttons: u32,
) -> Vec<u8> {
let mut buf = Vec::with_capacity(32);
buf.push(MsgType::InjectTouchEvent as u8);
buf.push(action as u8);
buf.extend_from_slice(&pointer_id.to_be_bytes());
buf.extend_from_slice(&x.to_be_bytes());
buf.extend_from_slice(&y.to_be_bytes());
buf.extend_from_slice(&screen_width.to_be_bytes());
buf.extend_from_slice(&screen_height.to_be_bytes());
buf.extend_from_slice(&float_to_u16fp(pressure).to_be_bytes());
buf.extend_from_slice(&action_button.to_be_bytes());
buf.extend_from_slice(&buttons.to_be_bytes());
debug_assert_eq!(buf.len(), 32);
buf
}
/// INJECT_KEYCODE — 14 bytes.
pub fn encode_keycode(action: KeyAction, keycode: u32, repeat: u32, metastate: u32) -> Vec<u8> {
let mut buf = Vec::with_capacity(14);
buf.push(MsgType::InjectKeycode as u8);
buf.push(action as u8);
buf.extend_from_slice(&keycode.to_be_bytes());
buf.extend_from_slice(&repeat.to_be_bytes());
buf.extend_from_slice(&metastate.to_be_bytes());
debug_assert_eq!(buf.len(), 14);
buf
}
/// BACK_OR_SCREEN_ON — 2 bytes (carries a key action).
pub fn encode_back_or_screen_on(action: KeyAction) -> Vec<u8> {
vec![MsgType::BackOrScreenOn as u8, action as u8]
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn touch_down_center_exact_bytes() {
// 540,960 on a 1080x1920 frame, finger pointer, full pressure.
let bytes = encode_touch(
MotionAction::Down,
POINTER_ID_GENERIC_FINGER,
540,
960,
1080,
1920,
1.0,
0,
0,
);
assert_eq!(
bytes,
vec![
0x02, // type INJECT_TOUCH_EVENT
0x00, // action DOWN
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFE, // pointer_id = -2
0x00, 0x00, 0x02, 0x1C, // x = 540
0x00, 0x00, 0x03, 0xC0, // y = 960
0x04, 0x38, // width = 1080
0x07, 0x80, // height = 1920
0xFF, 0xFF, // pressure 1.0 -> 0xffff
0x00, 0x00, 0x00, 0x00, // action_button
0x00, 0x00, 0x00, 0x00, // buttons
]
);
}
#[test]
fn touch_up_has_zero_pressure_encoding() {
let bytes = encode_touch(
MotionAction::Up,
POINTER_ID_GENERIC_FINGER,
0,
0,
1080,
1920,
0.0,
0,
0,
);
assert_eq!(bytes[1], 0x01); // action UP
assert_eq!(&bytes[22..24], &[0x00, 0x00]); // pressure 0.0
assert_eq!(bytes.len(), 32);
}
#[test]
fn keycode_back_exact_bytes() {
// KEYCODE_BACK = 4.
let bytes = encode_keycode(KeyAction::Down, 4, 0, 0);
assert_eq!(
bytes,
vec![
0x00, // type INJECT_KEYCODE
0x00, // action DOWN
0x00, 0x00, 0x00, 0x04, // keycode 4
0x00, 0x00, 0x00, 0x00, // repeat
0x00, 0x00, 0x00, 0x00, // metastate
]
);
}
#[test]
fn back_or_screen_on_is_two_bytes() {
assert_eq!(encode_back_or_screen_on(KeyAction::Up), vec![0x04, 0x01]);
}
#[test]
fn pressure_clamps_at_one() {
assert_eq!(float_to_u16fp(1.0), 0xffff);
assert_eq!(float_to_u16fp(0.0), 0x0000);
assert_eq!(float_to_u16fp(0.5), 0x8000);
}
}
+78
View File
@@ -0,0 +1,78 @@
//! Coordinate mapping for the input back-channel (§4.3, §9 risk #2 — taps landing
//! in the wrong place). The browser sends *normalized* pointer positions in [0,1]
//! (the "normalized events" of §4.3); the stream node scales them against the
//! current video frame size. Normalizing on the wire — rather than baking in a
//! resolution on the frontend — means an orientation/size change can't make a
//! stale device size land taps in the wrong spot: the live frame size is authoritative.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct VideoSize {
pub width: u32,
pub height: u32,
}
impl VideoSize {
pub fn new(width: u32, height: u32) -> Self {
Self { width, height }
}
}
/// Map a normalized position to device pixels, clamped to `[0, width-1] × [0, height-1]`.
/// Out-of-range inputs are clamped rather than rejected so a pointer dragged off the
/// edge still produces an in-bounds coordinate.
pub fn to_device_pixels(norm_x: f64, norm_y: f64, video: VideoSize) -> (i32, i32) {
let x = scale(norm_x, video.width);
let y = scale(norm_y, video.height);
(x, y)
}
fn scale(norm: f64, extent: u32) -> i32 {
if extent == 0 {
return 0;
}
let clamped = norm.clamp(0.0, 1.0);
let max = (extent - 1) as f64;
(clamped * extent as f64).round().min(max) as i32
}
#[cfg(test)]
mod tests {
use super::*;
const FHD: VideoSize = VideoSize {
width: 1080,
height: 1920,
};
#[test]
fn center_maps_to_mid_pixel() {
assert_eq!(to_device_pixels(0.5, 0.5, FHD), (540, 960));
}
#[test]
fn top_left_corner() {
assert_eq!(to_device_pixels(0.0, 0.0, FHD), (0, 0));
}
#[test]
fn bottom_right_clamps_inside_bounds() {
// 1.0 would scale to width/height; must clamp to the last valid pixel.
assert_eq!(to_device_pixels(1.0, 1.0, FHD), (1079, 1919));
}
#[test]
fn out_of_range_is_clamped_not_rejected() {
assert_eq!(to_device_pixels(1.5, -0.3, FHD), (1079, 0));
}
#[test]
fn landscape_size_scales_independently() {
let land = VideoSize::new(2340, 1080);
assert_eq!(to_device_pixels(0.5, 0.5, land), (1170, 540));
}
#[test]
fn zero_size_is_safe() {
assert_eq!(to_device_pixels(0.5, 0.5, VideoSize::new(0, 0)), (0, 0));
}
}
+88
View File
@@ -0,0 +1,88 @@
//! ClawReview stream node (§6.1). This crate currently holds the device-independent
//! input back-channel: translating normalized browser pointer/key events into scrcpy
//! control-protocol messages. The scrcpy supervisor, H.264 reader, and the webrtc-rs
//! peer connection are added once a device is attached for the single-device spike.
pub mod control;
pub mod coords;
pub mod scrcpy;
pub mod webrtc_bridge;
use control::{encode_keycode, encode_touch, KeyAction, MotionAction, POINTER_ID_GENERIC_FINGER};
use coords::{to_device_pixels, VideoSize};
/// A normalized pointer event arriving from the browser over the input WebSocket
/// (§4.3). Coordinates are in [0,1] relative to the displayed video.
pub struct PointerEvent {
pub action: MotionAction,
pub norm_x: f64,
pub norm_y: f64,
}
/// Translate a normalized pointer event into a scrcpy touch control message for the
/// current live video frame size — the mirrored, backend half of the §4.3 scaling.
pub fn touch_message(event: &PointerEvent, video: VideoSize) -> Vec<u8> {
let (x, y) = to_device_pixels(event.norm_x, event.norm_y, video);
let pressure = if event.action == MotionAction::Up { 0.0 } else { 1.0 };
encode_touch(
event.action,
POINTER_ID_GENERIC_FINGER,
x,
y,
video.width as u16,
video.height as u16,
pressure,
0,
0,
)
}
/// Translate a key press/release into a scrcpy keycode control message.
pub fn key_message(action: KeyAction, keycode: u32, metastate: u32) -> Vec<u8> {
encode_keycode(action, keycode, 0, metastate)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn center_tap_down_matches_protocol_encoding() {
let event = PointerEvent {
action: MotionAction::Down,
norm_x: 0.5,
norm_y: 0.5,
};
let msg = touch_message(&event, VideoSize::new(1080, 1920));
let expected = encode_touch(
MotionAction::Down,
POINTER_ID_GENERIC_FINGER,
540,
960,
1080,
1920,
1.0,
0,
0,
);
assert_eq!(msg, expected);
assert_eq!(msg.len(), 32);
}
#[test]
fn up_event_uses_zero_pressure() {
let event = PointerEvent {
action: MotionAction::Up,
norm_x: 0.5,
norm_y: 0.5,
};
let msg = touch_message(&event, VideoSize::new(1080, 1920));
assert_eq!(&msg[22..24], &[0x00, 0x00]);
}
#[test]
fn key_message_sets_zero_repeat() {
let msg = key_message(KeyAction::Down, 4, 0);
assert_eq!(&msg[6..10], &[0, 0, 0, 0]); // repeat
}
}
+355
View File
@@ -0,0 +1,355 @@
//! scrcpy v4.0 server supervisor and video-stream demuxer (§4.2, §6.1).
//!
//! Lifecycle: push the server jar, `adb forward` a TCP port to the device's
//! `localabstract:scrcpy_<scid>`, launch the server via `app_process`, then connect
//! the video socket (first — carries a dummy byte, the 64-byte device name, and the
//! 12-byte codec header) followed by the control socket. Frames arrive as
//! `[u64 pts+flags][u32 len][payload]`.
use std::process::Stdio;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use anyhow::{bail, Context, Result};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::{TcpListener, TcpStream};
use tokio::process::{Child, Command};
pub const SERVER_DEVICE_PATH: &str = "/data/local/tmp/scrcpy-server.jar";
pub const SCRCPY_VERSION: &str = "4.0";
/// scrcpy codec id for H.264, the ASCII bytes "h264".
pub const CODEC_ID_H264: u32 = 0x6832_3634;
const DEVICE_NAME_LEN: usize = 64;
const CODEC_ID_SIZE: usize = 4;
const PACKET_HEADER_SIZE: usize = 12;
const SESSION_PACKET_FLAG: u8 = 0x80;
const FLAG_CONFIG: u64 = 1 << 62;
const FLAG_KEY_FRAME: u64 = 1 << 61;
const PTS_MASK: u64 = FLAG_KEY_FRAME - 1;
/// A 12-byte header with the MSB set is a "session" packet: video metadata
/// (width/height) with no payload, rather than a media frame.
pub fn is_session_packet(header: &[u8; PACKET_HEADER_SIZE]) -> bool {
header[0] & SESSION_PACKET_FLAG != 0
}
/// Width and height carried by a session packet (bytes 4..8 and 8..12).
pub fn parse_session_packet(header: &[u8; PACKET_HEADER_SIZE]) -> (u32, u32) {
let width = u32::from_be_bytes(header[4..8].try_into().unwrap());
let height = u32::from_be_bytes(header[8..12].try_into().unwrap());
(width, height)
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct PacketHeader {
pub config: bool,
pub keyframe: bool,
pub pts: u64,
pub len: u32,
}
pub fn parse_packet_header(buf: &[u8; PACKET_HEADER_SIZE]) -> PacketHeader {
let pts_flags = u64::from_be_bytes(buf[0..8].try_into().unwrap());
let len = u32::from_be_bytes(buf[8..12].try_into().unwrap());
PacketHeader {
config: pts_flags & FLAG_CONFIG != 0,
keyframe: pts_flags & FLAG_KEY_FRAME != 0,
pts: pts_flags & PTS_MASK,
len,
}
}
/// A live H.264 access unit read off the video socket.
pub struct VideoPacket {
pub header: PacketHeader,
pub data: Vec<u8>,
}
/// The video half of a session: reads H.264 media access units off the socket,
/// tracking the current frame dimensions from session packets.
pub struct VideoHalf {
stream: TcpStream,
/// Current video dimensions, set from session packets (0 until the first one).
pub width: u32,
pub height: u32,
}
impl VideoHalf {
/// Read the next H.264 media access unit. Session packets (size metadata) are
/// consumed internally (updating `width`/`height`) and skipped.
pub async fn next_packet(&mut self) -> Result<VideoPacket> {
loop {
let mut header = [0u8; PACKET_HEADER_SIZE];
self.stream.read_exact(&mut header).await.context("packet header")?;
if is_session_packet(&header) {
let (w, h) = parse_session_packet(&header);
self.width = w;
self.height = h;
continue;
}
let parsed = parse_packet_header(&header);
let mut data = vec![0u8; parsed.len as usize];
self.stream.read_exact(&mut data).await.context("packet payload")?;
return Ok(VideoPacket { header: parsed, data });
}
}
}
/// Handle to the server process and reverse tunnel, kept for clean teardown after
/// a session is split into its independently-owned video and control halves.
pub struct ServerHandle {
serial: String,
socket_name: String,
server: Child,
}
impl ServerHandle {
pub async fn shutdown(mut self) -> Result<()> {
let _ = self.server.kill().await;
let _ = adb(&[
"-s",
&self.serial,
"reverse",
"--remove",
&format!("localabstract:{}", self.socket_name),
])
.await;
Ok(())
}
}
/// A running scrcpy session: the server process plus the connected video and
/// control sockets, with the negotiated device name and codec metadata.
pub struct StreamSession {
pub serial: String,
pub scid: u32,
pub socket_name: String,
pub device_name: String,
pub codec_id: u32,
pub video: VideoHalf,
pub control: TcpStream,
server: Child,
}
impl StreamSession {
/// Push the server, launch it, and connect the sockets for a video+control
/// session (audio disabled — video-only per v1). Uses a reverse tunnel exactly
/// like the scrcpy client: the host listens, the server connects out (video
/// socket first, then control). No dummy byte in reverse mode.
pub async fn start(serial: &str, server_jar: &str, max_size: u32) -> Result<Self> {
push_server(serial, server_jar).await?;
let scid = new_scid();
let socket_name = format!("scrcpy_{scid:08x}");
let listener = TcpListener::bind("127.0.0.1:0")
.await
.context("bind reverse-tunnel listener")?;
let port = listener.local_addr()?.port();
adb(&[
"-s",
serial,
"reverse",
&format!("localabstract:{socket_name}"),
&format!("tcp:{port}"),
])
.await
.context("adb reverse")?;
// Match the scrcpy client's minimal invocation; extra args abort the server.
let mut server_args: Vec<String> = vec![
"-s".into(),
serial.into(),
"shell".into(),
format!("CLASSPATH={SERVER_DEVICE_PATH}"),
"app_process".into(),
"/".into(),
"com.genymobile.scrcpy.Server".into(),
SCRCPY_VERSION.into(),
format!("scid={scid:08x}"),
"log_level=info".into(),
"audio=false".into(),
];
if max_size > 0 {
server_args.push(format!("max_size={max_size}"));
}
let server = Command::new("adb")
.args(&server_args)
.stdout(Stdio::inherit())
.stderr(Stdio::inherit())
.spawn()
.context("spawn scrcpy server")?;
// The server connects the video socket first.
let mut video = accept(&listener).await.context("accept video socket")?;
let mut name_buf = [0u8; DEVICE_NAME_LEN];
video.read_exact(&mut name_buf).await.context("read device name")?;
let device_name = String::from_utf8_lossy(&name_buf)
.trim_end_matches('\0')
.trim()
.to_string();
let mut codec_buf = [0u8; CODEC_ID_SIZE];
video.read_exact(&mut codec_buf).await.context("read codec id")?;
let codec_id = u32::from_be_bytes(codec_buf);
// The control socket connects second.
let control = accept(&listener).await.context("accept control socket")?;
Ok(Self {
serial: serial.to_string(),
scid,
socket_name,
device_name,
codec_id,
video: VideoHalf {
stream: video,
width: 0,
height: 0,
},
control,
server,
})
}
/// Read the next H.264 media access unit from the video socket.
pub async fn next_packet(&mut self) -> Result<VideoPacket> {
self.video.next_packet().await
}
pub fn width(&self) -> u32 {
self.video.width
}
pub fn height(&self) -> u32 {
self.video.height
}
/// Send a scrcpy control message (from `crate::control`) to the device.
pub async fn send_control(&mut self, message: &[u8]) -> Result<()> {
self.control.write_all(message).await.context("write control")?;
Ok(())
}
/// Split into independently-owned halves so the video pump and the input
/// handler can run concurrently, plus a handle for teardown.
pub fn split(self) -> (VideoHalf, TcpStream, ServerHandle) {
(
self.video,
self.control,
ServerHandle {
serial: self.serial,
socket_name: self.socket_name,
server: self.server,
},
)
}
/// Stop the server and remove the reverse tunnel.
pub async fn shutdown(self) -> Result<()> {
let (_video, _control, handle) = self.split();
handle.shutdown().await
}
}
/// Accept the next inbound socket from the server, with a startup timeout.
async fn accept(listener: &TcpListener) -> Result<TcpStream> {
let (stream, _) = tokio::time::timeout(Duration::from_secs(10), listener.accept())
.await
.context("timed out waiting for scrcpy server to connect")??;
stream.set_nodelay(true).ok();
Ok(stream)
}
async fn push_server(serial: &str, server_jar: &str) -> Result<()> {
adb(&["-s", serial, "push", server_jar, SERVER_DEVICE_PATH])
.await
.context("adb push scrcpy-server")?;
Ok(())
}
async fn adb(args: &[&str]) -> Result<()> {
let status = Command::new("adb")
.args(args)
.stdout(Stdio::null())
.stderr(Stdio::inherit())
.status()
.await
.context("running adb")?;
if !status.success() {
bail!("adb {:?} failed with {status}", args);
}
Ok(())
}
fn new_scid() -> u32 {
let nanos = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.subsec_nanos())
.unwrap_or(1);
(nanos & 0x7fff_ffff).max(1)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn session_packet_carries_dimensions_and_is_flagged() {
let mut buf = [0u8; 12];
buf[0] = 0x80; // session-packet flag (MSB set)
buf[4..8].copy_from_slice(&720u32.to_be_bytes());
buf[8..12].copy_from_slice(&1600u32.to_be_bytes());
assert!(is_session_packet(&buf));
assert_eq!(parse_session_packet(&buf), (720, 1600));
}
#[test]
fn media_packet_is_not_a_session_packet() {
let mut buf = [0u8; 12];
buf[0..8].copy_from_slice(&(1u64 << 61).to_be_bytes()); // keyframe, MSB clear
assert!(!is_session_packet(&buf));
}
#[test]
fn config_packet_flag_is_bit_62() {
let mut buf = [0u8; 12];
let pts_flags: u64 = 1 << 62; // CONFIG, pts 0
buf[0..8].copy_from_slice(&pts_flags.to_be_bytes());
buf[8..12].copy_from_slice(&37u32.to_be_bytes());
let h = parse_packet_header(&buf);
assert!(h.config);
assert!(!h.keyframe);
assert_eq!(h.len, 37);
assert_eq!(h.pts, 0);
}
#[test]
fn keyframe_flag_is_bit_61_and_pts_masked() {
let mut buf = [0u8; 12];
let pts: u64 = 123_456;
let pts_flags: u64 = (1 << 61) | pts; // KEY_FRAME + pts
buf[0..8].copy_from_slice(&pts_flags.to_be_bytes());
buf[8..12].copy_from_slice(&9000u32.to_be_bytes());
let h = parse_packet_header(&buf);
assert!(h.keyframe);
assert!(!h.config);
assert_eq!(h.pts, pts);
assert_eq!(h.len, 9000);
}
#[test]
fn plain_frame_has_no_flags() {
let mut buf = [0u8; 12];
buf[0..8].copy_from_slice(&500u64.to_be_bytes());
buf[8..12].copy_from_slice(&1234u32.to_be_bytes());
let h = parse_packet_header(&buf);
assert!(!h.config && !h.keyframe);
assert_eq!(h.pts, 500);
}
}
+155
View File
@@ -0,0 +1,155 @@
//! webrtc-rs bridge (§4.2, §4.6): owns a peer connection with a single H.264 video
//! track and feeds it the scrcpy access units. SDP is exchanged vanilla-ICE (gather
//! to completion, then hand over the full description) — simplest for the spike;
//! trickle/TURN come with the relay path (Phase 3).
use std::sync::atomic::{AtomicU32, Ordering};
use std::sync::Arc;
use std::time::Duration;
use anyhow::{Context, Result};
use bytes::Bytes;
use webrtc::api::interceptor_registry::register_default_interceptors;
use webrtc::api::media_engine::{MediaEngine, MIME_TYPE_H264};
use webrtc::api::APIBuilder;
use webrtc::ice_transport::ice_server::RTCIceServer;
use webrtc::interceptor::registry::Registry;
use webrtc::media::Sample;
use webrtc::peer_connection::configuration::RTCConfiguration;
use webrtc::peer_connection::sdp::session_description::RTCSessionDescription;
use webrtc::peer_connection::RTCPeerConnection;
use webrtc::rtp_transceiver::rtp_codec::RTCRtpCodecCapability;
use webrtc::track::track_local::track_local_static_sample::TrackLocalStaticSample;
use crate::scrcpy::VideoHalf;
/// Shared current video dimensions (width, height), published by the video pump so
/// the input handler can scale normalized coordinates against the live frame size.
pub type SharedSize = Arc<(AtomicU32, AtomicU32)>;
pub struct VideoBridge {
pub pc: Arc<RTCPeerConnection>,
track: Arc<TrackLocalStaticSample>,
}
impl VideoBridge {
/// Build a peer connection with a single sendonly H.264 track.
pub async fn new(stun_urls: &[String]) -> Result<Self> {
let mut media = MediaEngine::default();
media.register_default_codecs().context("register codecs")?;
let mut registry = Registry::new();
registry = register_default_interceptors(registry, &mut media)
.context("register interceptors")?;
let api = APIBuilder::new()
.with_media_engine(media)
.with_interceptor_registry(registry)
.build();
let config = RTCConfiguration {
ice_servers: if stun_urls.is_empty() {
vec![]
} else {
vec![RTCIceServer {
urls: stun_urls.to_vec(),
..Default::default()
}]
},
..Default::default()
};
let pc = Arc::new(api.new_peer_connection(config).await.context("new peer connection")?);
let track = Arc::new(TrackLocalStaticSample::new(
RTCRtpCodecCapability {
mime_type: MIME_TYPE_H264.to_owned(),
..Default::default()
},
"video".to_owned(),
"scrcpy".to_owned(),
));
pc.add_track(track.clone()).await.context("add video track")?;
Ok(Self { pc, track })
}
/// Accept a remote offer and produce an SDP answer (waiting for ICE gathering).
pub async fn answer(&self, offer_sdp: String) -> Result<String> {
let offer = RTCSessionDescription::offer(offer_sdp).context("parse offer")?;
self.pc.set_remote_description(offer).await.context("set remote")?;
let answer = self.pc.create_answer(None).await.context("create answer")?;
let mut gather_complete = self.pc.gathering_complete_promise().await;
self.pc.set_local_description(answer).await.context("set local")?;
let _ = gather_complete.recv().await;
let local = self.pc.local_description().await.context("no local description")?;
Ok(local.sdp)
}
pub fn track(&self) -> Arc<TrackLocalStaticSample> {
self.track.clone()
}
/// Write one H.264 access unit (Annex-B) to the track; webrtc-rs packetizes it.
pub async fn write_access_unit(&self, data: Bytes, duration: Duration) -> Result<()> {
write_access_unit(&self.track, data, duration).await
}
}
async fn write_access_unit(
track: &TrackLocalStaticSample,
data: Bytes,
duration: Duration,
) -> Result<()> {
track
.write_sample(&Sample {
data,
duration,
..Default::default()
})
.await
.context("write sample")?;
Ok(())
}
/// Pump scrcpy access units into the track until the video socket closes. SPS/PPS
/// (the config packet) is cached and prepended to each keyframe so a freshly-joined
/// decoder always has parameter sets. Frame duration comes from the pts delta.
pub async fn pump_video(
video: &mut VideoHalf,
track: Arc<TrackLocalStaticSample>,
size: SharedSize,
) -> Result<()> {
let mut config: Vec<u8> = Vec::new();
let mut last_pts: Option<u64> = None;
loop {
let pkt = video.next_packet().await?;
size.0.store(video.width, Ordering::Relaxed);
size.1.store(video.height, Ordering::Relaxed);
if pkt.header.config {
config = pkt.data;
continue;
}
let duration = match last_pts {
Some(prev) if pkt.header.pts > prev => Duration::from_micros(pkt.header.pts - prev),
_ => Duration::from_millis(33),
};
last_pts = Some(pkt.header.pts);
let data = if pkt.header.keyframe && !config.is_empty() {
let mut unit = Vec::with_capacity(config.len() + pkt.data.len());
unit.extend_from_slice(&config);
unit.extend_from_slice(&pkt.data);
Bytes::from(unit)
} else {
Bytes::from(pkt.data)
};
write_access_unit(&track, data, duration).await?;
}
}
+100
View File
@@ -0,0 +1,100 @@
//! End-to-end test of the full phone→browser path: spawn streamd, act as the
//! browser (webrtc-rs receiver), POST a real offer, and confirm actual RTP video
//! arrives from the device. Ignored by default — needs a device attached and adb.
//! cargo test -p stream-node --test streamd_e2e -- --ignored
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::mpsc;
use webrtc::api::interceptor_registry::register_default_interceptors;
use webrtc::api::media_engine::MediaEngine;
use webrtc::api::APIBuilder;
use webrtc::interceptor::registry::Registry;
use webrtc::peer_connection::configuration::RTCConfiguration;
use webrtc::peer_connection::sdp::session_description::RTCSessionDescription;
use webrtc::rtp_transceiver::rtp_codec::RTPCodecType;
const BASE: &str = "http://127.0.0.1:18096";
#[tokio::test]
#[ignore = "needs a device + adb; spawns streamd"]
async fn streamd_streams_real_device_video() {
let serial = std::env::var("DEVICE_SERIAL").unwrap_or_else(|_| "A11PRO052500527".into());
let mut streamd = tokio::process::Command::new(env!("CARGO_BIN_EXE_streamd"))
.env("BIND_ADDR", "127.0.0.1:18096")
.env("DEVICE_SERIAL", serial)
.kill_on_drop(true)
.spawn()
.expect("spawn streamd");
wait_until_up().await;
// Build a receiver peer (the "browser").
let mut media = MediaEngine::default();
media.register_default_codecs().unwrap();
let mut registry = Registry::new();
registry = register_default_interceptors(registry, &mut media).unwrap();
let api = APIBuilder::new()
.with_media_engine(media)
.with_interceptor_registry(registry)
.build();
let pc = Arc::new(api.new_peer_connection(RTCConfiguration::default()).await.unwrap());
pc.add_transceiver_from_kind(RTPCodecType::Video, None).await.unwrap();
let (tx, mut rx) = mpsc::channel::<usize>(4);
pc.on_track(Box::new(move |track, _, _| {
let tx = tx.clone();
Box::pin(async move {
tokio::spawn(async move {
while let Ok((pkt, _)) = track.read_rtp().await {
if !pkt.payload.is_empty() {
let _ = tx.send(pkt.payload.len()).await;
break;
}
}
});
})
}));
// Offer (vanilla ICE) → POST to streamd → answer.
pc.set_local_description(pc.create_offer(None).await.unwrap()).await.unwrap();
let mut gather = pc.gathering_complete_promise().await;
let _ = gather.recv().await;
let offer_sdp = pc.local_description().await.unwrap().sdp;
let resp: serde_json::Value = reqwest::Client::new()
.post(format!("{BASE}/offer"))
.json(&serde_json::json!({ "sdp": offer_sdp }))
.send()
.await
.expect("POST /offer")
.json()
.await
.expect("answer json");
let answer_sdp = resp["sdp"].as_str().expect("answer sdp").to_string();
pc.set_remote_description(RTCSessionDescription::answer(answer_sdp).unwrap())
.await
.unwrap();
let got = tokio::time::timeout(Duration::from_secs(20), rx.recv()).await;
let _ = pc.close().await;
let _ = streamd.kill().await;
assert!(
matches!(got, Ok(Some(n)) if n > 0),
"no RTP video arrived from the device through streamd: {got:?}"
);
}
async fn wait_until_up() {
let client = reqwest::Client::new();
for _ in 0..100 {
if client.get(format!("{BASE}/")).send().await.map(|r| r.status().is_success()).unwrap_or(false) {
return;
}
tokio::time::sleep(Duration::from_millis(100)).await;
}
panic!("streamd did not come up");
}
+93
View File
@@ -0,0 +1,93 @@
//! Headless verification of the webrtc-rs media path: the VideoBridge (sender) and a
//! second webrtc-rs peer (receiver) negotiate in-process, the bridge writes H.264
//! samples, and we confirm RTP actually arrives on the receiver's track — no browser.
use std::sync::Arc;
use std::time::Duration;
use bytes::Bytes;
use stream_node::webrtc_bridge::VideoBridge;
use tokio::sync::mpsc;
use webrtc::api::interceptor_registry::register_default_interceptors;
use webrtc::api::media_engine::MediaEngine;
use webrtc::api::APIBuilder;
use webrtc::media::Sample;
use webrtc::peer_connection::configuration::RTCConfiguration;
use webrtc::peer_connection::sdp::session_description::RTCSessionDescription;
use webrtc::rtp_transceiver::rtp_codec::RTPCodecType;
#[tokio::test]
async fn h264_track_flows_to_a_receiver() {
// Sender: our bridge with a sendonly H.264 track.
let sender = VideoBridge::new(&[]).await.unwrap();
// Receiver: a plain webrtc-rs peer that offers a recvonly video transceiver.
let mut media = MediaEngine::default();
media.register_default_codecs().unwrap();
let mut registry = webrtc::interceptor::registry::Registry::new();
registry = register_default_interceptors(registry, &mut media).unwrap();
let api = APIBuilder::new()
.with_media_engine(media)
.with_interceptor_registry(registry)
.build();
let receiver = Arc::new(api.new_peer_connection(RTCConfiguration::default()).await.unwrap());
receiver
.add_transceiver_from_kind(RTPCodecType::Video, None)
.await
.unwrap();
// Signal "first RTP payload arrived" out of the on_track handler.
let (tx, mut rx) = mpsc::channel::<usize>(4);
receiver.on_track(Box::new(move |track, _receiver, _transceiver| {
let tx = tx.clone();
Box::pin(async move {
tokio::spawn(async move {
while let Ok((pkt, _)) = track.read_rtp().await {
if !pkt.payload.is_empty() {
let _ = tx.send(pkt.payload.len()).await;
break;
}
}
});
})
}));
// Vanilla-ICE signaling: receiver offers, sender answers.
let offer = receiver.create_offer(None).await.unwrap();
let mut gather = receiver.gathering_complete_promise().await;
receiver.set_local_description(offer).await.unwrap();
let _ = gather.recv().await;
let offer_sdp = receiver.local_description().await.unwrap().sdp;
let answer_sdp = sender.answer(offer_sdp).await.unwrap();
receiver
.set_remote_description(RTCSessionDescription::answer(answer_sdp).unwrap())
.await
.unwrap();
// Feed synthetic Annex-B NALs until the receiver reports a payload (content is
// irrelevant for verifying the transport — the H.264 payloader just splits NALs).
let track = sender.track();
let feeder = tokio::spawn(async move {
for _ in 0..400 {
let nal = Bytes::from_static(&[0, 0, 0, 1, 0x65, 0x88, 0x84, 0x21, 0x42, 0x63]);
let _ = track
.write_sample(&Sample {
data: nal,
duration: Duration::from_millis(33),
..Default::default()
})
.await;
tokio::time::sleep(Duration::from_millis(20)).await;
}
});
let got = tokio::time::timeout(Duration::from_secs(15), rx.recv()).await;
feeder.abort();
let _ = receiver.close().await;
assert!(
matches!(got, Ok(Some(n)) if n > 0),
"receiver never got an RTP payload over the negotiated H.264 track: {got:?}"
);
}
+1
View File
@@ -0,0 +1 @@
models/
+18
View File
@@ -0,0 +1,18 @@
[package]
name = "transcription-svc"
version = "0.1.0"
edition = "2021"
[dependencies]
axum = "0.7"
tokio = { version = "1", features = ["full"] }
async-trait = "0.1"
anyhow = "1"
serde_json = "1"
uuid = { version = "1", features = ["v4"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
whisper-rs = "0.14"
[dev-dependencies]
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json"] }
+18
View File
@@ -0,0 +1,18 @@
#!/usr/bin/env bash
# Fetch a ggml Whisper model for the transcription service.
# Usage: ./scripts/fetch-model.sh [model-name] (default: tiny.en)
set -euo pipefail
MODEL="${1:-tiny.en}"
DIR="$(cd "$(dirname "$0")/.." && pwd)/models"
OUT="$DIR/ggml-${MODEL}.bin"
URL="https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-${MODEL}.bin"
mkdir -p "$DIR"
if [ -f "$OUT" ]; then
echo "model already present: $OUT"
exit 0
fi
echo "downloading $URL"
curl -fL --retry 3 -o "$OUT" "$URL"
echo "saved $OUT"
+63
View File
@@ -0,0 +1,63 @@
//! Self-hosted Whisper-class transcription service (§6.4). A separate service from
//! the review API so transcription load never touches the real-time path, and on
//! RedClaw infrastructure so partner findings never leave RedClaw control.
pub mod whisper;
use std::sync::Arc;
use async_trait::async_trait;
use axum::{
body::Bytes,
extract::State,
http::{HeaderMap, StatusCode},
routing::{get, post},
Json, Router,
};
use serde_json::{json, Value};
/// Audio → text. Injected so the HTTP layer is testable without a model, and so
/// the real engine (whisper.cpp) can be swapped without touching the service.
#[async_trait]
pub trait Transcriber: Send + Sync {
async fn transcribe(&self, audio: &[u8], content_type: &str) -> anyhow::Result<String>;
}
#[derive(Clone)]
pub struct AppState {
pub transcriber: Arc<dyn Transcriber>,
}
pub fn build_router(state: AppState) -> Router {
Router::new()
.route("/health", get(health))
.route("/transcribe", post(transcribe))
.with_state(state)
}
async fn health() -> Json<Value> {
Json(json!({ "status": "ok" }))
}
/// Accepts a raw audio body (any container ffmpeg can read) and returns `{ "text": ... }`.
async fn transcribe(
State(st): State<AppState>,
headers: HeaderMap,
body: Bytes,
) -> Result<Json<Value>, (StatusCode, String)> {
if body.is_empty() {
return Err((StatusCode::BAD_REQUEST, "empty audio body".to_string()));
}
let content_type = headers
.get("content-type")
.and_then(|v| v.to_str().ok())
.unwrap_or("application/octet-stream");
match st.transcriber.transcribe(&body, content_type).await {
Ok(text) => Ok(Json(json!({ "text": text }))),
Err(e) => Err((
StatusCode::INTERNAL_SERVER_ERROR,
format!("transcription failed: {e}"),
)),
}
}
+30
View File
@@ -0,0 +1,30 @@
use std::sync::Arc;
use anyhow::Context;
use transcription_svc::{build_router, whisper::WhisperTranscriber, AppState};
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
tracing_subscriber::registry()
.with(
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| "transcription_svc=info,info".into()),
)
.with(tracing_subscriber::fmt::layer())
.init();
let model = std::env::var("WHISPER_MODEL").unwrap_or_else(|_| "models/ggml-tiny.bin".into());
let transcriber = Arc::new(
WhisperTranscriber::new(&model)
.with_context(|| format!("initializing whisper from {model}"))?,
);
let app = build_router(AppState { transcriber });
let addr = std::env::var("BIND_ADDR").unwrap_or_else(|_| "0.0.0.0:8099".to_string());
let listener = tokio::net::TcpListener::bind(&addr).await?;
tracing::info!("transcription-svc listening on http://{addr} (model {model})");
axum::serve(listener, app).await?;
Ok(())
}
+92
View File
@@ -0,0 +1,92 @@
//! Real transcription engine: ffmpeg normalizes any input container to 16 kHz mono
//! f32 PCM, then whisper.cpp (via `whisper-rs`) runs inference. This is the path the
//! deployed service uses; `WHISPER_MODEL` points at a ggml model file.
use std::sync::Arc;
use async_trait::async_trait;
use anyhow::{bail, Context};
use whisper_rs::{FullParams, SamplingStrategy, WhisperContext, WhisperContextParameters};
use crate::Transcriber;
pub struct WhisperTranscriber {
ctx: Arc<WhisperContext>,
}
impl WhisperTranscriber {
pub fn new(model_path: &str) -> anyhow::Result<Self> {
let ctx = WhisperContext::new_with_params(model_path, WhisperContextParameters::default())
.with_context(|| format!("loading whisper model from {model_path}"))?;
Ok(Self { ctx: Arc::new(ctx) })
}
}
#[async_trait]
impl Transcriber for WhisperTranscriber {
async fn transcribe(&self, audio: &[u8], _content_type: &str) -> anyhow::Result<String> {
let samples = decode_to_pcm_f32_16k_mono(audio).await?;
if samples.is_empty() {
bail!("decoded audio is empty");
}
// Inference is CPU-bound and blocking; move it off the async runtime.
let ctx = self.ctx.clone();
let text = tokio::task::spawn_blocking(move || run_whisper(&ctx, &samples)).await??;
Ok(text)
}
}
fn run_whisper(ctx: &WhisperContext, samples: &[f32]) -> anyhow::Result<String> {
let mut state = ctx.create_state().context("creating whisper state")?;
let mut params = FullParams::new(SamplingStrategy::Greedy { best_of: 1 });
let threads = std::thread::available_parallelism()
.map(|n| n.get() as i32)
.unwrap_or(4);
params.set_n_threads(threads);
params.set_print_special(false);
params.set_print_progress(false);
params.set_print_realtime(false);
params.set_print_timestamps(false);
state.full(params, samples).context("whisper inference")?;
let n = state.full_n_segments().context("counting segments")?;
let mut text = String::new();
for i in 0..n {
text.push_str(&state.full_get_segment_text(i).context("reading segment")?);
}
Ok(text.trim().to_string())
}
/// Decode arbitrary audio bytes to 16 kHz mono f32 PCM via ffmpeg.
async fn decode_to_pcm_f32_16k_mono(audio: &[u8]) -> anyhow::Result<Vec<f32>> {
let input = std::env::temp_dir().join(format!("clawreview-asr-{}", uuid::Uuid::new_v4()));
tokio::fs::write(&input, audio)
.await
.context("writing temp audio")?;
let result = tokio::process::Command::new("ffmpeg")
.args(["-nostdin", "-loglevel", "error", "-i"])
.arg(&input)
.args(["-f", "f32le", "-ac", "1", "-ar", "16000", "pipe:1"])
.output()
.await
.context("spawning ffmpeg (is it installed?)")?;
tokio::fs::remove_file(&input).await.ok();
if !result.status.success() {
bail!(
"ffmpeg decode failed: {}",
String::from_utf8_lossy(&result.stderr)
);
}
Ok(result
.stdout
.chunks_exact(4)
.map(|b| f32::from_le_bytes([b[0], b[1], b[2], b[3]]))
.collect())
}
Binary file not shown.
+69
View File
@@ -0,0 +1,69 @@
//! HTTP-contract tests for the transcription service, using a fake transcriber so
//! they run fast with no model. The real whisper path is covered in `whisper.rs`.
use std::sync::Arc;
use async_trait::async_trait;
use serde_json::Value;
use transcription_svc::{build_router, AppState, Transcriber};
struct FakeTranscriber {
reply: String,
}
#[async_trait]
impl Transcriber for FakeTranscriber {
async fn transcribe(&self, audio: &[u8], _content_type: &str) -> anyhow::Result<String> {
assert!(!audio.is_empty(), "handler must not call transcriber on empty body");
Ok(self.reply.clone())
}
}
async fn spawn(reply: &str) -> (String, reqwest::Client) {
let state = AppState {
transcriber: Arc::new(FakeTranscriber {
reply: reply.to_string(),
}),
};
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, build_router(state)).await.unwrap();
});
(format!("http://{addr}"), reqwest::Client::new())
}
#[tokio::test]
async fn transcribe_returns_text_json() {
let (base, client) = spawn("nfc read resolved quickly").await;
let res = client
.post(format!("{base}/transcribe"))
.header("content-type", "audio/webm")
.body(b"-some-audio-".to_vec())
.send()
.await
.unwrap();
assert_eq!(res.status(), 200);
let body: Value = res.json().await.unwrap();
assert_eq!(body["text"], "nfc read resolved quickly");
}
#[tokio::test]
async fn empty_body_is_400() {
let (base, client) = spawn("unused").await;
let res = client
.post(format!("{base}/transcribe"))
.header("content-type", "audio/webm")
.body(Vec::<u8>::new())
.send()
.await
.unwrap();
assert_eq!(res.status(), 400);
}
#[tokio::test]
async fn health_ok() {
let (base, client) = spawn("unused").await;
let res = client.get(format!("{base}/health")).send().await.unwrap();
assert_eq!(res.status(), 200);
}
+38
View File
@@ -0,0 +1,38 @@
//! End-to-end test of the real whisper.cpp engine: a recorded speech fixture is
//! decoded by ffmpeg and transcribed. Ignored by default because it needs a
//! downloaded model (`scripts/fetch-model.sh`) and ffmpeg on PATH; run with:
//! cargo test -p transcription-svc --test whisper -- --ignored
use std::path::PathBuf;
use transcription_svc::{whisper::WhisperTranscriber, Transcriber};
fn manifest(rel: &str) -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR")).join(rel)
}
#[tokio::test]
#[ignore = "needs a downloaded model and ffmpeg"]
async fn transcribes_real_speech() {
let model = std::env::var("WHISPER_MODEL")
.map(PathBuf::from)
.unwrap_or_else(|_| manifest("models/ggml-tiny.en.bin"));
assert!(
model.exists(),
"model not found at {model:?} — run scripts/fetch-model.sh"
);
let transcriber = WhisperTranscriber::new(model.to_str().unwrap()).expect("load model");
let audio = std::fs::read(manifest("tests/fixtures/speech.wav")).expect("read fixture");
let text = transcriber
.transcribe(&audio, "audio/wav")
.await
.expect("transcribe")
.to_lowercase();
assert!(
text.contains("fox"),
"expected 'fox' in transcript, got: {text:?}"
);
}