seeding
This commit is contained in:
@@ -0,0 +1,8 @@
|
|||||||
|
/target
|
||||||
|
**/target
|
||||||
|
node_modules
|
||||||
|
dist
|
||||||
|
.env
|
||||||
|
*.log
|
||||||
|
/data
|
||||||
|
|
||||||
Generated
+4139
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,3 @@
|
|||||||
|
[workspace]
|
||||||
|
members = ["review-api", "transcription-svc", "stream-node"]
|
||||||
|
resolver = "2"
|
||||||
@@ -0,0 +1,80 @@
|
|||||||
|
# ClawReview (`mobifeedback`)
|
||||||
|
|
||||||
|
A pure-Rust real-device review platform: stream live Android handsets into the browser,
|
||||||
|
with side-by-side notes and voice-transcribed findings, for engineers and partners to
|
||||||
|
review RedClaw apps.
|
||||||
|
|
||||||
|
See [`SPEC.md`](./SPEC.md) for the full spec, PRD, and roadmap (Draft v1.1).
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
**Phase 0, Track 1 (review surface against a placeholder video source) — complete.**
|
||||||
|
The right-hand product and the full backend persistence path work with zero streaming
|
||||||
|
code: claim/release, debounced-autosave typed notes, voice capture with real
|
||||||
|
transcription, one-tap frame capture attached to findings, and the anchoring chain
|
||||||
|
(finding → session → device → build).
|
||||||
|
|
||||||
|
**Phase 0, Track 2 (single-device WebRTC spike) — in progress.**
|
||||||
|
- Input back-channel: scrcpy control-message encoding (touch/keycode) + normalized→device
|
||||||
|
coordinate mapping, tested against the exact scrcpy v4.0 byte layout (§4.3, §9 risk #2).
|
||||||
|
- scrcpy supervisor + H.264 demuxer (`scrcpy.rs`): pushes the server, reverse-tunnels, reads
|
||||||
|
live H.264 access units, injects taps/keys. **Validated on a real device** (LAGENIO A11
|
||||||
|
Pro) — the dumped config+keyframe decodes as valid Baseline H.264. Run:
|
||||||
|
`cargo run -p stream-node --bin scrcpy-probe -- <serial>`.
|
||||||
|
- webrtc-rs bridge (`webrtc_bridge.rs`): peer connection + H.264 track + SDP answer + access-unit
|
||||||
|
pump. Pinned to **webrtc 0.11** (the 0.20 line is an unusable sans-IO alpha — §9 #5).
|
||||||
|
- `streamd` (`src/bin/streamd.rs`): serves a WebRTC client page, brokers SDP (offer→answer), runs a
|
||||||
|
scrcpy session, pumps its H.264 into the peer connection, and forwards browser pointer/key events
|
||||||
|
to the device control socket. **The full phone→browser path is verified end to end** by
|
||||||
|
`tests/streamd_e2e.rs` (real RTP video from the device through streamd to a webrtc-rs receiver).
|
||||||
|
|
||||||
|
Run it: `cargo run -p stream-node --bin streamd` then open `http://<host>:8095/` and drive the phone.
|
||||||
|
|
||||||
|
**Phase 0 Track 2 (single-device WebRTC spike) is functionally complete** — the spike's goal (a real
|
||||||
|
device streaming to a browser with working input, no re-encode hop) is met and validated on hardware.
|
||||||
|
|
||||||
|
**Phase 1 (single-device vertical slice) — in progress.** The two Phase 0 halves are now joined: the
|
||||||
|
React review surface's left pane is a live WebRTC client (`frontend/src/components/LeftPane.tsx`) that
|
||||||
|
streams the real device from `streamd` with working touch/key control and frame capture, while the
|
||||||
|
right pane captures typed/voice findings anchored to the review-api session. The real device is
|
||||||
|
registered in the registry. Vite proxies `/api`→review-api and `/streamd`→streamd. Run all three
|
||||||
|
(review-api, streamd, `frontend` dev server) and open the app to drive the device and log findings.
|
||||||
|
Still ahead for Phase 1: user auth and the operator provisioning workflow.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
- `review-api/` — Rust / Axum / sqlx review API: registry, atomic claim/release,
|
||||||
|
findings CRUD, audio + frame upload, transcription handoff. Listens on `:8090`.
|
||||||
|
- `transcription-svc/` — self-hosted Whisper-class service (ffmpeg + whisper.cpp via
|
||||||
|
`whisper-rs`). Separate process so transcription never touches the real-time path.
|
||||||
|
Listens on `:8099`.
|
||||||
|
- `stream-node/` — per-device streaming plane (§6.1). Currently the input back-channel:
|
||||||
|
scrcpy control-protocol encoding + coordinate mapping. WebRTC/scrcpy wiring pending.
|
||||||
|
- `frontend/` — React + TypeScript + Tailwind v4 split-screen review surface.
|
||||||
|
- `docker-compose.yml` — Postgres 16 for local dev.
|
||||||
|
|
||||||
|
## Run it locally
|
||||||
|
|
||||||
|
```sh
|
||||||
|
docker compose up -d postgres # Postgres on :5432
|
||||||
|
./transcription-svc/scripts/fetch-model.sh tiny.en # download a ggml model (once)
|
||||||
|
WHISPER_MODEL=transcription-svc/models/ggml-tiny.en.bin cargo run -p transcription-svc # :8099
|
||||||
|
cargo run -p review-api # :8090 (auto-migrates + seeds a placeholder device)
|
||||||
|
cd frontend && npm install && npm run dev # Vite on :5173, proxies /api -> :8090
|
||||||
|
```
|
||||||
|
|
||||||
|
Open http://localhost:5173, claim the placeholder device, and capture findings. Voice
|
||||||
|
recording and frame capture need a real browser (mic + canvas permissions).
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export DATABASE_URL=postgres://clawreview:clawreview@localhost:5432/clawreview
|
||||||
|
cargo test --workspace # API + transcription HTTP contract (ephemeral DBs)
|
||||||
|
cargo test -p transcription-svc --test whisper -- --ignored # real whisper.cpp E2E (needs model + ffmpeg)
|
||||||
|
```
|
||||||
|
|
||||||
|
Config via env: `DATABASE_URL`, `BIND_ADDR`, `STORAGE_DIR` (local R2 stand-in, default
|
||||||
|
`./data`), `TRANSCRIBE_URL` (review-api → transcription-svc, default `http://127.0.0.1:8099/transcribe`),
|
||||||
|
`WHISPER_MODEL` (transcription-svc model path).
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,374 @@
|
|||||||
|
# ClawReview — Platform Plan (Revised v1.1)
|
||||||
|
|
||||||
|
**A pure-Rust real-device review platform: stream live Android handsets into the browser, with side-by-side notes and voice-transcribed findings, for engineers and partners to review RedClaw apps.**
|
||||||
|
|
||||||
|
- Document type: Combined Spec + PRD + Roadmap
|
||||||
|
- Owner: Omar Sobh, RedClaw Systems LLC
|
||||||
|
- Status: Draft v1.1
|
||||||
|
- Date: June 2026
|
||||||
|
|
||||||
|
## Changelog vs v1.0
|
||||||
|
- §3 / §10 Phase 3 / §11 — remote = software-only; on-site operator owns physical hardware events.
|
||||||
|
- §4.3 — input back-channel settled on WebSocket (was "data channel or WebSocket").
|
||||||
|
- §4.4 / §6.2 — atomic claim (conditional UPDATE), heartbeat, orphaned-claim timeout, stream-node reconciliation.
|
||||||
|
- §4.4 / §4.5 / §6.2 / §8 — device state reset on release.
|
||||||
|
- §4.5 / §7 — findings carry a captured frame **and** a session timestamp offset; optional short clip noted for temporal findings.
|
||||||
|
- Media transport stays pure-Rust webrtc-rs everywhere (v1.0 decision retained).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Executive Summary
|
||||||
|
|
||||||
|
ClawReview is an internal and partner-facing web platform that lets a reviewer drive a
|
||||||
|
real, physical Android device from a browser while capturing structured findings beside
|
||||||
|
it. The screen of each handset is streamed live into the left half of the browser. The
|
||||||
|
right half holds two stacked review surfaces: a text notes panel on top and a voice
|
||||||
|
recorder with automatic transcription beneath it. Reviewers claim a device, navigate the
|
||||||
|
pre-installed app, type or speak their findings, and every finding is automatically
|
||||||
|
anchored to the device, the build under review, the reviewer, a timestamp, and a captured
|
||||||
|
frame.
|
||||||
|
|
||||||
|
The platform exists because several RedClaw products (Digital Hallmark, CardClaws,
|
||||||
|
StackSticker) depend on native hardware behavior — NFC and UHF RFID — that no emulator can
|
||||||
|
reproduce. Reviews must run on real silicon near real readers. ClawReview turns a rack of
|
||||||
|
physical phones into a shared, remotely accessible review fleet, so engineers on-site and
|
||||||
|
partners across MENA, Southeast Asia, and the US can review the same handsets without
|
||||||
|
shipping hardware. On-site engineers drive both software and hardware flows directly;
|
||||||
|
remote partners drive software flows, with hardware events presented by an on-site
|
||||||
|
operator.
|
||||||
|
|
||||||
|
The system is built pure-Rust on the backend (Axum, Tokio, the `webrtc` crate) and React
|
||||||
|
with TailwindCSS and shadcn/ui on the frontend. Owning the full WebRTC media path in Rust
|
||||||
|
— rather than delegating it to a third-party media server — keeps the entire stack in one
|
||||||
|
language, keeps confidential partner builds on RedClaw infrastructure, and gives full
|
||||||
|
control over the latency-critical streaming and input back-channel.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Goals and Non-Goals
|
||||||
|
|
||||||
|
### 2.1 Goals
|
||||||
|
See and control a real Android device from any modern browser at responsive latency.
|
||||||
|
Support multiple physical devices per host, each independently claimable. Capture typed and
|
||||||
|
spoken findings, transcribe speech on RedClaw infrastructure, and anchor every finding to
|
||||||
|
device, build, reviewer, time, and on-screen frame. Work for on-site engineers (LAN) and
|
||||||
|
remote partners (relayed) through the same pipeline. Keep all confidential builds and all
|
||||||
|
reviewer findings inside RedClaw-controlled infrastructure at every step.
|
||||||
|
|
||||||
|
### 2.2 Non-Goals (v1)
|
||||||
|
No software emulators — real devices only. No iOS (cannot run on Linux hosts; a simulator
|
||||||
|
cannot exercise NFC/RFID). No app install at review time — builds are pre-installed out of
|
||||||
|
band by an operator. No automated test execution, scripting, or CI integration. No full
|
||||||
|
session video recording (frame captures, plus optional short clips, only). No app signing
|
||||||
|
or Play Store provisioning beyond the documented operator install step. **No remote
|
||||||
|
partner-driven physical hardware interaction** — remote partners review software flows;
|
||||||
|
physical NFC/RFID presentation is an on-site operator action.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Personas and Primary Use Cases
|
||||||
|
|
||||||
|
The **on-site engineer** sits on the same network as the rig. They claim a handset,
|
||||||
|
exercise a build end to end — including physically tapping NFC/RFID against a nearby reader
|
||||||
|
— and log defects as fast typed notes. Latency is near zero; their priority is throughput
|
||||||
|
and precise frame-anchored notes.
|
||||||
|
|
||||||
|
The **remote partner** (Jakarta, Istanbul, Muscat, …) reviews a build running on a phone
|
||||||
|
physically located in the US. Their stream is relayed through coturn. They drive the
|
||||||
|
software/UI flows and dictate findings by voice when typing in a second language is slower.
|
||||||
|
Because they cannot physically present a tag to the US-located reader, **any hardware step
|
||||||
|
in their review is triggered by an on-site operator**, with the partner watching the result
|
||||||
|
resolve on screen and logging the finding. Confidentiality matters most here: their findings
|
||||||
|
and the build they see must never leave RedClaw infrastructure.
|
||||||
|
|
||||||
|
The **review operator / admin** provisions the rig: attaches phones, installs and tags
|
||||||
|
builds, registers devices, monitors fleet health, and — for remote sessions — performs the
|
||||||
|
physical hardware presentation on request. They never author review findings themselves;
|
||||||
|
they keep the fleet ready and the hardware exercised.
|
||||||
|
|
||||||
|
**Defining use case (on-site):** an operator installs the latest Digital Hallmark build on
|
||||||
|
three handsets and registers them. An on-site engineer sees three available devices labeled
|
||||||
|
with the build version, claims one, drives the provenance flow, taps a real RFID tag against
|
||||||
|
the reader attached to that phone, watches the read resolve, and dictates a finding about
|
||||||
|
read latency. The finding is saved with the device serial, the build hash, the engineer's
|
||||||
|
identity, the timestamp, the session time-offset, and a captured frame (and, for a timing
|
||||||
|
finding, an optional short clip) of the result screen. The engineer releases the device; its
|
||||||
|
app state is reset and it returns to the available pool.
|
||||||
|
|
||||||
|
**Defining use case (remote):** a partner in Istanbul claims one of the same devices, drives
|
||||||
|
the on-screen provenance flow, and — at the read step — asks the on-site operator (in-session)
|
||||||
|
to present the tag. The operator taps; the partner watches the read resolve, captures the
|
||||||
|
frame, and dictates the latency finding. Same anchoring chain, same confidentiality guarantees.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. System Architecture
|
||||||
|
|
||||||
|
### 4.1 Overview
|
||||||
|
Two backend planes, a frontend, a relay, and the physical rig.
|
||||||
|
|
||||||
|
- **Stream node** — latency-sensitive, per-device, co-located with the rig. Manages the
|
||||||
|
scrcpy lifecycle over ADB, reads each device's H.264 stream, packetizes into WebRTC, runs
|
||||||
|
per-device peer connections, and routes input back to the device via the scrcpy control
|
||||||
|
protocol. Home of the `webrtc` crate, H.264 handling, and Tokio per-device tasks.
|
||||||
|
- **Review API** — request/response plane, runs anywhere reachable by reviewers and the
|
||||||
|
stream node. Owns the device registry, claim/release orchestration, signaling brokerage,
|
||||||
|
auth, the findings store, audio upload, and the transcription handoff. Talks to PostgreSQL
|
||||||
|
and Cloudflare R2.
|
||||||
|
- **Frontend** — React/Tailwind/shadcn split-screen surface: WebRTC client + input-capture
|
||||||
|
on the left, notes/voice review on the right.
|
||||||
|
- **TURN relay** — existing RedClaw coturn (reused from LiveCast) for reviewers who can't
|
||||||
|
reach the stream node directly (most remote partners).
|
||||||
|
- **Device rig** — Android handsets on a powered USB hub on the stream-node host.
|
||||||
|
|
||||||
|
### 4.2 Media Path (device → browser)
|
||||||
|
Each device runs the scrcpy server (pushed via ADB), emitting an H.264 elementary stream on
|
||||||
|
a local socket plus a control socket for input injection. The stream node reads the H.264
|
||||||
|
stream and — because browsers reject H.264 with B-frames over WebRTC — ensures the encoder is
|
||||||
|
configured for a baseline-compatible, B-frame-free profile so the stream forwards without
|
||||||
|
re-encoding. Access units feed a per-device `webrtc`-crate peer connection handling RTP
|
||||||
|
packetization, SPS/PPS parameter-set handling, keyframe cadence, SRTP, and ICE. The browser
|
||||||
|
receives the track on a standard `<video>` element. Where a device model's encoder cannot
|
||||||
|
avoid B-frames, a per-stream re-encode hop is the documented fallback at a measured CPU cost;
|
||||||
|
the default path assumes no re-encode.
|
||||||
|
|
||||||
|
### 4.3 Input Path (browser → device)
|
||||||
|
The hardest custom component, with no off-the-shelf pure-Rust equivalent. The browser
|
||||||
|
captures pointer/touch/keyboard events over the video element and scales each event's
|
||||||
|
coordinates from the rendered element size to the device's real resolution, accounting for
|
||||||
|
current orientation. **These normalized events travel over the existing session WebSocket**
|
||||||
|
(ordered, reliable, trivially debuggable — chosen over a WebRTC data channel for simplicity;
|
||||||
|
the LAN latency delta is negligible and the control channel is low-bandwidth). The stream
|
||||||
|
node validates and translates them into scrcpy control-protocol messages written to the
|
||||||
|
device's control socket. Coordinate-scaling is mirrored: the frontend scales into device
|
||||||
|
space; the backend re-validates and forwards.
|
||||||
|
|
||||||
|
### 4.4 Signaling and Session Lifecycle
|
||||||
|
A reviewer claims a device through the review API. **The claim is atomic** — a conditional
|
||||||
|
`UPDATE devices SET status='claimed', … WHERE serial=$1 AND status='free' RETURNING …` so two
|
||||||
|
reviewers can never hold the same handset (no read-then-write race). On success the API
|
||||||
|
brokers the WebRTC handshake: SDP offer/answer and ICE candidates pass over a WebSocket
|
||||||
|
between browser and stream node, with the review API mediating authorization. ICE negotiates
|
||||||
|
the best path — direct on LAN, relayed through coturn for remote partners.
|
||||||
|
|
||||||
|
**A session heartbeat** (browser → API) keeps the claim alive. On explicit release,
|
||||||
|
disconnect, missed-heartbeat timeout, or admin override, the peer connection tears down, the
|
||||||
|
scrcpy session resets cleanly, **the app-under-review's state is reset** (clear app data /
|
||||||
|
relaunch, see §8), and the device returns to the available pool. The stream node reports
|
||||||
|
device health to the API, which **reconciles** registry status against that health (e.g. a
|
||||||
|
stream-node restart re-derives free/claimed/offline) so an orphaned claim never strands a
|
||||||
|
device permanently.
|
||||||
|
|
||||||
|
### 4.5 Review Surface (right pane)
|
||||||
|
The notes panel is a debounced-autosave text surface writing to the findings store, each note
|
||||||
|
carrying the active session's anchoring metadata. The voice panel captures microphone audio
|
||||||
|
via the MediaRecorder API, uploads the blob to the review API, which stores it in R2 and
|
||||||
|
submits it to a self-hosted Whisper-class transcription service on RedClaw infrastructure
|
||||||
|
(Hetzner/Vultr) so audio and transcripts never leave RedClaw control; the transcript renders
|
||||||
|
inline beneath the recording. A one-tap **capture-frame** control draws the current `<video>`
|
||||||
|
frame to a canvas, producing a still that anchors the finding visually **and records the
|
||||||
|
session time-offset** so the finding is locatable in time, not just in space. For temporal
|
||||||
|
findings (e.g. read latency), an **optional short clip** (a few seconds of rolling buffer
|
||||||
|
around the capture) may be retained — single stills cannot evidence timing, which is exactly
|
||||||
|
what several target findings are about. Full session recording remains a non-goal.
|
||||||
|
|
||||||
|
### 4.6 Why Pure Rust, and What That Commits Us To
|
||||||
|
Owning the WebRTC media plane in Rust via the `webrtc` crate keeps the backend in one language
|
||||||
|
and toolchain, keeps confidential builds and findings fully on RedClaw infrastructure, and
|
||||||
|
gives complete control over the latency-critical path. The commitment is real: the `webrtc`
|
||||||
|
crate is younger and less battle-hardened than the Go/C++ equivalents, and the
|
||||||
|
H.264-to-WebRTC packetization and parameter-set negotiation is the single most failure-prone
|
||||||
|
area of the system. This risk is concentrated, identified, and addressed first via a
|
||||||
|
single-device spike before any orchestration is built around it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Technical Stack
|
||||||
|
|
||||||
|
Backend: Rust throughout. Axum on Tokio. WebRTC via the `webrtc` crate (peer connections,
|
||||||
|
RTP/SRTP, ICE). Device comms via ADB + the scrcpy server protocol (video and control).
|
||||||
|
PostgreSQL for registry, sessions, users, and findings metadata. Cloudflare R2 for audio
|
||||||
|
blobs, frame captures, and optional clips. Transcription: a self-hosted Whisper-class service
|
||||||
|
on Hetzner/Vultr, called over HTTP from the review API so it stays off the real-time path.
|
||||||
|
TURN relay: existing RedClaw coturn.
|
||||||
|
|
||||||
|
Frontend: React + TypeScript, TailwindCSS, shadcn/ui. Left pane uses native
|
||||||
|
`RTCPeerConnection` and `<video>`, the peer connection held in an explicit owned lifecycle
|
||||||
|
object (never tied to component render). Right pane uses MediaRecorder and shadcn primitives
|
||||||
|
(resizable panels, tabs, cards).
|
||||||
|
|
||||||
|
Infra follows RedClaw standards: Hetzner/Vultr compute, Cloudflare R2, coturn relay. The
|
||||||
|
stream node is co-located with the rig.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Component Specifications
|
||||||
|
|
||||||
|
### 6.1 Stream Node (Rust)
|
||||||
|
scrcpy supervisor (push server per device, open video/control sockets, detect drops/USB
|
||||||
|
disconnects, recover cleanly), H.264 reader + WebRTC bridge (read access units, packetize
|
||||||
|
into RTP, manage parameter sets and keyframe cadence, run the per-device peer connection),
|
||||||
|
and control receiver (validate normalized input, translate to scrcpy control messages, write
|
||||||
|
to the control socket). Exposes its half of the signaling handshake and reports device health
|
||||||
|
to the review API. Resilient by design: one device dropping must not disturb the others.
|
||||||
|
|
||||||
|
### 6.2 Review API (Rust / Axum)
|
||||||
|
Owns the device registry (serial → installed build/version → status: free/claimed/offline),
|
||||||
|
populated by operator provisioning and kept current by stream-node health signals. Owns
|
||||||
|
claim/release with the **atomic conditional-update claim model** and **heartbeat-based
|
||||||
|
orphan recovery** of §4.4, including **app-state reset on release**. Brokers signaling.
|
||||||
|
Authenticates users and gates partner access. Manages findings CRUD, audio upload to R2, and
|
||||||
|
the transcription handoff. The registry is the source of truth that auto-stamps every finding
|
||||||
|
with the correct build metadata.
|
||||||
|
|
||||||
|
### 6.3 Frontend (React / Tailwind / shadcn)
|
||||||
|
Split-screen shell with resizable panes. Left: WebRTC video consumer + input-capture and
|
||||||
|
coordinate-scaling layer. Right: debounced-autosave notes, MediaRecorder voice capture with
|
||||||
|
playback and inline transcript, one-tap frame capture (still + time-offset, optional clip).
|
||||||
|
Three state concerns kept deliberately separate: the WebRTC/peer-connection lifecycle, the
|
||||||
|
claim state, and the findings buffer. The peer connection is owned explicitly and never
|
||||||
|
re-created on render.
|
||||||
|
|
||||||
|
### 6.4 Transcription Service (self-hosted)
|
||||||
|
A Whisper-class model over HTTP on RedClaw infrastructure. Accepts an audio blob, returns a
|
||||||
|
transcript. Separate from the review API so transcription load never touches the real-time
|
||||||
|
path; on RedClaw infrastructure so partner findings never leave RedClaw control. Model size
|
||||||
|
chosen for multilingual accuracy (Turkish, Indonesian, Arabic) within latency budget.
|
||||||
|
|
||||||
|
### 6.5 Provisioning / Admin (out of band)
|
||||||
|
An operator workflow — not coupled to runtime sessions — to attach devices, install builds via
|
||||||
|
ADB, tag each device with build/version, and register/retire devices in the registry. Because
|
||||||
|
installs happen here, the runtime session layer never handles APK installs, signing, or install
|
||||||
|
latency. The operator also performs in-session physical hardware presentation for remote partners.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Data Model (high level)
|
||||||
|
|
||||||
|
- **device** — serial, model, status, attached stream-node, currently installed build reference.
|
||||||
|
- **build** — app identity, version, hash, install metadata.
|
||||||
|
- **session** — links a reviewer to a device for a time window; references the build installed
|
||||||
|
at claim time; carries a claim start time so findings can record a relative time-offset.
|
||||||
|
- **finding** — note text or audio reference + transcript; anchors to session, device serial,
|
||||||
|
build hash, reviewer, timestamp, **session time-offset**, and an optional captured-frame and
|
||||||
|
optional short-clip reference.
|
||||||
|
|
||||||
|
Audio blobs, frame captures, and clips live in R2; their references live in PostgreSQL. The
|
||||||
|
anchoring chain finding → session → device → build guarantees every finding is unambiguously
|
||||||
|
attributable.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Security, Privacy, and Access
|
||||||
|
|
||||||
|
All confidential builds remain on RedClaw infrastructure — installed on physically controlled
|
||||||
|
devices, never uploaded to third-party services. All reviewer findings (typed, spoken,
|
||||||
|
transcribed) stay on RedClaw infrastructure end to end, including transcription. Partner access
|
||||||
|
is authenticated and gated; partners see and claim only the devices and builds intended for
|
||||||
|
them. Remote streams relay through RedClaw-controlled coturn. The threat model treats partner
|
||||||
|
accounts as semi-trusted: scoped access, no fleet-wide visibility, clear separation of what
|
||||||
|
each partner may review.
|
||||||
|
|
||||||
|
**Cross-session device isolation:** because one physical phone is shared across partners and
|
||||||
|
builds, the app-under-review's data and on-screen state persist between claims unless cleared.
|
||||||
|
On every release the lifecycle **resets app state** (clear app data / relaunch) so one
|
||||||
|
reviewer's session — including any confidential interaction — does not leak into the next.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Key Risks and Mitigations
|
||||||
|
|
||||||
|
1. **H.264→WebRTC on the `webrtc` crate** (dominant) — parameter-set handling, profile
|
||||||
|
negotiation, packetization; a mismatch yields a black screen that is painful to debug.
|
||||||
|
Mitigated by isolating it in the Phase 0 single-device spike before anything depends on it,
|
||||||
|
and by configuring scrcpy for a B-frame-free, baseline-compatible profile.
|
||||||
|
2. **Input back-channel coordinate scaling** across rendered-vs-device resolution and
|
||||||
|
orientation; if wrong, taps land in the wrong place. Mitigated by mirroring scaling on both
|
||||||
|
ends and validating against multiple device models.
|
||||||
|
3. **Physical rig** — phones drop off USB under load. Mitigated with a powered USB hub,
|
||||||
|
per-device drop detection/recovery in the scrcpy supervisor, and treating one device's
|
||||||
|
failure as isolated.
|
||||||
|
4. **Remote stream quality** across long internet paths. Mitigated by the coturn relay,
|
||||||
|
TCP-transport fallback when UDP is blocked, and setting partner latency expectations.
|
||||||
|
5. **`webrtc`-crate maturity under multi-stream load.** Mitigated by validating the
|
||||||
|
single-device path thoroughly first and scaling stream count deliberately while watching for
|
||||||
|
instability, with the explicit option to reassess if the crate cannot carry target concurrency.
|
||||||
|
6. **Orphaned claims / stale registry** — a crashed browser or stream-node restart stranding a
|
||||||
|
device. Mitigated by the atomic claim, session heartbeat with timeout, and stream-node ↔
|
||||||
|
registry health reconciliation (§4.4).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Roadmap
|
||||||
|
|
||||||
|
### Phase 0 — Foundations and De-Risking
|
||||||
|
Two parallel tracks (no shared dependencies):
|
||||||
|
- **Review surface end to end against a placeholder video source** — split-screen shell, notes
|
||||||
|
with autosave, voice capture with MediaRecorder, the self-hosted transcription service, R2
|
||||||
|
audio storage, the findings data model, and the anchoring chain — all without streaming code.
|
||||||
|
Validates the entire right-hand product and the full persistence path at zero streaming risk.
|
||||||
|
- **Single-device WebRTC spike** — one phone, scrcpy video into the `webrtc` crate, into one
|
||||||
|
browser tab, with working taps over the WebSocket back-channel. No orchestration, no TURN, no
|
||||||
|
multi-device. Proves/disproves the riskiest decision cheaply. Also determines, per target
|
||||||
|
device model, whether the encoder needs the baseline re-encode hop, setting the per-stream CPU
|
||||||
|
budget and therefore device density per host.
|
||||||
|
|
||||||
|
**Exit:** review surface works against a placeholder; one phone streams to one browser with
|
||||||
|
working input.
|
||||||
|
|
||||||
|
### Phase 1 — Single-Device Vertical Slice
|
||||||
|
Join the two tracks: a real device streams into the real review surface; a reviewer drives it
|
||||||
|
and logs typed and voice findings; every finding is correctly anchored (device, build, reviewer,
|
||||||
|
time, time-offset, frame). Add user authentication and the operator provisioning workflow (install,
|
||||||
|
tag, register). **Include a minimal on-site NFC/RFID smoke test** — one hardware flow on one
|
||||||
|
device — so the premise that justified real devices is validated before the fleet is built, not
|
||||||
|
deferred to Phase 4.
|
||||||
|
|
||||||
|
**Exit:** an on-site engineer reviews a real pre-installed build on one device with full finding
|
||||||
|
capture, including at least one physical hardware flow.
|
||||||
|
|
||||||
|
### Phase 2 — Multi-Device Fleet and Orchestration
|
||||||
|
Scale to the rig: device registry as live source of truth, the atomic claim/release model with
|
||||||
|
heartbeat and reconciliation, fleet-health monitoring, and the stream node's capacity for multiple
|
||||||
|
concurrent per-device peer connections and scrcpy sessions. Validate device density per host
|
||||||
|
against the Phase 0 CPU budget. Harden the USB rig and the scrcpy supervisor's drop-recovery.
|
||||||
|
|
||||||
|
**Exit:** multiple on-site engineers concurrently review different devices in the same fleet.
|
||||||
|
|
||||||
|
### Phase 3 — Remote Partners and Relay
|
||||||
|
Bring in the coturn-relayed path so remote partners review US-located devices. Validate ICE path
|
||||||
|
selection (direct on LAN, relayed for remote), TCP-transport fallback for restrictive firewalls,
|
||||||
|
and partner-scoped auth/access. **Implement the operator-assisted hardware flow** for remote
|
||||||
|
sessions: an in-session channel by which a remote partner requests a physical tag presentation and
|
||||||
|
the on-site operator performs it while the partner watches and logs the finding. Tune for
|
||||||
|
cross-continental latency and set partner expectations. Confirm end to end that partner builds and
|
||||||
|
findings never leave RedClaw infrastructure.
|
||||||
|
|
||||||
|
**Exit:** a partner abroad reviews a US-located device's software flows, with operator-assisted
|
||||||
|
hardware steps, confidentiality preserved end to end.
|
||||||
|
|
||||||
|
### Phase 4 — Hardening and Native-Flow Coverage
|
||||||
|
Validate the full NFC/UHF RFID review workflows against physical readers on specific handsets for
|
||||||
|
Digital Hallmark, CardClaws, and StackSticker, on-site and operator-assisted-remote. Strengthen
|
||||||
|
resilience, observability, and recovery across the fleet. Refine the review surface from real
|
||||||
|
feedback (frame-anchoring ergonomics, temporal-clip ergonomics, transcript editing).
|
||||||
|
|
||||||
|
**Exit:** the platform reliably supports the NFC/RFID review workflows that motivated it, for
|
||||||
|
on-site and (operator-assisted) remote reviewers.
|
||||||
|
|
||||||
|
### Future Considerations (post-v1)
|
||||||
|
iOS via Mac hosts and real iPhones. Full session recording/playback. Multiple stream-node hosts and
|
||||||
|
geographic rig distribution. Automated/scripted review flows. An automated tag-presentation rig so
|
||||||
|
remote partners can trigger hardware reads without an on-site operator. Deeper integration with
|
||||||
|
RedClaw build pipelines (EAS/Expo build → rig).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Success Criteria
|
||||||
|
|
||||||
|
The platform succeeds when an on-site engineer or remote partner can, without shipping hardware,
|
||||||
|
claim a real device from a browser, drive a pre-installed RedClaw build — its full NFC/RFID flows
|
||||||
|
on-site, its software flows with operator-assisted hardware steps remotely — and capture typed and
|
||||||
|
spoken findings automatically and unambiguously anchored to what they reviewed (device, build,
|
||||||
|
reviewer, time, time-offset, frame), with all builds and findings remaining entirely on RedClaw
|
||||||
|
infrastructure — on-site at near-zero latency and remotely over a relayed but usable stream.
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
services:
|
||||||
|
postgres:
|
||||||
|
image: postgres:16
|
||||||
|
environment:
|
||||||
|
POSTGRES_USER: clawreview
|
||||||
|
POSTGRES_PASSWORD: clawreview
|
||||||
|
POSTGRES_DB: clawreview
|
||||||
|
ports:
|
||||||
|
- "5432:5432"
|
||||||
|
volumes:
|
||||||
|
- pgdata:/var/lib/postgresql/data
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U clawreview"]
|
||||||
|
interval: 2s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 30
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
pgdata:
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
certs/
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en" class="dark">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||||
|
<title>ClawReview</title>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div id="root"></div>
|
||||||
|
<script type="module" src="/src/main.tsx"></script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
Generated
+3462
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,39 @@
|
|||||||
|
{
|
||||||
|
"name": "clawreview-frontend",
|
||||||
|
"private": true,
|
||||||
|
"version": "0.1.0",
|
||||||
|
"type": "module",
|
||||||
|
"scripts": {
|
||||||
|
"dev": "vite",
|
||||||
|
"build": "vite build",
|
||||||
|
"preview": "vite preview"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"@radix-ui/react-dialog": "^1.1.17",
|
||||||
|
"@radix-ui/react-dropdown-menu": "^2.1.18",
|
||||||
|
"@radix-ui/react-scroll-area": "^1.2.12",
|
||||||
|
"@radix-ui/react-separator": "^1.1.10",
|
||||||
|
"@radix-ui/react-slot": "^1.3.0",
|
||||||
|
"@radix-ui/react-tabs": "^1.1.15",
|
||||||
|
"@radix-ui/react-tooltip": "^1.2.10",
|
||||||
|
"class-variance-authority": "^0.7.1",
|
||||||
|
"clsx": "^2.1.1",
|
||||||
|
"cmdk": "^1.1.1",
|
||||||
|
"lucide-react": "^1.18.0",
|
||||||
|
"react": "^18.3.1",
|
||||||
|
"react-dom": "^18.3.1",
|
||||||
|
"react-resizable-panels": "^2.1.7",
|
||||||
|
"sonner": "^2.0.7",
|
||||||
|
"tailwind-merge": "^3.6.0"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@tailwindcss/vite": "^4.0.0",
|
||||||
|
"@types/node": "^25.9.3",
|
||||||
|
"@types/react": "^18.3.12",
|
||||||
|
"@types/react-dom": "^18.3.1",
|
||||||
|
"@vitejs/plugin-react": "^4.3.4",
|
||||||
|
"tailwindcss": "^4.0.0",
|
||||||
|
"typescript": "^5.6.3",
|
||||||
|
"vite": "^6.0.5"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,215 @@
|
|||||||
|
import { useEffect, useMemo, useRef, useState } from "react";
|
||||||
|
import { toast } from "sonner";
|
||||||
|
import { Settings as SettingsIcon, MonitorSmartphone } from "lucide-react";
|
||||||
|
import { api, type Device, type Session } from "./api";
|
||||||
|
import { ReviewShell } from "./components/ReviewShell";
|
||||||
|
import { SettingsDialog } from "./components/SettingsDialog";
|
||||||
|
import { CommandPalette, type CommandAction } from "./components/CommandPalette";
|
||||||
|
import { DeviceCard } from "./components/DeviceCard";
|
||||||
|
import { Button } from "@/components/ui/button";
|
||||||
|
|
||||||
|
// Top-level owns the *claim state* only — one of the three deliberately separate
|
||||||
|
// state concerns from §6.3 (claim / findings buffer / peer-connection lifecycle).
|
||||||
|
export default function App() {
|
||||||
|
const [devices, setDevices] = useState<Device[]>([]);
|
||||||
|
const [session, setSession] = useState<Session | null>(null);
|
||||||
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
const [settingsOpen, setSettingsOpen] = useState(false);
|
||||||
|
const [paletteOpen, setPaletteOpen] = useState(false);
|
||||||
|
// Bump every 5 minutes to refresh device thumbnails.
|
||||||
|
const [thumbTick, setThumbTick] = useState(0);
|
||||||
|
useEffect(() => {
|
||||||
|
const t = window.setInterval(() => setThumbTick((n) => n + 1), 5 * 60 * 1000);
|
||||||
|
return () => window.clearInterval(t);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const refreshDevices = () =>
|
||||||
|
api.listDevices().then(setDevices).catch((e) => setError(String(e)));
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
refreshDevices();
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
// Live registry/session events (Workstream 2): update the device list in place and
|
||||||
|
// surface toasts, replacing the poll-only model.
|
||||||
|
useEffect(() => {
|
||||||
|
const es = new EventSource("/api/events");
|
||||||
|
es.onmessage = (e) => {
|
||||||
|
const ev = JSON.parse(e.data);
|
||||||
|
if (ev.type === "device_status") {
|
||||||
|
setDevices((prev) =>
|
||||||
|
prev.map((d) => (d.serial === ev.serial ? { ...d, status: ev.status } : d)),
|
||||||
|
);
|
||||||
|
if (ev.status === "offline") toast.warning(`${ev.serial} went offline`);
|
||||||
|
} else if (ev.type === "session_ended") {
|
||||||
|
setSession((cur) => {
|
||||||
|
if (cur && cur.id === ev.session_id) {
|
||||||
|
toast.warning("Session ended — device released or taken over");
|
||||||
|
refreshDevices();
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
return cur;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
};
|
||||||
|
return () => es.close();
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
// Keep the claim alive while a session is held. The heartbeat reports whether the
|
||||||
|
// reviewer interacted since the last beat, so a dormant (no-interaction) session is
|
||||||
|
// auto-reclaimed even with the tab open; a closed tab releases via the unload beacon.
|
||||||
|
const activeSinceBeat = useRef(false);
|
||||||
|
useEffect(() => {
|
||||||
|
if (!session) return;
|
||||||
|
const id = session.id;
|
||||||
|
const markActive = () => {
|
||||||
|
activeSinceBeat.current = true;
|
||||||
|
};
|
||||||
|
window.addEventListener("pointerdown", markActive);
|
||||||
|
window.addEventListener("keydown", markActive);
|
||||||
|
window.addEventListener("wheel", markActive, { passive: true });
|
||||||
|
|
||||||
|
const beat = () => {
|
||||||
|
api.heartbeat(id, activeSinceBeat.current);
|
||||||
|
activeSinceBeat.current = false;
|
||||||
|
};
|
||||||
|
beat();
|
||||||
|
const timer = window.setInterval(beat, 10_000);
|
||||||
|
const onUnload = () => navigator.sendBeacon(`/api/sessions/${id}/release`);
|
||||||
|
window.addEventListener("pagehide", onUnload);
|
||||||
|
return () => {
|
||||||
|
window.clearInterval(timer);
|
||||||
|
window.removeEventListener("pagehide", onUnload);
|
||||||
|
window.removeEventListener("pointerdown", markActive);
|
||||||
|
window.removeEventListener("keydown", markActive);
|
||||||
|
window.removeEventListener("wheel", markActive);
|
||||||
|
};
|
||||||
|
}, [session]);
|
||||||
|
|
||||||
|
async function claim(serial: string) {
|
||||||
|
setError(null);
|
||||||
|
try {
|
||||||
|
const s = await api.createSession(serial);
|
||||||
|
setSession(s);
|
||||||
|
const model = devices.find((d) => d.serial === serial)?.model ?? serial;
|
||||||
|
toast.success(`Claimed ${model}`);
|
||||||
|
} catch (e) {
|
||||||
|
const msg = String(e);
|
||||||
|
setError(msg);
|
||||||
|
toast.error(msg.replace(/^Error:\s*/, ""));
|
||||||
|
refreshDevices();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function release() {
|
||||||
|
if (!session) return;
|
||||||
|
try {
|
||||||
|
await api.releaseSession(session.id);
|
||||||
|
toast("Device released");
|
||||||
|
} finally {
|
||||||
|
setSession(null);
|
||||||
|
refreshDevices();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function switchDevice(serial: string) {
|
||||||
|
if (session) {
|
||||||
|
try {
|
||||||
|
await api.releaseSession(session.id);
|
||||||
|
} catch {
|
||||||
|
/* ignore */
|
||||||
|
}
|
||||||
|
setSession(null);
|
||||||
|
}
|
||||||
|
await claim(serial);
|
||||||
|
}
|
||||||
|
|
||||||
|
const actions = useMemo<CommandAction[]>(() => {
|
||||||
|
const list: CommandAction[] = [];
|
||||||
|
for (const d of devices) {
|
||||||
|
if (d.status === "offline") continue;
|
||||||
|
const claimed = d.serial === session?.device_serial;
|
||||||
|
if (claimed) continue;
|
||||||
|
list.push({
|
||||||
|
id: `device-${d.serial}`,
|
||||||
|
group: session ? "Switch device" : "Devices",
|
||||||
|
label: `${d.status === "claimed" ? "Take over" : "Claim"} ${d.model ?? d.serial}`,
|
||||||
|
icon: <MonitorSmartphone />,
|
||||||
|
onSelect: () => switchDevice(d.serial),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if (session) {
|
||||||
|
list.push({ id: "release", group: "Session", label: "Release device", onSelect: release });
|
||||||
|
}
|
||||||
|
list.push({
|
||||||
|
id: "settings",
|
||||||
|
group: "App",
|
||||||
|
label: "Open settings",
|
||||||
|
icon: <SettingsIcon />,
|
||||||
|
onSelect: () => setSettingsOpen(true),
|
||||||
|
});
|
||||||
|
return list;
|
||||||
|
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||||
|
}, [devices, session]);
|
||||||
|
|
||||||
|
const overlays = (
|
||||||
|
<>
|
||||||
|
<SettingsDialog open={settingsOpen} onOpenChange={setSettingsOpen} />
|
||||||
|
<CommandPalette open={paletteOpen} onOpenChange={setPaletteOpen} actions={actions} />
|
||||||
|
</>
|
||||||
|
);
|
||||||
|
|
||||||
|
if (session) {
|
||||||
|
const device = devices.find((d) => d.serial === session.device_serial) ?? null;
|
||||||
|
return (
|
||||||
|
<>
|
||||||
|
<ReviewShell
|
||||||
|
session={session}
|
||||||
|
device={device}
|
||||||
|
devices={devices}
|
||||||
|
onRelease={release}
|
||||||
|
onSwitch={switchDevice}
|
||||||
|
onOpenSettings={() => setSettingsOpen(true)}
|
||||||
|
onOpenPalette={() => setPaletteOpen(true)}
|
||||||
|
/>
|
||||||
|
{overlays}
|
||||||
|
</>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="mx-auto max-w-3xl px-6 py-10">
|
||||||
|
<header className="mb-8 flex items-start justify-between">
|
||||||
|
<div>
|
||||||
|
<h1 className="text-2xl font-semibold tracking-tight">ClawReview</h1>
|
||||||
|
<p className="mt-1 text-sm text-muted-foreground">
|
||||||
|
Claim a device to start a review session.{" "}
|
||||||
|
<button onClick={() => setPaletteOpen(true)} className="underline-offset-2 hover:underline">
|
||||||
|
⌘K
|
||||||
|
</button>{" "}
|
||||||
|
for commands.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
<Button variant="ghost" size="icon" onClick={() => setSettingsOpen(true)} aria-label="Settings">
|
||||||
|
<SettingsIcon />
|
||||||
|
</Button>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
{error && (
|
||||||
|
<div className="mb-4 rounded-md border border-destructive/40 bg-destructive/10 px-4 py-2 text-sm text-destructive">
|
||||||
|
{error}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
<div className="grid grid-cols-1 gap-4 sm:grid-cols-2 lg:grid-cols-3">
|
||||||
|
{devices.map((d) => (
|
||||||
|
<DeviceCard key={d.serial} device={d} thumbTick={thumbTick} onClaim={() => claim(d.serial)} />
|
||||||
|
))}
|
||||||
|
{devices.length === 0 && !error && (
|
||||||
|
<p className="text-sm text-muted-foreground">No devices registered.</p>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
{overlays}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,122 @@
|
|||||||
|
// Thin client over the review-api (§6.2). Types mirror the Rust models.
|
||||||
|
|
||||||
|
export interface Device {
|
||||||
|
serial: string;
|
||||||
|
model: string | null;
|
||||||
|
status: "free" | "claimed" | "offline";
|
||||||
|
stream_node: string | null;
|
||||||
|
current_build_id: string | null;
|
||||||
|
updated_at: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface Session {
|
||||||
|
id: string;
|
||||||
|
reviewer_id: string;
|
||||||
|
device_serial: string;
|
||||||
|
build_id: string | null;
|
||||||
|
started_at: string;
|
||||||
|
ended_at: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface Finding {
|
||||||
|
id: string;
|
||||||
|
session_id: string;
|
||||||
|
device_serial: string;
|
||||||
|
build_hash: string | null;
|
||||||
|
reviewer_id: string;
|
||||||
|
kind: "note" | "voice";
|
||||||
|
note_text: string | null;
|
||||||
|
audio_ref: string | null;
|
||||||
|
transcript: string | null;
|
||||||
|
frame_ref: string | null;
|
||||||
|
clip_ref: string | null;
|
||||||
|
time_offset_ms: number | null;
|
||||||
|
created_at: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function json<T>(res: Response): Promise<T> {
|
||||||
|
if (!res.ok) {
|
||||||
|
const body = await res.json().catch(() => ({}));
|
||||||
|
throw new Error(body.error ?? `${res.status} ${res.statusText}`);
|
||||||
|
}
|
||||||
|
return res.json() as Promise<T>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const api = {
|
||||||
|
listDevices: () => fetch("/api/devices").then(json<Device[]>),
|
||||||
|
|
||||||
|
createSession: (deviceSerial: string) =>
|
||||||
|
fetch("/api/sessions", {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
body: JSON.stringify({ device_serial: deviceSerial }),
|
||||||
|
}).then(json<Session>),
|
||||||
|
|
||||||
|
releaseSession: (sessionId: string) =>
|
||||||
|
fetch(`/api/sessions/${sessionId}/release`, { method: "POST" }).then((r) => {
|
||||||
|
if (!r.ok) throw new Error("release failed");
|
||||||
|
}),
|
||||||
|
|
||||||
|
// `active` marks recent interaction since the last beat; a dormant (no-interaction)
|
||||||
|
// session ages toward auto-reclaim even while its tab stays open.
|
||||||
|
heartbeat: (sessionId: string, active = false) =>
|
||||||
|
fetch(`/api/sessions/${sessionId}/heartbeat?active=${active}`, { method: "POST" })
|
||||||
|
.then((r) => r.ok)
|
||||||
|
.catch(() => false),
|
||||||
|
|
||||||
|
listFindings: (sessionId: string) =>
|
||||||
|
fetch(`/api/sessions/${sessionId}/findings`).then(json<Finding[]>),
|
||||||
|
|
||||||
|
createNote: (sessionId: string, noteText: string) =>
|
||||||
|
fetch(`/api/sessions/${sessionId}/findings`, {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
body: JSON.stringify({ kind: "note", note_text: noteText }),
|
||||||
|
}).then(json<Finding>),
|
||||||
|
|
||||||
|
updateNote: (findingId: string, noteText: string) =>
|
||||||
|
fetch(`/api/findings/${findingId}`, {
|
||||||
|
method: "PATCH",
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
body: JSON.stringify({ note_text: noteText }),
|
||||||
|
}).then(json<Finding>),
|
||||||
|
|
||||||
|
uploadAudio: (sessionId: string, blob: Blob) => {
|
||||||
|
const form = new FormData();
|
||||||
|
form.append("audio", blob, "recording.webm");
|
||||||
|
return fetch(`/api/sessions/${sessionId}/findings/audio`, {
|
||||||
|
method: "POST",
|
||||||
|
body: form,
|
||||||
|
}).then(json<Finding>);
|
||||||
|
},
|
||||||
|
|
||||||
|
uploadFrame: (findingId: string, blob: Blob) => {
|
||||||
|
const form = new FormData();
|
||||||
|
form.append("frame", blob, "frame.png");
|
||||||
|
return fetch(`/api/findings/${findingId}/frame`, {
|
||||||
|
method: "POST",
|
||||||
|
body: form,
|
||||||
|
}).then(json<Finding>);
|
||||||
|
},
|
||||||
|
|
||||||
|
// Transcribe audio without persisting — for live, interim dictation.
|
||||||
|
transcribeLive: (blob: Blob) => {
|
||||||
|
const form = new FormData();
|
||||||
|
form.append("audio", blob, "live.webm");
|
||||||
|
return fetch("/api/transcribe", { method: "POST", body: form }).then(
|
||||||
|
json<{ text: string }>,
|
||||||
|
);
|
||||||
|
},
|
||||||
|
|
||||||
|
// Create a standalone screen-snapshot finding from a captured frame.
|
||||||
|
createFrameFinding: (sessionId: string, blob: Blob) => {
|
||||||
|
const form = new FormData();
|
||||||
|
form.append("frame", blob, "frame.png");
|
||||||
|
return fetch(`/api/sessions/${sessionId}/findings/frame`, {
|
||||||
|
method: "POST",
|
||||||
|
body: form,
|
||||||
|
}).then(json<Finding>);
|
||||||
|
},
|
||||||
|
|
||||||
|
frameUrl: (findingId: string) => `/api/findings/${findingId}/frame`,
|
||||||
|
};
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
import { useEffect, type ReactNode } from "react";
|
||||||
|
import {
|
||||||
|
CommandDialog,
|
||||||
|
CommandEmpty,
|
||||||
|
CommandGroup,
|
||||||
|
CommandInput,
|
||||||
|
CommandItem,
|
||||||
|
CommandList,
|
||||||
|
} from "@/components/ui/command";
|
||||||
|
|
||||||
|
export interface CommandAction {
|
||||||
|
id: string;
|
||||||
|
label: string;
|
||||||
|
group: string;
|
||||||
|
icon?: ReactNode;
|
||||||
|
onSelect: () => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function CommandPalette({
|
||||||
|
open,
|
||||||
|
onOpenChange,
|
||||||
|
actions,
|
||||||
|
}: {
|
||||||
|
open: boolean;
|
||||||
|
onOpenChange: (open: boolean) => void;
|
||||||
|
actions: CommandAction[];
|
||||||
|
}) {
|
||||||
|
// Global ⌘K / Ctrl-K toggle.
|
||||||
|
useEffect(() => {
|
||||||
|
const onKey = (e: KeyboardEvent) => {
|
||||||
|
if (e.key === "k" && (e.metaKey || e.ctrlKey)) {
|
||||||
|
e.preventDefault();
|
||||||
|
onOpenChange(!open);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
window.addEventListener("keydown", onKey);
|
||||||
|
return () => window.removeEventListener("keydown", onKey);
|
||||||
|
}, [open, onOpenChange]);
|
||||||
|
|
||||||
|
const groups = Array.from(new Set(actions.map((a) => a.group)));
|
||||||
|
|
||||||
|
return (
|
||||||
|
<CommandDialog open={open} onOpenChange={onOpenChange}>
|
||||||
|
<CommandInput placeholder="Type a command or search…" />
|
||||||
|
<CommandList>
|
||||||
|
<CommandEmpty>No matching commands.</CommandEmpty>
|
||||||
|
{groups.map((group) => (
|
||||||
|
<CommandGroup key={group} heading={group}>
|
||||||
|
{actions
|
||||||
|
.filter((a) => a.group === group)
|
||||||
|
.map((a) => (
|
||||||
|
<CommandItem
|
||||||
|
key={a.id}
|
||||||
|
onSelect={() => {
|
||||||
|
onOpenChange(false);
|
||||||
|
a.onSelect();
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
{a.icon}
|
||||||
|
{a.label}
|
||||||
|
</CommandItem>
|
||||||
|
))}
|
||||||
|
</CommandGroup>
|
||||||
|
))}
|
||||||
|
</CommandList>
|
||||||
|
</CommandDialog>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
import { useEffect, useState } from "react";
|
||||||
|
import { MonitorSmartphone } from "lucide-react";
|
||||||
|
import { Button } from "@/components/ui/button";
|
||||||
|
import { Badge } from "@/components/ui/badge";
|
||||||
|
import { Card } from "@/components/ui/card";
|
||||||
|
import type { Device } from "../api";
|
||||||
|
|
||||||
|
// A device tile: title, a live-ish screen thumbnail (captured by streamd via adb
|
||||||
|
// screencap, refreshed on each `thumbTick`), status, and the claim action.
|
||||||
|
export function DeviceCard({
|
||||||
|
device,
|
||||||
|
thumbTick,
|
||||||
|
onClaim,
|
||||||
|
}: {
|
||||||
|
device: Device;
|
||||||
|
thumbTick: number;
|
||||||
|
onClaim: () => void;
|
||||||
|
}) {
|
||||||
|
const [errored, setErrored] = useState(false);
|
||||||
|
useEffect(() => setErrored(false), [thumbTick]); // retry on each refresh
|
||||||
|
|
||||||
|
const variant =
|
||||||
|
device.status === "free" ? "success" : device.status === "claimed" ? "warning" : "muted";
|
||||||
|
const src = `/streamd/thumbnail?serial=${encodeURIComponent(device.serial)}&t=${thumbTick}`;
|
||||||
|
const showImage = !errored && device.status !== "offline";
|
||||||
|
|
||||||
|
return (
|
||||||
|
<Card className="flex flex-col overflow-hidden">
|
||||||
|
<div className="flex items-center justify-between gap-2 px-3 py-2">
|
||||||
|
<div className="truncate font-medium">{device.model ?? device.serial}</div>
|
||||||
|
<Badge variant={variant}>{device.status}</Badge>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="flex aspect-[9/16] items-center justify-center border-y border-border bg-black">
|
||||||
|
{showImage ? (
|
||||||
|
<img
|
||||||
|
src={src}
|
||||||
|
alt={`${device.model ?? device.serial} screen`}
|
||||||
|
onError={() => setErrored(true)}
|
||||||
|
className="h-full w-full object-contain"
|
||||||
|
/>
|
||||||
|
) : (
|
||||||
|
<MonitorSmartphone className="size-10 text-muted-foreground/40" />
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="flex items-center justify-between gap-2 p-3">
|
||||||
|
<span className="truncate text-xs text-muted-foreground">{device.serial}</span>
|
||||||
|
<Button
|
||||||
|
size="sm"
|
||||||
|
disabled={device.status === "offline"}
|
||||||
|
variant={device.status === "claimed" ? "secondary" : "default"}
|
||||||
|
onClick={onClaim}
|
||||||
|
>
|
||||||
|
{device.status === "claimed" ? "Take over" : "Claim"}
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
</Card>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
import { api, type Finding } from "../api";
|
||||||
|
|
||||||
|
// Read-only view of the session's findings buffer, each showing the anchoring that
|
||||||
|
// makes it attributable (§7): build hash + session time-offset. Frame snapshots show
|
||||||
|
// a thumbnail served by the review API.
|
||||||
|
export function FindingsList({ findings }: { findings: Finding[] }) {
|
||||||
|
return (
|
||||||
|
<div className="flex h-full flex-col p-3">
|
||||||
|
<h2 className="mb-2 text-sm font-semibold text-foreground">
|
||||||
|
Findings <span className="text-muted-foreground/60">({findings.length})</span>
|
||||||
|
</h2>
|
||||||
|
<div className="flex-1 space-y-2 overflow-y-auto">
|
||||||
|
{findings.map((f) => (
|
||||||
|
<div key={f.id} className="rounded border border-border bg-card/40 p-2">
|
||||||
|
<div className="mb-1 flex items-center gap-2 text-xs text-muted-foreground">
|
||||||
|
<span className="rounded bg-muted px-1.5 py-0.5 uppercase">{f.kind}</span>
|
||||||
|
<span>build {f.build_hash ?? "—"}</span>
|
||||||
|
{f.time_offset_ms != null && <span>· t+{(f.time_offset_ms / 1000).toFixed(1)}s</span>}
|
||||||
|
</div>
|
||||||
|
{f.frame_ref && (
|
||||||
|
<img
|
||||||
|
src={api.frameUrl(f.id)}
|
||||||
|
alt="screen capture"
|
||||||
|
className="mb-1 max-h-40 w-auto rounded border border-border"
|
||||||
|
/>
|
||||||
|
)}
|
||||||
|
{f.kind === "voice" ? (
|
||||||
|
<div className="text-sm italic text-muted-foreground">{f.transcript ?? "(no transcript)"}</div>
|
||||||
|
) : (
|
||||||
|
f.note_text && <div className="text-sm text-foreground">{f.note_text}</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
{findings.length === 0 && <p className="text-sm text-muted-foreground/60">No findings yet.</p>}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,155 @@
|
|||||||
|
import { useEffect, useRef, useState } from "react";
|
||||||
|
|
||||||
|
// Live WebRTC client for the device feed (§4.2/§4.3). Connects to streamd: POSTs an
|
||||||
|
// SDP offer, renders the H.264 track in <video>, and forwards pointer/key events over
|
||||||
|
// the input WebSocket. The peer connection is owned in a ref and never recreated on
|
||||||
|
// render (§6.3). Frame capture draws the real <video> to a canvas.
|
||||||
|
const OFFER_URL = "/streamd/offer";
|
||||||
|
|
||||||
|
function inputWsUrl(): string {
|
||||||
|
const proto = location.protocol === "https:" ? "wss" : "ws";
|
||||||
|
return `${proto}://${location.host}/streamd/input`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function iceComplete(pc: RTCPeerConnection): Promise<void> {
|
||||||
|
return new Promise((resolve) => {
|
||||||
|
if (pc.iceGatheringState === "complete") return resolve();
|
||||||
|
pc.addEventListener("icegatheringstatechange", () => {
|
||||||
|
if (pc.iceGatheringState === "complete") resolve();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function LeftPane({
|
||||||
|
label,
|
||||||
|
onCapture,
|
||||||
|
onStatus,
|
||||||
|
}: {
|
||||||
|
label: string;
|
||||||
|
onCapture: (dataUrl: string) => void;
|
||||||
|
onStatus?: (status: string) => void;
|
||||||
|
}) {
|
||||||
|
const videoRef = useRef<HTMLVideoElement>(null);
|
||||||
|
const pcRef = useRef<RTCPeerConnection | null>(null);
|
||||||
|
const wsRef = useRef<WebSocket | null>(null);
|
||||||
|
const dragging = useRef(false);
|
||||||
|
const [status, setStatusRaw] = useState("connecting…");
|
||||||
|
const setStatus = (s: string) => {
|
||||||
|
setStatusRaw(s);
|
||||||
|
onStatus?.(s);
|
||||||
|
};
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
let cancelled = false;
|
||||||
|
const pc = new RTCPeerConnection();
|
||||||
|
pcRef.current = pc;
|
||||||
|
pc.addTransceiver("video", { direction: "recvonly" });
|
||||||
|
pc.ontrack = (e) => {
|
||||||
|
if (videoRef.current) {
|
||||||
|
videoRef.current.srcObject = e.streams[0];
|
||||||
|
setStatus("streaming");
|
||||||
|
}
|
||||||
|
};
|
||||||
|
pc.oniceconnectionstatechange = () => setStatus(`ice: ${pc.iceConnectionState}`);
|
||||||
|
|
||||||
|
(async () => {
|
||||||
|
await pc.setLocalDescription(await pc.createOffer());
|
||||||
|
await iceComplete(pc);
|
||||||
|
if (cancelled) return;
|
||||||
|
const res = await fetch(OFFER_URL, {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
body: JSON.stringify({ sdp: pc.localDescription!.sdp }),
|
||||||
|
});
|
||||||
|
if (!res.ok) {
|
||||||
|
setStatus(`offer failed: ${await res.text()}`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const { sdp } = await res.json();
|
||||||
|
await pc.setRemoteDescription({ type: "answer", sdp });
|
||||||
|
if (cancelled) return;
|
||||||
|
wsRef.current = new WebSocket(inputWsUrl());
|
||||||
|
})().catch((e) => setStatus(`error: ${e}`));
|
||||||
|
|
||||||
|
return () => {
|
||||||
|
cancelled = true;
|
||||||
|
wsRef.current?.close();
|
||||||
|
pcRef.current?.close();
|
||||||
|
pcRef.current = null;
|
||||||
|
wsRef.current = null;
|
||||||
|
};
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
function sendTouch(action: "down" | "move" | "up", e: React.PointerEvent<HTMLVideoElement>) {
|
||||||
|
const ws = wsRef.current;
|
||||||
|
const video = videoRef.current;
|
||||||
|
if (!ws || ws.readyState !== WebSocket.OPEN || !video) return;
|
||||||
|
const r = video.getBoundingClientRect();
|
||||||
|
const x = (e.clientX - r.left) / r.width;
|
||||||
|
const y = (e.clientY - r.top) / r.height;
|
||||||
|
if (x < 0 || x > 1 || y < 0 || y > 1) return;
|
||||||
|
ws.send(JSON.stringify({ kind: "touch", action, x, y }));
|
||||||
|
}
|
||||||
|
|
||||||
|
function sendKey(keycode: number) {
|
||||||
|
const ws = wsRef.current;
|
||||||
|
if (!ws || ws.readyState !== WebSocket.OPEN) return;
|
||||||
|
ws.send(JSON.stringify({ kind: "key", action: "down", keycode }));
|
||||||
|
ws.send(JSON.stringify({ kind: "key", action: "up", keycode }));
|
||||||
|
}
|
||||||
|
|
||||||
|
function capture() {
|
||||||
|
const video = videoRef.current;
|
||||||
|
if (!video || !video.videoWidth) return;
|
||||||
|
const canvas = document.createElement("canvas");
|
||||||
|
canvas.width = video.videoWidth;
|
||||||
|
canvas.height = video.videoHeight;
|
||||||
|
const ctx = canvas.getContext("2d");
|
||||||
|
if (!ctx) return;
|
||||||
|
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
|
||||||
|
onCapture(canvas.toDataURL("image/png"));
|
||||||
|
}
|
||||||
|
|
||||||
|
const keyBtn = "rounded-md border border-input bg-card/80 px-3 py-1.5 text-xs hover:bg-accent";
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="relative flex h-full items-center justify-center gap-3 bg-black">
|
||||||
|
<video
|
||||||
|
ref={videoRef}
|
||||||
|
autoPlay
|
||||||
|
playsInline
|
||||||
|
muted
|
||||||
|
className="h-full w-auto max-w-full rounded-lg"
|
||||||
|
style={{ touchAction: "none" }}
|
||||||
|
onPointerDown={(e) => {
|
||||||
|
dragging.current = true;
|
||||||
|
e.currentTarget.setPointerCapture(e.pointerId);
|
||||||
|
sendTouch("down", e);
|
||||||
|
}}
|
||||||
|
onPointerMove={(e) => {
|
||||||
|
if (dragging.current) sendTouch("move", e);
|
||||||
|
}}
|
||||||
|
onPointerUp={(e) => {
|
||||||
|
dragging.current = false;
|
||||||
|
sendTouch("up", e);
|
||||||
|
}}
|
||||||
|
onPointerCancel={() => {
|
||||||
|
dragging.current = false;
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
|
||||||
|
<div className="absolute left-3 top-3 rounded bg-card/80 px-2 py-1 text-xs text-muted-foreground">
|
||||||
|
{label} · {status}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="flex flex-col gap-2">
|
||||||
|
<button onClick={() => sendKey(3)} className={keyBtn}>HOME</button>
|
||||||
|
<button onClick={() => sendKey(4)} className={keyBtn}>BACK</button>
|
||||||
|
<button onClick={() => sendKey(187)} className={keyBtn}>RECENTS</button>
|
||||||
|
<button onClick={capture} className="mt-2 rounded-md bg-primary px-3 py-1.5 text-xs font-medium text-primary-foreground hover:bg-primary/90">
|
||||||
|
Capture frame
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
import { useEffect, useRef, useState } from "react";
|
||||||
|
import { api } from "../api";
|
||||||
|
|
||||||
|
type SaveState = "idle" | "saving" | "saved";
|
||||||
|
|
||||||
|
// Debounced-autosave notes surface (§4.5), controlled by the parent so voice
|
||||||
|
// dictation can flow into the same note. The first non-empty value creates a note
|
||||||
|
// finding; subsequent changes PATCH it. `interim` is the live (not-yet-committed)
|
||||||
|
// dictation transcript, shown as a preview while recording.
|
||||||
|
export function NotesPanel({
|
||||||
|
sessionId,
|
||||||
|
value,
|
||||||
|
onChange,
|
||||||
|
interim,
|
||||||
|
onSaved,
|
||||||
|
}: {
|
||||||
|
sessionId: string;
|
||||||
|
value: string;
|
||||||
|
onChange: (text: string) => void;
|
||||||
|
interim: string;
|
||||||
|
onSaved: () => void;
|
||||||
|
}) {
|
||||||
|
const [findingId, setFindingId] = useState<string | null>(null);
|
||||||
|
const [saveState, setSaveState] = useState<SaveState>("idle");
|
||||||
|
const timer = useRef<number | undefined>(undefined);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (value.trim() === "") return;
|
||||||
|
setSaveState("saving");
|
||||||
|
window.clearTimeout(timer.current);
|
||||||
|
timer.current = window.setTimeout(async () => {
|
||||||
|
try {
|
||||||
|
if (findingId) {
|
||||||
|
await api.updateNote(findingId, value);
|
||||||
|
} else {
|
||||||
|
const created = await api.createNote(sessionId, value);
|
||||||
|
setFindingId(created.id);
|
||||||
|
}
|
||||||
|
setSaveState("saved");
|
||||||
|
onSaved();
|
||||||
|
} catch {
|
||||||
|
setSaveState("idle");
|
||||||
|
}
|
||||||
|
}, 600);
|
||||||
|
return () => window.clearTimeout(timer.current);
|
||||||
|
}, [value, findingId, sessionId, onSaved]);
|
||||||
|
|
||||||
|
function newNote() {
|
||||||
|
onChange("");
|
||||||
|
setFindingId(null);
|
||||||
|
setSaveState("idle");
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="flex h-full flex-col p-3">
|
||||||
|
<div className="mb-2 flex items-center justify-between">
|
||||||
|
<h2 className="text-sm font-semibold text-foreground">Notes</h2>
|
||||||
|
<div className="flex items-center gap-3 text-xs text-muted-foreground">
|
||||||
|
<span>
|
||||||
|
{saveState === "saving" ? "saving…" : saveState === "saved" ? "saved" : ""}
|
||||||
|
</span>
|
||||||
|
<button onClick={newNote} className="rounded border border-input px-2 py-0.5 hover:bg-accent">
|
||||||
|
New note
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{interim && (
|
||||||
|
<div className="mb-2 rounded border border-success/30 bg-success/10 px-2 py-1.5 text-sm text-success">
|
||||||
|
<span className="mr-1 inline-block h-2 w-2 animate-pulse rounded-full bg-success align-middle" />
|
||||||
|
{interim}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
<textarea
|
||||||
|
value={value}
|
||||||
|
onChange={(e) => onChange(e.target.value)}
|
||||||
|
placeholder="Type a finding, or dictate with the recorder below… (autosaves)"
|
||||||
|
className="flex-1 resize-none rounded-md border border-border bg-background p-2.5 text-sm text-foreground outline-none focus:border-ring"
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,110 @@
|
|||||||
|
import { useCallback, useEffect, useState } from "react";
|
||||||
|
import { Panel, PanelGroup, PanelResizeHandle } from "react-resizable-panels";
|
||||||
|
import { toast } from "sonner";
|
||||||
|
import { TopBar } from "./TopBar";
|
||||||
|
import { api, type Device, type Finding, type Session } from "../api";
|
||||||
|
import { LeftPane } from "./LeftPane";
|
||||||
|
import { NotesPanel } from "./NotesPanel";
|
||||||
|
import { VoicePanel } from "./VoicePanel";
|
||||||
|
import { FindingsList } from "./FindingsList";
|
||||||
|
|
||||||
|
// The split-screen review surface (§4.5 / §6.3): live device left, stacked notes +
|
||||||
|
// voice review surfaces right. Owns the *findings buffer* state concern.
|
||||||
|
export function ReviewShell({
|
||||||
|
session,
|
||||||
|
device,
|
||||||
|
devices,
|
||||||
|
onRelease,
|
||||||
|
onSwitch,
|
||||||
|
onOpenSettings,
|
||||||
|
onOpenPalette,
|
||||||
|
}: {
|
||||||
|
session: Session;
|
||||||
|
device: Device | null;
|
||||||
|
devices: Device[];
|
||||||
|
onRelease: () => void;
|
||||||
|
onSwitch: (serial: string) => void;
|
||||||
|
onOpenSettings: () => void;
|
||||||
|
onOpenPalette: () => void;
|
||||||
|
}) {
|
||||||
|
const [findings, setFindings] = useState<Finding[]>([]);
|
||||||
|
// Note draft + live dictation are owned here so voice dictation can flow into notes.
|
||||||
|
const [noteText, setNoteText] = useState("");
|
||||||
|
const [interim, setInterim] = useState("");
|
||||||
|
const [connection, setConnection] = useState("connecting…");
|
||||||
|
|
||||||
|
const refreshFindings = useCallback(() => {
|
||||||
|
api.listFindings(session.id).then(setFindings).catch(() => {});
|
||||||
|
}, [session.id]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
refreshFindings();
|
||||||
|
}, [refreshFindings]);
|
||||||
|
|
||||||
|
// One-tap capture → a standalone snapshot finding of the current screen.
|
||||||
|
async function captureFrame(dataUrl: string) {
|
||||||
|
const blob = await (await fetch(dataUrl)).blob();
|
||||||
|
try {
|
||||||
|
await api.createFrameFinding(session.id, blob);
|
||||||
|
refreshFindings();
|
||||||
|
toast.success("Screenshot saved");
|
||||||
|
} catch {
|
||||||
|
toast.error("Failed to save screenshot");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Final dictation gets appended to the note text; the interim preview clears.
|
||||||
|
function commitDictation(text: string) {
|
||||||
|
setNoteText((prev) => (prev.trim() ? `${prev.trimEnd()} ${text}` : text));
|
||||||
|
setInterim("");
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="flex h-screen flex-col">
|
||||||
|
<TopBar
|
||||||
|
session={session}
|
||||||
|
device={device}
|
||||||
|
devices={devices}
|
||||||
|
connection={connection}
|
||||||
|
onSwitch={onSwitch}
|
||||||
|
onRelease={onRelease}
|
||||||
|
onOpenSettings={onOpenSettings}
|
||||||
|
onOpenPalette={onOpenPalette}
|
||||||
|
/>
|
||||||
|
|
||||||
|
<PanelGroup direction="horizontal" className="flex-1">
|
||||||
|
<Panel defaultSize={62} minSize={35}>
|
||||||
|
<LeftPane
|
||||||
|
label={device?.model ?? session.device_serial}
|
||||||
|
onCapture={captureFrame}
|
||||||
|
onStatus={setConnection}
|
||||||
|
/>
|
||||||
|
</Panel>
|
||||||
|
|
||||||
|
<PanelResizeHandle className="w-1 bg-border transition-colors hover:bg-primary" />
|
||||||
|
|
||||||
|
<Panel defaultSize={38} minSize={25}>
|
||||||
|
<PanelGroup direction="vertical">
|
||||||
|
<Panel defaultSize={45} minSize={20}>
|
||||||
|
<NotesPanel
|
||||||
|
sessionId={session.id}
|
||||||
|
value={noteText}
|
||||||
|
onChange={setNoteText}
|
||||||
|
interim={interim}
|
||||||
|
onSaved={refreshFindings}
|
||||||
|
/>
|
||||||
|
</Panel>
|
||||||
|
<PanelResizeHandle className="h-1 bg-border transition-colors hover:bg-primary" />
|
||||||
|
<Panel defaultSize={30} minSize={15}>
|
||||||
|
<VoicePanel onInterim={setInterim} onFinal={commitDictation} />
|
||||||
|
</Panel>
|
||||||
|
<PanelResizeHandle className="h-1 bg-border transition-colors hover:bg-primary" />
|
||||||
|
<Panel defaultSize={25} minSize={10}>
|
||||||
|
<FindingsList findings={findings} />
|
||||||
|
</Panel>
|
||||||
|
</PanelGroup>
|
||||||
|
</Panel>
|
||||||
|
</PanelGroup>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
import {
|
||||||
|
Dialog,
|
||||||
|
DialogContent,
|
||||||
|
DialogDescription,
|
||||||
|
DialogHeader,
|
||||||
|
DialogTitle,
|
||||||
|
} from "@/components/ui/dialog";
|
||||||
|
import { Button } from "@/components/ui/button";
|
||||||
|
import { useSettings, type Accent, type Quality } from "@/lib/settings";
|
||||||
|
import { cn } from "@/lib/utils";
|
||||||
|
|
||||||
|
const LANGUAGES: { value: string; label: string }[] = [
|
||||||
|
{ value: "auto", label: "Auto-detect" },
|
||||||
|
{ value: "en", label: "English" },
|
||||||
|
{ value: "tr", label: "Turkish" },
|
||||||
|
{ value: "id", label: "Indonesian" },
|
||||||
|
{ value: "ar", label: "Arabic" },
|
||||||
|
{ value: "es", label: "Spanish" },
|
||||||
|
{ value: "fr", label: "French" },
|
||||||
|
{ value: "zh", label: "Chinese" },
|
||||||
|
];
|
||||||
|
|
||||||
|
const QUALITIES: Quality[] = ["high", "balanced", "low"];
|
||||||
|
const ACCENTS: Accent[] = ["emerald", "sky", "violet", "amber"];
|
||||||
|
const ACCENT_SWATCH: Record<Accent, string> = {
|
||||||
|
emerald: "#10b981",
|
||||||
|
sky: "#0ea5e9",
|
||||||
|
violet: "#8b5cf6",
|
||||||
|
amber: "#f59e0b",
|
||||||
|
};
|
||||||
|
|
||||||
|
function Row({ label, children }: { label: string; children: React.ReactNode }) {
|
||||||
|
return (
|
||||||
|
<div className="flex items-center justify-between gap-4 py-2">
|
||||||
|
<span className="text-sm text-muted-foreground">{label}</span>
|
||||||
|
<div className="flex items-center gap-1.5">{children}</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function SettingsDialog({
|
||||||
|
open,
|
||||||
|
onOpenChange,
|
||||||
|
}: {
|
||||||
|
open: boolean;
|
||||||
|
onOpenChange: (open: boolean) => void;
|
||||||
|
}) {
|
||||||
|
const { settings, update } = useSettings();
|
||||||
|
|
||||||
|
return (
|
||||||
|
<Dialog open={open} onOpenChange={onOpenChange}>
|
||||||
|
<DialogContent>
|
||||||
|
<DialogHeader>
|
||||||
|
<DialogTitle>Settings</DialogTitle>
|
||||||
|
<DialogDescription>Preferences are saved in this browser.</DialogDescription>
|
||||||
|
</DialogHeader>
|
||||||
|
|
||||||
|
<div className="divide-y divide-border">
|
||||||
|
<Row label="Accent">
|
||||||
|
{ACCENTS.map((a) => (
|
||||||
|
<button
|
||||||
|
key={a}
|
||||||
|
onClick={() => update({ accent: a })}
|
||||||
|
aria-label={a}
|
||||||
|
className={cn(
|
||||||
|
"size-6 rounded-full border-2 transition-transform hover:scale-110",
|
||||||
|
settings.accent === a ? "border-foreground" : "border-transparent",
|
||||||
|
)}
|
||||||
|
style={{ backgroundColor: ACCENT_SWATCH[a] }}
|
||||||
|
/>
|
||||||
|
))}
|
||||||
|
</Row>
|
||||||
|
|
||||||
|
<Row label="Transcription language">
|
||||||
|
<select
|
||||||
|
value={settings.language}
|
||||||
|
onChange={(e) => update({ language: e.target.value })}
|
||||||
|
className="h-8 rounded-md border border-input bg-background px-2 text-sm text-foreground outline-none focus-visible:ring-2 focus-visible:ring-ring"
|
||||||
|
>
|
||||||
|
{LANGUAGES.map((l) => (
|
||||||
|
<option key={l.value} value={l.value}>
|
||||||
|
{l.label}
|
||||||
|
</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
</Row>
|
||||||
|
|
||||||
|
<Row label="Stream quality">
|
||||||
|
{QUALITIES.map((q) => (
|
||||||
|
<Button
|
||||||
|
key={q}
|
||||||
|
size="sm"
|
||||||
|
variant={settings.quality === q ? "default" : "outline"}
|
||||||
|
onClick={() => update({ quality: q })}
|
||||||
|
className="capitalize"
|
||||||
|
>
|
||||||
|
{q}
|
||||||
|
</Button>
|
||||||
|
))}
|
||||||
|
</Row>
|
||||||
|
</div>
|
||||||
|
</DialogContent>
|
||||||
|
</Dialog>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
import { useEffect, useState } from "react";
|
||||||
|
import { ChevronDown, Settings, Command as CommandIcon, LogOut } from "lucide-react";
|
||||||
|
import { Button } from "@/components/ui/button";
|
||||||
|
import { Badge } from "@/components/ui/badge";
|
||||||
|
import {
|
||||||
|
DropdownMenu,
|
||||||
|
DropdownMenuContent,
|
||||||
|
DropdownMenuItem,
|
||||||
|
DropdownMenuLabel,
|
||||||
|
DropdownMenuSeparator,
|
||||||
|
DropdownMenuTrigger,
|
||||||
|
} from "@/components/ui/dropdown-menu";
|
||||||
|
import type { Device, Session } from "../api";
|
||||||
|
|
||||||
|
function elapsed(since: string): string {
|
||||||
|
const secs = Math.max(0, Math.floor((Date.now() - new Date(since).getTime()) / 1000));
|
||||||
|
const m = Math.floor(secs / 60);
|
||||||
|
const s = secs % 60;
|
||||||
|
return `${m}:${s.toString().padStart(2, "0")}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function TopBar({
|
||||||
|
session,
|
||||||
|
device,
|
||||||
|
devices,
|
||||||
|
connection,
|
||||||
|
onSwitch,
|
||||||
|
onRelease,
|
||||||
|
onOpenSettings,
|
||||||
|
onOpenPalette,
|
||||||
|
}: {
|
||||||
|
session: Session;
|
||||||
|
device: Device | null;
|
||||||
|
devices: Device[];
|
||||||
|
connection: string;
|
||||||
|
onSwitch: (serial: string) => void;
|
||||||
|
onRelease: () => void;
|
||||||
|
onOpenSettings: () => void;
|
||||||
|
onOpenPalette: () => void;
|
||||||
|
}) {
|
||||||
|
const [, tick] = useState(0);
|
||||||
|
useEffect(() => {
|
||||||
|
const t = window.setInterval(() => tick((n) => n + 1), 1000);
|
||||||
|
return () => window.clearInterval(t);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const live = connection === "streaming" || connection === "connected" || connection === "ice: connected";
|
||||||
|
|
||||||
|
return (
|
||||||
|
<header className="flex items-center justify-between border-b border-border px-3 py-2">
|
||||||
|
<div className="flex items-center gap-3">
|
||||||
|
<span className="font-semibold">ClawReview</span>
|
||||||
|
|
||||||
|
<DropdownMenu>
|
||||||
|
<DropdownMenuTrigger asChild>
|
||||||
|
<Button variant="outline" size="sm" className="gap-1.5">
|
||||||
|
{device?.model ?? session.device_serial}
|
||||||
|
<ChevronDown />
|
||||||
|
</Button>
|
||||||
|
</DropdownMenuTrigger>
|
||||||
|
<DropdownMenuContent align="start">
|
||||||
|
<DropdownMenuLabel>Switch device</DropdownMenuLabel>
|
||||||
|
{devices.map((d) => (
|
||||||
|
<DropdownMenuItem
|
||||||
|
key={d.serial}
|
||||||
|
disabled={d.serial === session.device_serial || d.status === "offline"}
|
||||||
|
onSelect={() => onSwitch(d.serial)}
|
||||||
|
>
|
||||||
|
<span className="flex-1">{d.model ?? d.serial}</span>
|
||||||
|
<Badge variant={d.status === "free" ? "success" : d.status === "claimed" ? "warning" : "muted"}>
|
||||||
|
{d.status}
|
||||||
|
</Badge>
|
||||||
|
</DropdownMenuItem>
|
||||||
|
))}
|
||||||
|
<DropdownMenuSeparator />
|
||||||
|
<DropdownMenuItem onSelect={onRelease}>
|
||||||
|
<LogOut /> Release device
|
||||||
|
</DropdownMenuItem>
|
||||||
|
</DropdownMenuContent>
|
||||||
|
</DropdownMenu>
|
||||||
|
|
||||||
|
<span className="text-xs text-muted-foreground">session {session.id.slice(0, 8)}</span>
|
||||||
|
<span className="font-mono text-xs text-muted-foreground tabular-nums">{elapsed(session.started_at)}</span>
|
||||||
|
<Badge variant={live ? "success" : "muted"} className="gap-1.5">
|
||||||
|
<span className={`size-1.5 rounded-full ${live ? "bg-success" : "bg-muted-foreground"}`} />
|
||||||
|
{connection}
|
||||||
|
</Badge>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="flex items-center gap-2">
|
||||||
|
<Button variant="ghost" size="sm" onClick={onOpenPalette} className="gap-1.5 text-muted-foreground">
|
||||||
|
<CommandIcon /> <span className="hidden sm:inline">⌘K</span>
|
||||||
|
</Button>
|
||||||
|
<Button variant="ghost" size="icon" onClick={onOpenSettings} aria-label="Settings">
|
||||||
|
<Settings />
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,112 @@
|
|||||||
|
import { useRef, useState } from "react";
|
||||||
|
import { api } from "../api";
|
||||||
|
|
||||||
|
type Phase = "idle" | "recording" | "finalizing";
|
||||||
|
|
||||||
|
// Voice dictation (§4.5): captures mic audio with MediaRecorder and transcribes it
|
||||||
|
// live on RedClaw infrastructure (the self-hosted Whisper service via /api/transcribe).
|
||||||
|
// The growing audio is re-transcribed every few seconds for an interim preview; on
|
||||||
|
// stop, the final transcript is committed into the notes above.
|
||||||
|
const INTERIM_MS = 3000;
|
||||||
|
|
||||||
|
export function VoicePanel({
|
||||||
|
onInterim,
|
||||||
|
onFinal,
|
||||||
|
}: {
|
||||||
|
onInterim: (text: string) => void;
|
||||||
|
onFinal: (text: string) => void;
|
||||||
|
}) {
|
||||||
|
const [phase, setPhase] = useState<Phase>("idle");
|
||||||
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
const recorder = useRef<MediaRecorder | null>(null);
|
||||||
|
const chunks = useRef<Blob[]>([]);
|
||||||
|
const interval = useRef<number | undefined>(undefined);
|
||||||
|
|
||||||
|
async function transcribe(): Promise<string> {
|
||||||
|
const blob = new Blob(chunks.current, { type: "audio/webm" });
|
||||||
|
const { text } = await api.transcribeLive(blob);
|
||||||
|
return text.trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
async function start() {
|
||||||
|
setError(null);
|
||||||
|
// getUserMedia only exists in a secure context (HTTPS or http://localhost). Over
|
||||||
|
// plain HTTP on a LAN/Tailscale IP the browser withholds navigator.mediaDevices.
|
||||||
|
if (!navigator.mediaDevices?.getUserMedia) {
|
||||||
|
setError(
|
||||||
|
"Microphone needs a secure context. Open the app over HTTPS (https://<host>.ts.net) or http://localhost.",
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
|
||||||
|
const rec = new MediaRecorder(stream, { mimeType: "audio/webm" });
|
||||||
|
chunks.current = [];
|
||||||
|
rec.ondataavailable = (e) => {
|
||||||
|
if (e.data.size > 0) chunks.current.push(e.data);
|
||||||
|
};
|
||||||
|
rec.onstop = async () => {
|
||||||
|
window.clearInterval(interval.current);
|
||||||
|
stream.getTracks().forEach((t) => t.stop());
|
||||||
|
setPhase("finalizing");
|
||||||
|
try {
|
||||||
|
const text = await transcribe();
|
||||||
|
onInterim("");
|
||||||
|
if (text) onFinal(text);
|
||||||
|
} catch (e) {
|
||||||
|
setError(String(e));
|
||||||
|
onInterim("");
|
||||||
|
}
|
||||||
|
setPhase("idle");
|
||||||
|
};
|
||||||
|
|
||||||
|
rec.start(1000); // 1s timeslice so chunks accumulate for interim passes
|
||||||
|
recorder.current = rec;
|
||||||
|
setPhase("recording");
|
||||||
|
|
||||||
|
interval.current = window.setInterval(async () => {
|
||||||
|
if (chunks.current.length === 0) return;
|
||||||
|
try {
|
||||||
|
onInterim(await transcribe());
|
||||||
|
} catch {
|
||||||
|
/* a partial buffer may fail to decode; the next pass retries */
|
||||||
|
}
|
||||||
|
}, INTERIM_MS);
|
||||||
|
} catch (e) {
|
||||||
|
setError(`mic unavailable: ${e}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function stop() {
|
||||||
|
recorder.current?.stop();
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="flex h-full flex-col p-3">
|
||||||
|
<h2 className="mb-2 text-sm font-semibold text-foreground">
|
||||||
|
Voice <span className="text-muted-foreground/60">— dictate into notes</span>
|
||||||
|
</h2>
|
||||||
|
|
||||||
|
<div className="flex items-center gap-3">
|
||||||
|
{phase === "idle" && (
|
||||||
|
<button onClick={start} className="rounded-md bg-destructive px-3 py-1.5 text-sm font-medium text-white hover:bg-destructive/90">
|
||||||
|
● Record
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
{phase === "recording" && (
|
||||||
|
<>
|
||||||
|
<button onClick={stop} className="rounded-md bg-secondary px-3 py-1.5 text-sm font-medium text-secondary-foreground hover:bg-secondary/80">
|
||||||
|
■ Stop
|
||||||
|
</button>
|
||||||
|
<span className="flex items-center gap-1.5 text-sm text-destructive">
|
||||||
|
<span className="h-2 w-2 animate-pulse rounded-full bg-destructive" /> recording — words appear in Notes
|
||||||
|
</span>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
{phase === "finalizing" && <span className="text-sm text-muted-foreground">finalizing transcript…</span>}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{error && <div className="mt-2 text-xs text-destructive">{error}</div>}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
import * as React from "react";
|
||||||
|
import { cva, type VariantProps } from "class-variance-authority";
|
||||||
|
import { cn } from "@/lib/utils";
|
||||||
|
|
||||||
|
const badgeVariants = cva(
|
||||||
|
"inline-flex items-center rounded-full border px-2 py-0.5 text-xs font-medium transition-colors",
|
||||||
|
{
|
||||||
|
variants: {
|
||||||
|
variant: {
|
||||||
|
default: "border-transparent bg-primary/15 text-primary",
|
||||||
|
secondary: "border-transparent bg-secondary text-secondary-foreground",
|
||||||
|
outline: "border-border text-foreground",
|
||||||
|
success: "border-success/30 bg-success/10 text-success",
|
||||||
|
warning: "border-warning/30 bg-warning/10 text-warning",
|
||||||
|
destructive: "border-destructive/30 bg-destructive/10 text-destructive",
|
||||||
|
muted: "border-transparent bg-muted text-muted-foreground",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
defaultVariants: { variant: "default" },
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
export interface BadgeProps
|
||||||
|
extends React.HTMLAttributes<HTMLSpanElement>,
|
||||||
|
VariantProps<typeof badgeVariants> {}
|
||||||
|
|
||||||
|
export function Badge({ className, variant, ...props }: BadgeProps) {
|
||||||
|
return <span className={cn(badgeVariants({ variant }), className)} {...props} />;
|
||||||
|
}
|
||||||
|
|
||||||
|
export { badgeVariants };
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
import * as React from "react";
|
||||||
|
import { Slot } from "@radix-ui/react-slot";
|
||||||
|
import { cva, type VariantProps } from "class-variance-authority";
|
||||||
|
import { cn } from "@/lib/utils";
|
||||||
|
|
||||||
|
const buttonVariants = cva(
|
||||||
|
"inline-flex items-center justify-center gap-2 whitespace-nowrap rounded-md text-sm font-medium transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring disabled:pointer-events-none disabled:opacity-50 [&_svg]:size-4 [&_svg]:shrink-0",
|
||||||
|
{
|
||||||
|
variants: {
|
||||||
|
variant: {
|
||||||
|
default: "bg-primary text-primary-foreground hover:bg-primary/90",
|
||||||
|
destructive: "bg-destructive text-destructive-foreground hover:bg-destructive/90",
|
||||||
|
outline: "border border-input bg-transparent hover:bg-accent hover:text-accent-foreground",
|
||||||
|
secondary: "bg-secondary text-secondary-foreground hover:bg-secondary/80",
|
||||||
|
ghost: "hover:bg-accent hover:text-accent-foreground",
|
||||||
|
link: "text-primary underline-offset-4 hover:underline",
|
||||||
|
},
|
||||||
|
size: {
|
||||||
|
default: "h-9 px-4 py-2",
|
||||||
|
sm: "h-8 rounded-md px-3 text-xs",
|
||||||
|
lg: "h-10 rounded-md px-6",
|
||||||
|
icon: "h-9 w-9",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
defaultVariants: { variant: "default", size: "default" },
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
export interface ButtonProps
|
||||||
|
extends React.ButtonHTMLAttributes<HTMLButtonElement>,
|
||||||
|
VariantProps<typeof buttonVariants> {
|
||||||
|
asChild?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const Button = React.forwardRef<HTMLButtonElement, ButtonProps>(
|
||||||
|
({ className, variant, size, asChild = false, ...props }, ref) => {
|
||||||
|
const Comp = asChild ? Slot : "button";
|
||||||
|
return <Comp ref={ref} className={cn(buttonVariants({ variant, size, className }))} {...props} />;
|
||||||
|
},
|
||||||
|
);
|
||||||
|
Button.displayName = "Button";
|
||||||
|
|
||||||
|
export { buttonVariants };
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
import * as React from "react";
|
||||||
|
import { cn } from "@/lib/utils";
|
||||||
|
|
||||||
|
export const Card = React.forwardRef<HTMLDivElement, React.HTMLAttributes<HTMLDivElement>>(
|
||||||
|
({ className, ...props }, ref) => (
|
||||||
|
<div
|
||||||
|
ref={ref}
|
||||||
|
className={cn("rounded-lg border border-border bg-card text-card-foreground shadow-sm", className)}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
),
|
||||||
|
);
|
||||||
|
Card.displayName = "Card";
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
import * as React from "react";
|
||||||
|
import { Command as CommandPrimitive } from "cmdk";
|
||||||
|
import { Search } from "lucide-react";
|
||||||
|
import { Dialog, DialogContent } from "@/components/ui/dialog";
|
||||||
|
import { cn } from "@/lib/utils";
|
||||||
|
|
||||||
|
export const Command = React.forwardRef<
|
||||||
|
React.ElementRef<typeof CommandPrimitive>,
|
||||||
|
React.ComponentPropsWithoutRef<typeof CommandPrimitive>
|
||||||
|
>(({ className, ...props }, ref) => (
|
||||||
|
<CommandPrimitive
|
||||||
|
ref={ref}
|
||||||
|
className={cn(
|
||||||
|
"flex h-full w-full flex-col overflow-hidden rounded-md bg-popover text-popover-foreground",
|
||||||
|
className,
|
||||||
|
)}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
));
|
||||||
|
Command.displayName = CommandPrimitive.displayName;
|
||||||
|
|
||||||
|
export function CommandDialog({
|
||||||
|
children,
|
||||||
|
...props
|
||||||
|
}: React.ComponentProps<typeof Dialog>) {
|
||||||
|
return (
|
||||||
|
<Dialog {...props}>
|
||||||
|
<DialogContent hideClose className="overflow-hidden p-0">
|
||||||
|
<Command className="[&_[cmdk-group-heading]]:px-2 [&_[cmdk-group-heading]]:py-1.5 [&_[cmdk-group-heading]]:text-xs [&_[cmdk-group-heading]]:font-medium [&_[cmdk-group-heading]]:text-muted-foreground [&_[cmdk-item]]:px-2 [&_[cmdk-item]]:py-2">
|
||||||
|
{children}
|
||||||
|
</Command>
|
||||||
|
</DialogContent>
|
||||||
|
</Dialog>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export const CommandInput = React.forwardRef<
|
||||||
|
React.ElementRef<typeof CommandPrimitive.Input>,
|
||||||
|
React.ComponentPropsWithoutRef<typeof CommandPrimitive.Input>
|
||||||
|
>(({ className, ...props }, ref) => (
|
||||||
|
<div className="flex items-center border-b border-border px-3">
|
||||||
|
<Search className="mr-2 size-4 shrink-0 text-muted-foreground" />
|
||||||
|
<CommandPrimitive.Input
|
||||||
|
ref={ref}
|
||||||
|
className={cn(
|
||||||
|
"flex h-11 w-full bg-transparent py-3 text-sm text-foreground outline-none placeholder:text-muted-foreground disabled:cursor-not-allowed disabled:opacity-50",
|
||||||
|
className,
|
||||||
|
)}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
));
|
||||||
|
CommandInput.displayName = CommandPrimitive.Input.displayName;
|
||||||
|
|
||||||
|
export const CommandList = React.forwardRef<
|
||||||
|
React.ElementRef<typeof CommandPrimitive.List>,
|
||||||
|
React.ComponentPropsWithoutRef<typeof CommandPrimitive.List>
|
||||||
|
>(({ className, ...props }, ref) => (
|
||||||
|
<CommandPrimitive.List
|
||||||
|
ref={ref}
|
||||||
|
className={cn("max-h-80 overflow-y-auto overflow-x-hidden p-1", className)}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
));
|
||||||
|
CommandList.displayName = CommandPrimitive.List.displayName;
|
||||||
|
|
||||||
|
export const CommandEmpty = React.forwardRef<
|
||||||
|
React.ElementRef<typeof CommandPrimitive.Empty>,
|
||||||
|
React.ComponentPropsWithoutRef<typeof CommandPrimitive.Empty>
|
||||||
|
>((props, ref) => (
|
||||||
|
<CommandPrimitive.Empty
|
||||||
|
ref={ref}
|
||||||
|
className="py-6 text-center text-sm text-muted-foreground"
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
));
|
||||||
|
CommandEmpty.displayName = CommandPrimitive.Empty.displayName;
|
||||||
|
|
||||||
|
export const CommandGroup = React.forwardRef<
|
||||||
|
React.ElementRef<typeof CommandPrimitive.Group>,
|
||||||
|
React.ComponentPropsWithoutRef<typeof CommandPrimitive.Group>
|
||||||
|
>(({ className, ...props }, ref) => (
|
||||||
|
<CommandPrimitive.Group ref={ref} className={cn("overflow-hidden p-1", className)} {...props} />
|
||||||
|
));
|
||||||
|
CommandGroup.displayName = CommandPrimitive.Group.displayName;
|
||||||
|
|
||||||
|
export const CommandItem = React.forwardRef<
|
||||||
|
React.ElementRef<typeof CommandPrimitive.Item>,
|
||||||
|
React.ComponentPropsWithoutRef<typeof CommandPrimitive.Item>
|
||||||
|
>(({ className, ...props }, ref) => (
|
||||||
|
<CommandPrimitive.Item
|
||||||
|
ref={ref}
|
||||||
|
className={cn(
|
||||||
|
"relative flex cursor-pointer select-none items-center gap-2 rounded-sm px-2 py-2 text-sm text-foreground outline-none data-[selected=true]:bg-accent data-[selected=true]:text-accent-foreground data-[disabled=true]:pointer-events-none data-[disabled=true]:opacity-50 [&_svg]:size-4 [&_svg]:text-muted-foreground",
|
||||||
|
className,
|
||||||
|
)}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
));
|
||||||
|
CommandItem.displayName = CommandPrimitive.Item.displayName;
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
import * as React from "react";
|
||||||
|
import * as DialogPrimitive from "@radix-ui/react-dialog";
|
||||||
|
import { X } from "lucide-react";
|
||||||
|
import { cn } from "@/lib/utils";
|
||||||
|
|
||||||
|
export const Dialog = DialogPrimitive.Root;
|
||||||
|
export const DialogTrigger = DialogPrimitive.Trigger;
|
||||||
|
export const DialogClose = DialogPrimitive.Close;
|
||||||
|
|
||||||
|
const DialogOverlay = React.forwardRef<
|
||||||
|
React.ElementRef<typeof DialogPrimitive.Overlay>,
|
||||||
|
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Overlay>
|
||||||
|
>(({ className, ...props }, ref) => (
|
||||||
|
<DialogPrimitive.Overlay
|
||||||
|
ref={ref}
|
||||||
|
className={cn(
|
||||||
|
"fixed inset-0 z-50 bg-black/70 backdrop-blur-sm animate-in fade-in-0",
|
||||||
|
className,
|
||||||
|
)}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
));
|
||||||
|
DialogOverlay.displayName = DialogPrimitive.Overlay.displayName;
|
||||||
|
|
||||||
|
export const DialogContent = React.forwardRef<
|
||||||
|
React.ElementRef<typeof DialogPrimitive.Content>,
|
||||||
|
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Content> & { hideClose?: boolean }
|
||||||
|
>(({ className, children, hideClose, ...props }, ref) => (
|
||||||
|
<DialogPrimitive.Portal>
|
||||||
|
<DialogOverlay />
|
||||||
|
<DialogPrimitive.Content
|
||||||
|
ref={ref}
|
||||||
|
className={cn(
|
||||||
|
"fixed left-1/2 top-1/2 z-50 grid w-full max-w-lg -translate-x-1/2 -translate-y-1/2 gap-4 rounded-lg border border-border bg-card p-6 shadow-lg animate-in fade-in-0 zoom-in-95",
|
||||||
|
className,
|
||||||
|
)}
|
||||||
|
{...props}
|
||||||
|
>
|
||||||
|
{children}
|
||||||
|
{!hideClose && (
|
||||||
|
<DialogPrimitive.Close className="absolute right-4 top-4 rounded-sm text-muted-foreground opacity-70 transition-opacity hover:opacity-100 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring">
|
||||||
|
<X className="size-4" />
|
||||||
|
<span className="sr-only">Close</span>
|
||||||
|
</DialogPrimitive.Close>
|
||||||
|
)}
|
||||||
|
</DialogPrimitive.Content>
|
||||||
|
</DialogPrimitive.Portal>
|
||||||
|
));
|
||||||
|
DialogContent.displayName = DialogPrimitive.Content.displayName;
|
||||||
|
|
||||||
|
export function DialogHeader({ className, ...props }: React.HTMLAttributes<HTMLDivElement>) {
|
||||||
|
return <div className={cn("flex flex-col gap-1.5", className)} {...props} />;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const DialogTitle = React.forwardRef<
|
||||||
|
React.ElementRef<typeof DialogPrimitive.Title>,
|
||||||
|
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Title>
|
||||||
|
>(({ className, ...props }, ref) => (
|
||||||
|
<DialogPrimitive.Title
|
||||||
|
ref={ref}
|
||||||
|
className={cn("text-lg font-semibold text-foreground", className)}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
));
|
||||||
|
DialogTitle.displayName = DialogPrimitive.Title.displayName;
|
||||||
|
|
||||||
|
export const DialogDescription = React.forwardRef<
|
||||||
|
React.ElementRef<typeof DialogPrimitive.Description>,
|
||||||
|
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Description>
|
||||||
|
>(({ className, ...props }, ref) => (
|
||||||
|
<DialogPrimitive.Description
|
||||||
|
ref={ref}
|
||||||
|
className={cn("text-sm text-muted-foreground", className)}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
));
|
||||||
|
DialogDescription.displayName = DialogPrimitive.Description.displayName;
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
import * as React from "react";
|
||||||
|
import * as DropdownMenuPrimitive from "@radix-ui/react-dropdown-menu";
|
||||||
|
import { Check } from "lucide-react";
|
||||||
|
import { cn } from "@/lib/utils";
|
||||||
|
|
||||||
|
export const DropdownMenu = DropdownMenuPrimitive.Root;
|
||||||
|
export const DropdownMenuTrigger = DropdownMenuPrimitive.Trigger;
|
||||||
|
|
||||||
|
export const DropdownMenuContent = React.forwardRef<
|
||||||
|
React.ElementRef<typeof DropdownMenuPrimitive.Content>,
|
||||||
|
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Content>
|
||||||
|
>(({ className, sideOffset = 4, ...props }, ref) => (
|
||||||
|
<DropdownMenuPrimitive.Portal>
|
||||||
|
<DropdownMenuPrimitive.Content
|
||||||
|
ref={ref}
|
||||||
|
sideOffset={sideOffset}
|
||||||
|
className={cn(
|
||||||
|
"z-50 min-w-[12rem] overflow-hidden rounded-md border border-border bg-popover p-1 text-popover-foreground shadow-md animate-in fade-in-0 zoom-in-95",
|
||||||
|
className,
|
||||||
|
)}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
</DropdownMenuPrimitive.Portal>
|
||||||
|
));
|
||||||
|
DropdownMenuContent.displayName = DropdownMenuPrimitive.Content.displayName;
|
||||||
|
|
||||||
|
export const DropdownMenuItem = React.forwardRef<
|
||||||
|
React.ElementRef<typeof DropdownMenuPrimitive.Item>,
|
||||||
|
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Item> & { inset?: boolean }
|
||||||
|
>(({ className, inset, ...props }, ref) => (
|
||||||
|
<DropdownMenuPrimitive.Item
|
||||||
|
ref={ref}
|
||||||
|
className={cn(
|
||||||
|
"relative flex cursor-pointer select-none items-center gap-2 rounded-sm px-2 py-1.5 text-sm outline-none transition-colors focus:bg-accent focus:text-accent-foreground data-[disabled]:pointer-events-none data-[disabled]:opacity-50 [&_svg]:size-4 [&_svg]:text-muted-foreground",
|
||||||
|
inset && "pl-8",
|
||||||
|
className,
|
||||||
|
)}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
));
|
||||||
|
DropdownMenuItem.displayName = DropdownMenuPrimitive.Item.displayName;
|
||||||
|
|
||||||
|
export const DropdownMenuCheckboxItem = React.forwardRef<
|
||||||
|
React.ElementRef<typeof DropdownMenuPrimitive.CheckboxItem>,
|
||||||
|
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.CheckboxItem>
|
||||||
|
>(({ className, children, checked, ...props }, ref) => (
|
||||||
|
<DropdownMenuPrimitive.CheckboxItem
|
||||||
|
ref={ref}
|
||||||
|
checked={checked}
|
||||||
|
className={cn(
|
||||||
|
"relative flex cursor-pointer select-none items-center rounded-sm py-1.5 pl-8 pr-2 text-sm outline-none transition-colors focus:bg-accent focus:text-accent-foreground",
|
||||||
|
className,
|
||||||
|
)}
|
||||||
|
{...props}
|
||||||
|
>
|
||||||
|
<span className="absolute left-2 flex size-4 items-center justify-center">
|
||||||
|
<DropdownMenuPrimitive.ItemIndicator>
|
||||||
|
<Check className="size-4" />
|
||||||
|
</DropdownMenuPrimitive.ItemIndicator>
|
||||||
|
</span>
|
||||||
|
{children}
|
||||||
|
</DropdownMenuPrimitive.CheckboxItem>
|
||||||
|
));
|
||||||
|
DropdownMenuCheckboxItem.displayName = DropdownMenuPrimitive.CheckboxItem.displayName;
|
||||||
|
|
||||||
|
export function DropdownMenuLabel({
|
||||||
|
className,
|
||||||
|
...props
|
||||||
|
}: React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Label>) {
|
||||||
|
return (
|
||||||
|
<DropdownMenuPrimitive.Label
|
||||||
|
className={cn("px-2 py-1.5 text-xs font-medium text-muted-foreground", className)}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function DropdownMenuSeparator({
|
||||||
|
className,
|
||||||
|
...props
|
||||||
|
}: React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Separator>) {
|
||||||
|
return (
|
||||||
|
<DropdownMenuPrimitive.Separator className={cn("-mx-1 my-1 h-px bg-border", className)} {...props} />
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
import * as React from "react";
|
||||||
|
import { cn } from "@/lib/utils";
|
||||||
|
|
||||||
|
export const Input = React.forwardRef<HTMLInputElement, React.InputHTMLAttributes<HTMLInputElement>>(
|
||||||
|
({ className, type, ...props }, ref) => (
|
||||||
|
<input
|
||||||
|
type={type}
|
||||||
|
ref={ref}
|
||||||
|
className={cn(
|
||||||
|
"flex h-9 w-full rounded-md border border-input bg-transparent px-3 py-1 text-sm text-foreground shadow-sm transition-colors placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring disabled:cursor-not-allowed disabled:opacity-50",
|
||||||
|
className,
|
||||||
|
)}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
),
|
||||||
|
);
|
||||||
|
Input.displayName = "Input";
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
import * as React from "react";
|
||||||
|
import * as SeparatorPrimitive from "@radix-ui/react-separator";
|
||||||
|
import { cn } from "@/lib/utils";
|
||||||
|
|
||||||
|
export const Separator = React.forwardRef<
|
||||||
|
React.ElementRef<typeof SeparatorPrimitive.Root>,
|
||||||
|
React.ComponentPropsWithoutRef<typeof SeparatorPrimitive.Root>
|
||||||
|
>(({ className, orientation = "horizontal", decorative = true, ...props }, ref) => (
|
||||||
|
<SeparatorPrimitive.Root
|
||||||
|
ref={ref}
|
||||||
|
decorative={decorative}
|
||||||
|
orientation={orientation}
|
||||||
|
className={cn(
|
||||||
|
"shrink-0 bg-border",
|
||||||
|
orientation === "horizontal" ? "h-px w-full" : "h-full w-px",
|
||||||
|
className,
|
||||||
|
)}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
));
|
||||||
|
Separator.displayName = SeparatorPrimitive.Root.displayName;
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
import { Toaster as Sonner, type ToasterProps } from "sonner";
|
||||||
|
|
||||||
|
// App-wide toast host (Workstream 2). Themed to our tokens.
|
||||||
|
export function Toaster(props: ToasterProps) {
|
||||||
|
return (
|
||||||
|
<Sonner
|
||||||
|
theme="dark"
|
||||||
|
position="bottom-right"
|
||||||
|
toastOptions={{
|
||||||
|
style: {
|
||||||
|
background: "var(--popover)",
|
||||||
|
color: "var(--popover-foreground)",
|
||||||
|
border: "1px solid var(--border)",
|
||||||
|
},
|
||||||
|
}}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
import * as React from "react";
|
||||||
|
import { cn } from "@/lib/utils";
|
||||||
|
|
||||||
|
export const Textarea = React.forwardRef<
|
||||||
|
HTMLTextAreaElement,
|
||||||
|
React.TextareaHTMLAttributes<HTMLTextAreaElement>
|
||||||
|
>(({ className, ...props }, ref) => (
|
||||||
|
<textarea
|
||||||
|
ref={ref}
|
||||||
|
className={cn(
|
||||||
|
"flex w-full rounded-md border border-input bg-transparent px-3 py-2 text-sm text-foreground shadow-sm transition-colors placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring disabled:cursor-not-allowed disabled:opacity-50",
|
||||||
|
className,
|
||||||
|
)}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
));
|
||||||
|
Textarea.displayName = "Textarea";
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
import * as React from "react";
|
||||||
|
import * as TooltipPrimitive from "@radix-ui/react-tooltip";
|
||||||
|
import { cn } from "@/lib/utils";
|
||||||
|
|
||||||
|
export const TooltipProvider = TooltipPrimitive.Provider;
|
||||||
|
export const Tooltip = TooltipPrimitive.Root;
|
||||||
|
export const TooltipTrigger = TooltipPrimitive.Trigger;
|
||||||
|
|
||||||
|
export const TooltipContent = React.forwardRef<
|
||||||
|
React.ElementRef<typeof TooltipPrimitive.Content>,
|
||||||
|
React.ComponentPropsWithoutRef<typeof TooltipPrimitive.Content>
|
||||||
|
>(({ className, sideOffset = 4, ...props }, ref) => (
|
||||||
|
<TooltipPrimitive.Portal>
|
||||||
|
<TooltipPrimitive.Content
|
||||||
|
ref={ref}
|
||||||
|
sideOffset={sideOffset}
|
||||||
|
className={cn(
|
||||||
|
"z-50 overflow-hidden rounded-md border border-border bg-popover px-2.5 py-1.5 text-xs text-popover-foreground shadow-md animate-in fade-in-0 zoom-in-95",
|
||||||
|
className,
|
||||||
|
)}
|
||||||
|
{...props}
|
||||||
|
/>
|
||||||
|
</TooltipPrimitive.Portal>
|
||||||
|
));
|
||||||
|
TooltipContent.displayName = TooltipPrimitive.Content.displayName;
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
@import "tailwindcss";
|
||||||
|
|
||||||
|
/* Dark theme is applied via a `.dark` class on <html>. */
|
||||||
|
@custom-variant dark (&:is(.dark *));
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Design tokens (Workstream 1). Semantic CSS variables, not hardcoded colors, so
|
||||||
|
* the whole UI is themeable. Dark values avoid pure black/white to prevent halation
|
||||||
|
* (WCAG dark-mode guidance); text/controls clear 4.5:1 / 3:1.
|
||||||
|
*/
|
||||||
|
:root {
|
||||||
|
--radius: 0.625rem;
|
||||||
|
|
||||||
|
/* Light (kept minimal for now — the app ships dark-first). */
|
||||||
|
--background: #fafafa;
|
||||||
|
--foreground: #18181b;
|
||||||
|
--card: #ffffff;
|
||||||
|
--card-foreground: #18181b;
|
||||||
|
--popover: #ffffff;
|
||||||
|
--popover-foreground: #18181b;
|
||||||
|
--primary: #059669;
|
||||||
|
--primary-foreground: #fafafa;
|
||||||
|
--secondary: #f4f4f5;
|
||||||
|
--secondary-foreground: #18181b;
|
||||||
|
--muted: #f4f4f5;
|
||||||
|
--muted-foreground: #52525b;
|
||||||
|
--accent: #f4f4f5;
|
||||||
|
--accent-foreground: #18181b;
|
||||||
|
--destructive: #dc2626;
|
||||||
|
--destructive-foreground: #fafafa;
|
||||||
|
--warning: #d97706;
|
||||||
|
--warning-foreground: #fafafa;
|
||||||
|
--success: #059669;
|
||||||
|
--border: #e4e4e7;
|
||||||
|
--input: #e4e4e7;
|
||||||
|
--ring: #10b981;
|
||||||
|
}
|
||||||
|
|
||||||
|
.dark {
|
||||||
|
--background: #09090b; /* zinc-950, not #000 */
|
||||||
|
--foreground: #e4e4e7; /* zinc-200, not #fff */
|
||||||
|
--card: #18181b;
|
||||||
|
--card-foreground: #e4e4e7;
|
||||||
|
--popover: #18181b;
|
||||||
|
--popover-foreground: #e4e4e7;
|
||||||
|
--primary: #10b981; /* emerald-500 */
|
||||||
|
--primary-foreground: #052e1f;
|
||||||
|
--secondary: #27272a;
|
||||||
|
--secondary-foreground: #e4e4e7;
|
||||||
|
--muted: #27272a;
|
||||||
|
--muted-foreground: #a1a1aa; /* zinc-400 */
|
||||||
|
--accent: #27272a;
|
||||||
|
--accent-foreground: #e4e4e7;
|
||||||
|
--destructive: #ef4444; /* red-500 */
|
||||||
|
--destructive-foreground: #fef2f2;
|
||||||
|
--warning: #f59e0b; /* amber-500 */
|
||||||
|
--warning-foreground: #1c1917;
|
||||||
|
--success: #34d399; /* emerald-400 */
|
||||||
|
--border: #27272a; /* zinc-800 */
|
||||||
|
--input: #3f3f46; /* zinc-700 */
|
||||||
|
--ring: #10b981;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Map tokens onto Tailwind color utilities (bg-background, text-muted-foreground, …). */
|
||||||
|
@theme inline {
|
||||||
|
--color-background: var(--background);
|
||||||
|
--color-foreground: var(--foreground);
|
||||||
|
--color-card: var(--card);
|
||||||
|
--color-card-foreground: var(--card-foreground);
|
||||||
|
--color-popover: var(--popover);
|
||||||
|
--color-popover-foreground: var(--popover-foreground);
|
||||||
|
--color-primary: var(--primary);
|
||||||
|
--color-primary-foreground: var(--primary-foreground);
|
||||||
|
--color-secondary: var(--secondary);
|
||||||
|
--color-secondary-foreground: var(--secondary-foreground);
|
||||||
|
--color-muted: var(--muted);
|
||||||
|
--color-muted-foreground: var(--muted-foreground);
|
||||||
|
--color-accent: var(--accent);
|
||||||
|
--color-accent-foreground: var(--accent-foreground);
|
||||||
|
--color-destructive: var(--destructive);
|
||||||
|
--color-destructive-foreground: var(--destructive-foreground);
|
||||||
|
--color-warning: var(--warning);
|
||||||
|
--color-warning-foreground: var(--warning-foreground);
|
||||||
|
--color-success: var(--success);
|
||||||
|
--color-border: var(--border);
|
||||||
|
--color-input: var(--input);
|
||||||
|
--color-ring: var(--ring);
|
||||||
|
|
||||||
|
--radius-sm: calc(var(--radius) - 4px);
|
||||||
|
--radius-md: calc(var(--radius) - 2px);
|
||||||
|
--radius-lg: var(--radius);
|
||||||
|
--radius-xl: calc(var(--radius) + 4px);
|
||||||
|
}
|
||||||
|
|
||||||
|
@layer base {
|
||||||
|
* {
|
||||||
|
border-color: var(--border);
|
||||||
|
}
|
||||||
|
html,
|
||||||
|
body,
|
||||||
|
#root {
|
||||||
|
height: 100%;
|
||||||
|
}
|
||||||
|
body {
|
||||||
|
margin: 0;
|
||||||
|
background-color: var(--background);
|
||||||
|
color: var(--foreground);
|
||||||
|
font-family: ui-sans-serif, system-ui, -apple-system, "Segoe UI", sans-serif;
|
||||||
|
}
|
||||||
|
/* Visible focus ring for keyboard users (accessibility). */
|
||||||
|
:focus-visible {
|
||||||
|
outline: 2px solid var(--ring);
|
||||||
|
outline-offset: 2px;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
import { createContext, useContext, useEffect, useState, type ReactNode } from "react";
|
||||||
|
|
||||||
|
export type Quality = "high" | "balanced" | "low";
|
||||||
|
export type Accent = "emerald" | "sky" | "violet" | "amber";
|
||||||
|
|
||||||
|
export interface Settings {
|
||||||
|
/** Transcription language ("auto" lets the model detect). Feeds W4. */
|
||||||
|
language: string;
|
||||||
|
/** Stream quality preset. Feeds W4. */
|
||||||
|
quality: Quality;
|
||||||
|
/** Primary accent color. */
|
||||||
|
accent: Accent;
|
||||||
|
}
|
||||||
|
|
||||||
|
const DEFAULTS: Settings = { language: "auto", quality: "balanced", accent: "emerald" };
|
||||||
|
|
||||||
|
const ACCENTS: Record<Accent, { primary: string; ring: string; fg: string }> = {
|
||||||
|
emerald: { primary: "#10b981", ring: "#10b981", fg: "#052e1f" },
|
||||||
|
sky: { primary: "#0ea5e9", ring: "#0ea5e9", fg: "#082f49" },
|
||||||
|
violet: { primary: "#8b5cf6", ring: "#8b5cf6", fg: "#1e1b4b" },
|
||||||
|
amber: { primary: "#f59e0b", ring: "#f59e0b", fg: "#1c1917" },
|
||||||
|
};
|
||||||
|
|
||||||
|
const STORAGE_KEY = "clawreview.settings";
|
||||||
|
|
||||||
|
interface SettingsContextValue {
|
||||||
|
settings: Settings;
|
||||||
|
update: (patch: Partial<Settings>) => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
const SettingsContext = createContext<SettingsContextValue | null>(null);
|
||||||
|
|
||||||
|
function load(): Settings {
|
||||||
|
try {
|
||||||
|
return { ...DEFAULTS, ...JSON.parse(localStorage.getItem(STORAGE_KEY) ?? "{}") };
|
||||||
|
} catch {
|
||||||
|
return DEFAULTS;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function applyAccent(accent: Accent) {
|
||||||
|
const a = ACCENTS[accent];
|
||||||
|
const root = document.documentElement;
|
||||||
|
root.style.setProperty("--primary", a.primary);
|
||||||
|
root.style.setProperty("--ring", a.ring);
|
||||||
|
root.style.setProperty("--primary-foreground", a.fg);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function SettingsProvider({ children }: { children: ReactNode }) {
|
||||||
|
const [settings, setSettings] = useState<Settings>(load);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
localStorage.setItem(STORAGE_KEY, JSON.stringify(settings));
|
||||||
|
applyAccent(settings.accent);
|
||||||
|
}, [settings]);
|
||||||
|
|
||||||
|
const update = (patch: Partial<Settings>) => setSettings((s) => ({ ...s, ...patch }));
|
||||||
|
|
||||||
|
return <SettingsContext.Provider value={{ settings, update }}>{children}</SettingsContext.Provider>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function useSettings(): SettingsContextValue {
|
||||||
|
const ctx = useContext(SettingsContext);
|
||||||
|
if (!ctx) throw new Error("useSettings must be used within SettingsProvider");
|
||||||
|
return ctx;
|
||||||
|
}
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
import { clsx, type ClassValue } from "clsx";
|
||||||
|
import { twMerge } from "tailwind-merge";
|
||||||
|
|
||||||
|
/** Merge conditional + conflicting Tailwind classes (shadcn convention). */
|
||||||
|
export function cn(...inputs: ClassValue[]) {
|
||||||
|
return twMerge(clsx(inputs));
|
||||||
|
}
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
import React from "react";
|
||||||
|
import ReactDOM from "react-dom/client";
|
||||||
|
import { TooltipProvider } from "@/components/ui/tooltip";
|
||||||
|
import { Toaster } from "@/components/ui/sonner";
|
||||||
|
import { SettingsProvider } from "@/lib/settings";
|
||||||
|
import App from "./App";
|
||||||
|
import "./index.css";
|
||||||
|
|
||||||
|
ReactDOM.createRoot(document.getElementById("root")!).render(
|
||||||
|
<React.StrictMode>
|
||||||
|
<SettingsProvider>
|
||||||
|
<TooltipProvider delayDuration={300}>
|
||||||
|
<App />
|
||||||
|
<Toaster />
|
||||||
|
</TooltipProvider>
|
||||||
|
</SettingsProvider>
|
||||||
|
</React.StrictMode>,
|
||||||
|
);
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2020",
|
||||||
|
"useDefineForClassFields": true,
|
||||||
|
"lib": ["ES2020", "DOM", "DOM.Iterable"],
|
||||||
|
"module": "ESNext",
|
||||||
|
"skipLibCheck": true,
|
||||||
|
"moduleResolution": "bundler",
|
||||||
|
"allowImportingTsExtensions": true,
|
||||||
|
"resolveJsonModule": true,
|
||||||
|
"isolatedModules": true,
|
||||||
|
"moduleDetection": "force",
|
||||||
|
"noEmit": true,
|
||||||
|
"jsx": "react-jsx",
|
||||||
|
"strict": true,
|
||||||
|
"noUnusedLocals": true,
|
||||||
|
"noUnusedParameters": true,
|
||||||
|
"baseUrl": ".",
|
||||||
|
"paths": { "@/*": ["./src/*"] }
|
||||||
|
},
|
||||||
|
"include": ["src", "vite.config.ts"]
|
||||||
|
}
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
import { defineConfig } from "vite";
|
||||||
|
import { fileURLToPath } from "node:url";
|
||||||
|
import { existsSync, readFileSync } from "node:fs";
|
||||||
|
import react from "@vitejs/plugin-react";
|
||||||
|
import tailwindcss from "@tailwindcss/vite";
|
||||||
|
|
||||||
|
// Serve over HTTPS when a Tailscale cert is present (mint with
|
||||||
|
// `tailscale cert --cert-file certs/tailnet.crt --key-file certs/tailnet.key <host>.ts.net`).
|
||||||
|
// A secure context is required for the microphone (getUserMedia) — plain HTTP on a
|
||||||
|
// LAN/Tailscale IP withholds navigator.mediaDevices. Access via the cert's ts.net host.
|
||||||
|
const keyPath = fileURLToPath(new URL("./certs/tailnet.key", import.meta.url));
|
||||||
|
const crtPath = fileURLToPath(new URL("./certs/tailnet.crt", import.meta.url));
|
||||||
|
const https =
|
||||||
|
existsSync(keyPath) && existsSync(crtPath)
|
||||||
|
? { key: readFileSync(keyPath), cert: readFileSync(crtPath) }
|
||||||
|
: undefined;
|
||||||
|
|
||||||
|
// /api is proxied to the review-api (§6.2) so the dev server and API share an origin.
|
||||||
|
export default defineConfig({
|
||||||
|
plugins: [react(), tailwindcss()],
|
||||||
|
resolve: {
|
||||||
|
alias: { "@": fileURLToPath(new URL("./src", import.meta.url)) },
|
||||||
|
},
|
||||||
|
server: {
|
||||||
|
port: 5173,
|
||||||
|
// Listen on all interfaces so the dev server is reachable over the Tailscale IP
|
||||||
|
// (mirrors the LAN/relay access model — on-site engineers reach it directly).
|
||||||
|
host: true,
|
||||||
|
https,
|
||||||
|
// Allow the tailnet hostnames (Vite blocks unknown Host headers by default).
|
||||||
|
allowedHosts: [".ts.net", "localhost"],
|
||||||
|
proxy: {
|
||||||
|
"/api": "http://localhost:8090",
|
||||||
|
// streamd (stream node): SDP signaling + input WebSocket. Rewrite strips the
|
||||||
|
// /streamd prefix; ws:true so the input socket is proxied too.
|
||||||
|
"/streamd": {
|
||||||
|
target: "http://localhost:8095",
|
||||||
|
ws: true,
|
||||||
|
rewrite: (p) => p.replace(/^\/streamd/, ""),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
[package]
|
||||||
|
name = "review-api"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2021"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
axum = { version = "0.7", features = ["multipart"] }
|
||||||
|
tokio = { version = "1", features = ["full"] }
|
||||||
|
tokio-stream = { version = "0.1", features = ["sync"] }
|
||||||
|
tower-http = { version = "0.6", features = ["cors", "trace"] }
|
||||||
|
serde = { version = "1", features = ["derive"] }
|
||||||
|
serde_json = "1"
|
||||||
|
sqlx = { version = "0.8", default-features = false, features = ["runtime-tokio", "tls-rustls", "postgres", "uuid", "chrono", "macros", "migrate"] }
|
||||||
|
uuid = { version = "1", features = ["v4", "serde"] }
|
||||||
|
chrono = { version = "0.4", features = ["serde"] }
|
||||||
|
anyhow = "1"
|
||||||
|
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json", "multipart"] }
|
||||||
|
tracing = "0.1"
|
||||||
|
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
-- ClawReview core data model (§7 of SPEC.md).
|
||||||
|
-- The anchoring chain finding -> session -> device -> build is what guarantees
|
||||||
|
-- every finding is unambiguously attributable. device_serial / build_hash /
|
||||||
|
-- reviewer_id are denormalized onto findings so attribution is immutable even if
|
||||||
|
-- a device is later re-flashed or re-registered.
|
||||||
|
|
||||||
|
CREATE TABLE users (
|
||||||
|
id UUID PRIMARY KEY,
|
||||||
|
email TEXT NOT NULL UNIQUE,
|
||||||
|
display_name TEXT NOT NULL,
|
||||||
|
role TEXT NOT NULL CHECK (role IN ('engineer','partner','operator','admin')),
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE builds (
|
||||||
|
id UUID PRIMARY KEY,
|
||||||
|
app_identity TEXT NOT NULL,
|
||||||
|
version TEXT NOT NULL,
|
||||||
|
hash TEXT NOT NULL,
|
||||||
|
install_metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE devices (
|
||||||
|
serial TEXT PRIMARY KEY,
|
||||||
|
model TEXT,
|
||||||
|
status TEXT NOT NULL DEFAULT 'offline' CHECK (status IN ('free','claimed','offline')),
|
||||||
|
stream_node TEXT,
|
||||||
|
current_build_id UUID REFERENCES builds(id),
|
||||||
|
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE sessions (
|
||||||
|
id UUID PRIMARY KEY,
|
||||||
|
reviewer_id UUID NOT NULL REFERENCES users(id),
|
||||||
|
device_serial TEXT NOT NULL REFERENCES devices(serial),
|
||||||
|
build_id UUID REFERENCES builds(id),
|
||||||
|
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
ended_at TIMESTAMPTZ
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE findings (
|
||||||
|
id UUID PRIMARY KEY,
|
||||||
|
session_id UUID NOT NULL REFERENCES sessions(id),
|
||||||
|
device_serial TEXT NOT NULL,
|
||||||
|
build_hash TEXT,
|
||||||
|
reviewer_id UUID NOT NULL REFERENCES users(id),
|
||||||
|
kind TEXT NOT NULL DEFAULT 'note' CHECK (kind IN ('note','voice')),
|
||||||
|
note_text TEXT,
|
||||||
|
audio_ref TEXT,
|
||||||
|
transcript TEXT,
|
||||||
|
frame_ref TEXT,
|
||||||
|
clip_ref TEXT,
|
||||||
|
time_offset_ms BIGINT,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX idx_findings_session ON findings(session_id);
|
||||||
|
CREATE INDEX idx_sessions_device ON sessions(device_serial);
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
-- Allow a dedicated 'frame' finding kind for one-tap screen snapshots (§4.5).
|
||||||
|
ALTER TABLE findings DROP CONSTRAINT findings_kind_check;
|
||||||
|
ALTER TABLE findings ADD CONSTRAINT findings_kind_check CHECK (kind IN ('note', 'voice', 'frame'));
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
-- Session heartbeat (§4.4): a claim is kept alive by periodic heartbeats. A claim
|
||||||
|
-- whose heartbeat has gone stale can be reclaimed, so a closed/refreshed browser
|
||||||
|
-- never strands a device.
|
||||||
|
ALTER TABLE sessions ADD COLUMN heartbeat_at TIMESTAMPTZ NOT NULL DEFAULT now();
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
-- Activity tracking for dormancy-based auto-reclaim (§4.4, sharing scenario).
|
||||||
|
-- heartbeat_at = connection liveness (browser still open); last_activity_at = the
|
||||||
|
-- last real interaction (tap/key/scroll/note/voice). A session is reclaimable when
|
||||||
|
-- it disconnects (stale heartbeat) OR goes dormant (no activity for the idle window).
|
||||||
|
ALTER TABLE sessions ADD COLUMN last_activity_at TIMESTAMPTZ NOT NULL DEFAULT now();
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
use axum::{
|
||||||
|
http::StatusCode,
|
||||||
|
response::{IntoResponse, Response},
|
||||||
|
Json,
|
||||||
|
};
|
||||||
|
use serde_json::json;
|
||||||
|
|
||||||
|
/// Application error mapped to an HTTP response. Any error that converts into
|
||||||
|
/// `anyhow::Error` (sqlx, etc.) becomes a 500 via the blanket `From` below.
|
||||||
|
pub enum AppError {
|
||||||
|
NotFound(String),
|
||||||
|
Conflict(String),
|
||||||
|
BadRequest(String),
|
||||||
|
Internal(anyhow::Error),
|
||||||
|
}
|
||||||
|
|
||||||
|
impl IntoResponse for AppError {
|
||||||
|
fn into_response(self) -> Response {
|
||||||
|
let (status, message) = match self {
|
||||||
|
AppError::NotFound(m) => (StatusCode::NOT_FOUND, m),
|
||||||
|
AppError::Conflict(m) => (StatusCode::CONFLICT, m),
|
||||||
|
AppError::BadRequest(m) => (StatusCode::BAD_REQUEST, m),
|
||||||
|
AppError::Internal(e) => {
|
||||||
|
tracing::error!(error = ?e, "internal error");
|
||||||
|
(
|
||||||
|
StatusCode::INTERNAL_SERVER_ERROR,
|
||||||
|
"internal server error".to_string(),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
};
|
||||||
|
(status, Json(json!({ "error": message }))).into_response()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<E> From<E> for AppError
|
||||||
|
where
|
||||||
|
E: Into<anyhow::Error>,
|
||||||
|
{
|
||||||
|
fn from(err: E) -> Self {
|
||||||
|
AppError::Internal(err.into())
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
//! Server-sent events (Workstream 2): a broadcast channel of registry/session events
|
||||||
|
//! the browser subscribes to for live fleet status and push notifications, replacing
|
||||||
|
//! the previous poll-only model.
|
||||||
|
|
||||||
|
use std::convert::Infallible;
|
||||||
|
|
||||||
|
use axum::extract::State;
|
||||||
|
use axum::response::sse::{Event, KeepAlive, Sse};
|
||||||
|
use serde::Serialize;
|
||||||
|
use tokio::sync::broadcast;
|
||||||
|
use tokio_stream::wrappers::BroadcastStream;
|
||||||
|
use tokio_stream::{Stream, StreamExt};
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
use crate::AppState;
|
||||||
|
|
||||||
|
#[derive(Clone, Debug, Serialize)]
|
||||||
|
#[serde(tag = "type", rename_all = "snake_case")]
|
||||||
|
pub enum AppEvent {
|
||||||
|
/// A device changed status (free/claimed/offline) — drives the switcher + toasts.
|
||||||
|
DeviceStatus { serial: String, status: String },
|
||||||
|
/// An active session ended (released or reaped by a takeover) — the holder's
|
||||||
|
/// browser can surface a toast and return to the device list.
|
||||||
|
SessionEnded { session_id: Uuid },
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn channel() -> broadcast::Sender<AppEvent> {
|
||||||
|
broadcast::channel(128).0
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publish an event to all subscribers; a no-op when nobody is listening.
|
||||||
|
pub fn emit(state: &AppState, event: AppEvent) {
|
||||||
|
let _ = state.events.send(event);
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn events_handler(
|
||||||
|
State(state): State<AppState>,
|
||||||
|
) -> Sse<impl Stream<Item = Result<Event, Infallible>>> {
|
||||||
|
let stream = BroadcastStream::new(state.events.subscribe())
|
||||||
|
.filter_map(|res| res.ok().map(|ev| Ok(Event::default().json_data(ev).unwrap())));
|
||||||
|
Sse::new(stream).keep_alive(KeepAlive::default())
|
||||||
|
}
|
||||||
@@ -0,0 +1,491 @@
|
|||||||
|
use axum::{
|
||||||
|
extract::{Multipart, Path, State},
|
||||||
|
http::StatusCode,
|
||||||
|
response::IntoResponse,
|
||||||
|
Json,
|
||||||
|
};
|
||||||
|
use chrono::Utc;
|
||||||
|
use serde_json::{json, Value};
|
||||||
|
use sqlx::PgPool;
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
use crate::error::AppError;
|
||||||
|
use crate::events::{emit, AppEvent};
|
||||||
|
use crate::models::*;
|
||||||
|
use crate::AppState;
|
||||||
|
|
||||||
|
// Fixed seed identifiers. Auth (real users) lands in Phase 1; for the Track-1
|
||||||
|
// placeholder surface every session is authored by this dev reviewer.
|
||||||
|
const DEV_USER_ID: Uuid = Uuid::from_u128(0x0000_0000_0000_0000_0000_0000_0000_0001);
|
||||||
|
const PLACEHOLDER_BUILD_ID: Uuid = Uuid::from_u128(0x0000_0000_0000_0000_0000_0000_0000_0002);
|
||||||
|
|
||||||
|
pub async fn health() -> Json<Value> {
|
||||||
|
Json(json!({ "status": "ok" }))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn list_devices(State(st): State<AppState>) -> Result<Json<Vec<Device>>, AppError> {
|
||||||
|
let devices = sqlx::query_as::<_, Device>("SELECT * FROM devices ORDER BY serial")
|
||||||
|
.fetch_all(&st.db)
|
||||||
|
.await?;
|
||||||
|
Ok(Json(devices))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn create_session(
|
||||||
|
State(st): State<AppState>,
|
||||||
|
Json(body): Json<CreateSession>,
|
||||||
|
) -> Result<Json<Session>, AppError> {
|
||||||
|
let reviewer_id = DEV_USER_ID; // real reviewer identity arrives with auth (Phase 1).
|
||||||
|
let mut tx = st.db.begin().await?;
|
||||||
|
|
||||||
|
// Lock the device row so concurrent claims on the same device serialize.
|
||||||
|
let build_id: Option<Uuid> =
|
||||||
|
sqlx::query_scalar("SELECT current_build_id FROM devices WHERE serial=$1 FOR UPDATE")
|
||||||
|
.bind(&body.device_serial)
|
||||||
|
.fetch_optional(&mut *tx)
|
||||||
|
.await?
|
||||||
|
.ok_or_else(|| AppError::NotFound(format!("device {} not found", body.device_serial)))?;
|
||||||
|
|
||||||
|
// Block only if someone *else* holds a claim that is both connected (fresh
|
||||||
|
// heartbeat) and active (recent interaction). Our own claims (a refresh),
|
||||||
|
// disconnected claims, and dormant claims are all reclaimable.
|
||||||
|
let blocked: Option<i32> = sqlx::query_scalar(
|
||||||
|
"SELECT 1 FROM sessions \
|
||||||
|
WHERE device_serial=$1 AND ended_at IS NULL AND reviewer_id <> $2 \
|
||||||
|
AND heartbeat_at > now() - make_interval(secs => $3) \
|
||||||
|
AND last_activity_at > now() - make_interval(secs => $4) LIMIT 1",
|
||||||
|
)
|
||||||
|
.bind(&body.device_serial)
|
||||||
|
.bind(reviewer_id)
|
||||||
|
.bind(st.disconnect_timeout_secs as f64)
|
||||||
|
.bind(st.idle_timeout_secs as f64)
|
||||||
|
.fetch_optional(&mut *tx)
|
||||||
|
.await?;
|
||||||
|
if blocked.is_some() {
|
||||||
|
return Err(AppError::Conflict(format!(
|
||||||
|
"device {} is in use by another reviewer",
|
||||||
|
body.device_serial
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reap every reclaimable active session for this device, then take it.
|
||||||
|
let reaped: Vec<Uuid> = sqlx::query_scalar(
|
||||||
|
"UPDATE sessions SET ended_at=now() WHERE device_serial=$1 AND ended_at IS NULL RETURNING id",
|
||||||
|
)
|
||||||
|
.bind(&body.device_serial)
|
||||||
|
.fetch_all(&mut *tx)
|
||||||
|
.await?;
|
||||||
|
sqlx::query("UPDATE devices SET status='claimed', updated_at=now() WHERE serial=$1")
|
||||||
|
.bind(&body.device_serial)
|
||||||
|
.execute(&mut *tx)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
let session = sqlx::query_as::<_, Session>(
|
||||||
|
"INSERT INTO sessions (id, reviewer_id, device_serial, build_id, started_at, heartbeat_at) \
|
||||||
|
VALUES ($1, $2, $3, $4, now(), now()) RETURNING *",
|
||||||
|
)
|
||||||
|
.bind(Uuid::new_v4())
|
||||||
|
.bind(reviewer_id)
|
||||||
|
.bind(&body.device_serial)
|
||||||
|
.bind(build_id)
|
||||||
|
.fetch_one(&mut *tx)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
tx.commit().await?;
|
||||||
|
|
||||||
|
for session_id in reaped {
|
||||||
|
emit(&st, AppEvent::SessionEnded { session_id });
|
||||||
|
}
|
||||||
|
emit(
|
||||||
|
&st,
|
||||||
|
AppEvent::DeviceStatus {
|
||||||
|
serial: body.device_serial.clone(),
|
||||||
|
status: "claimed".into(),
|
||||||
|
},
|
||||||
|
);
|
||||||
|
Ok(Json(session))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, serde::Deserialize)]
|
||||||
|
pub struct HeartbeatParams {
|
||||||
|
/// True when the reviewer interacted since the last heartbeat. Bumps the
|
||||||
|
/// dormancy clock; a connected-but-idle session still ages toward reclaim.
|
||||||
|
#[serde(default)]
|
||||||
|
pub active: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Keep a claim alive. The browser pings this periodically (connection liveness);
|
||||||
|
/// `?active=true` also marks recent interaction so a dormant session can be reaped
|
||||||
|
/// even while its tab stays open (§4.4).
|
||||||
|
pub async fn heartbeat(
|
||||||
|
State(st): State<AppState>,
|
||||||
|
Path(id): Path<Uuid>,
|
||||||
|
axum::extract::Query(params): axum::extract::Query<HeartbeatParams>,
|
||||||
|
) -> Result<StatusCode, AppError> {
|
||||||
|
let sql = if params.active {
|
||||||
|
"UPDATE sessions SET heartbeat_at=now(), last_activity_at=now() WHERE id=$1 AND ended_at IS NULL"
|
||||||
|
} else {
|
||||||
|
"UPDATE sessions SET heartbeat_at=now() WHERE id=$1 AND ended_at IS NULL"
|
||||||
|
};
|
||||||
|
let updated = sqlx::query(sql).bind(id).execute(&st.db).await?;
|
||||||
|
if updated.rows_affected() == 0 {
|
||||||
|
return Err(AppError::NotFound("session not active".into()));
|
||||||
|
}
|
||||||
|
Ok(StatusCode::NO_CONTENT)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn release_session(
|
||||||
|
State(st): State<AppState>,
|
||||||
|
Path(id): Path<Uuid>,
|
||||||
|
) -> Result<StatusCode, AppError> {
|
||||||
|
let session = sqlx::query_as::<_, Session>(
|
||||||
|
"UPDATE sessions SET ended_at=now() WHERE id=$1 AND ended_at IS NULL RETURNING *",
|
||||||
|
)
|
||||||
|
.bind(id)
|
||||||
|
.fetch_optional(&st.db)
|
||||||
|
.await?
|
||||||
|
.ok_or_else(|| AppError::NotFound("session not found or already ended".into()))?;
|
||||||
|
|
||||||
|
// Return the device to the available pool. App-state reset on release (§8)
|
||||||
|
// belongs to the stream node and arrives with Phase 1.
|
||||||
|
sqlx::query("UPDATE devices SET status='free', updated_at=now() WHERE serial=$1")
|
||||||
|
.bind(&session.device_serial)
|
||||||
|
.execute(&st.db)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
emit(
|
||||||
|
&st,
|
||||||
|
AppEvent::DeviceStatus {
|
||||||
|
serial: session.device_serial.clone(),
|
||||||
|
status: "free".into(),
|
||||||
|
},
|
||||||
|
);
|
||||||
|
emit(&st, AppEvent::SessionEnded { session_id: session.id });
|
||||||
|
|
||||||
|
Ok(StatusCode::NO_CONTENT)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn list_findings(
|
||||||
|
State(st): State<AppState>,
|
||||||
|
Path(session_id): Path<Uuid>,
|
||||||
|
) -> Result<Json<Vec<Finding>>, AppError> {
|
||||||
|
let findings = sqlx::query_as::<_, Finding>(
|
||||||
|
"SELECT * FROM findings WHERE session_id=$1 ORDER BY created_at",
|
||||||
|
)
|
||||||
|
.bind(session_id)
|
||||||
|
.fetch_all(&st.db)
|
||||||
|
.await?;
|
||||||
|
Ok(Json(findings))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn create_finding(
|
||||||
|
State(st): State<AppState>,
|
||||||
|
Path(session_id): Path<Uuid>,
|
||||||
|
Json(body): Json<CreateFinding>,
|
||||||
|
) -> Result<Json<Finding>, AppError> {
|
||||||
|
let session = load_session(&st.db, session_id).await?;
|
||||||
|
|
||||||
|
// Auto-stamp the anchoring metadata from the session's source of truth.
|
||||||
|
let build_hash = build_hash_for(&st.db, &session).await?;
|
||||||
|
let time_offset_ms = (Utc::now() - session.started_at).num_milliseconds();
|
||||||
|
|
||||||
|
let finding = sqlx::query_as::<_, Finding>(
|
||||||
|
"INSERT INTO findings \
|
||||||
|
(id, session_id, device_serial, build_hash, reviewer_id, kind, note_text, time_offset_ms) \
|
||||||
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING *",
|
||||||
|
)
|
||||||
|
.bind(Uuid::new_v4())
|
||||||
|
.bind(session_id)
|
||||||
|
.bind(&session.device_serial)
|
||||||
|
.bind(build_hash)
|
||||||
|
.bind(session.reviewer_id)
|
||||||
|
.bind(body.kind.as_deref().unwrap_or("note"))
|
||||||
|
.bind(body.note_text)
|
||||||
|
.bind(time_offset_ms)
|
||||||
|
.fetch_one(&st.db)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
Ok(Json(finding))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn update_finding(
|
||||||
|
State(st): State<AppState>,
|
||||||
|
Path(id): Path<Uuid>,
|
||||||
|
Json(body): Json<UpdateFinding>,
|
||||||
|
) -> Result<Json<Finding>, AppError> {
|
||||||
|
// Debounced-autosave target: patch note text, leave anchoring untouched.
|
||||||
|
let finding = sqlx::query_as::<_, Finding>(
|
||||||
|
"UPDATE findings SET note_text=COALESCE($2, note_text) WHERE id=$1 RETURNING *",
|
||||||
|
)
|
||||||
|
.bind(id)
|
||||||
|
.bind(body.note_text)
|
||||||
|
.fetch_optional(&st.db)
|
||||||
|
.await?
|
||||||
|
.ok_or_else(|| AppError::NotFound("finding not found".into()))?;
|
||||||
|
Ok(Json(finding))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Attach a captured `<video>` frame to an existing finding (§4.5). The still is
|
||||||
|
/// stored and its reference recorded; the rest of the finding is untouched.
|
||||||
|
pub async fn attach_frame(
|
||||||
|
State(st): State<AppState>,
|
||||||
|
Path(finding_id): Path<Uuid>,
|
||||||
|
mut multipart: Multipart,
|
||||||
|
) -> Result<Json<Finding>, AppError> {
|
||||||
|
let (content_type, data) = read_upload(&mut multipart, "frame", "image/png").await?;
|
||||||
|
|
||||||
|
let exists: Option<Uuid> = sqlx::query_scalar("SELECT id FROM findings WHERE id=$1")
|
||||||
|
.bind(finding_id)
|
||||||
|
.fetch_optional(&st.db)
|
||||||
|
.await?;
|
||||||
|
if exists.is_none() {
|
||||||
|
return Err(AppError::NotFound("finding not found".into()));
|
||||||
|
}
|
||||||
|
|
||||||
|
let frame_ref = crate::storage::put(
|
||||||
|
&st.storage_dir,
|
||||||
|
"frames",
|
||||||
|
&format!("{finding_id}.{}", image_ext(&content_type)),
|
||||||
|
&data,
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
let finding = sqlx::query_as::<_, Finding>(
|
||||||
|
"UPDATE findings SET frame_ref=$2 WHERE id=$1 RETURNING *",
|
||||||
|
)
|
||||||
|
.bind(finding_id)
|
||||||
|
.bind(frame_ref)
|
||||||
|
.fetch_one(&st.db)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
Ok(Json(finding))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn create_audio_finding(
|
||||||
|
State(st): State<AppState>,
|
||||||
|
Path(session_id): Path<Uuid>,
|
||||||
|
mut multipart: Multipart,
|
||||||
|
) -> Result<Json<Finding>, AppError> {
|
||||||
|
let session = load_session(&st.db, session_id).await?;
|
||||||
|
|
||||||
|
let (content_type, data) = read_upload(&mut multipart, "audio", "application/octet-stream").await?;
|
||||||
|
|
||||||
|
let finding_id = Uuid::new_v4();
|
||||||
|
let ext = match content_type.as_str() {
|
||||||
|
"audio/webm" => "webm",
|
||||||
|
"audio/ogg" => "ogg",
|
||||||
|
"audio/mpeg" => "mp3",
|
||||||
|
"audio/mp4" | "audio/aac" => "m4a",
|
||||||
|
"audio/wav" | "audio/x-wav" => "wav",
|
||||||
|
_ => "bin",
|
||||||
|
};
|
||||||
|
let audio_ref =
|
||||||
|
crate::storage::put(&st.storage_dir, "audio", &format!("{finding_id}.{ext}"), &data)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
// Transcription handoff to the separate service (§6.4). A transcription failure
|
||||||
|
// must not lose the recording: save the finding with a null transcript and log.
|
||||||
|
let transcript = match crate::transcription::transcribe(
|
||||||
|
st.transcribe_url.as_deref(),
|
||||||
|
&data,
|
||||||
|
&content_type,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
Ok(t) => t,
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!(error = ?e, "transcription failed; saving finding without transcript");
|
||||||
|
None
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
let build_hash = build_hash_for(&st.db, &session).await?;
|
||||||
|
let time_offset_ms = (Utc::now() - session.started_at).num_milliseconds();
|
||||||
|
|
||||||
|
let finding = sqlx::query_as::<_, Finding>(
|
||||||
|
"INSERT INTO findings \
|
||||||
|
(id, session_id, device_serial, build_hash, reviewer_id, kind, audio_ref, transcript, time_offset_ms) \
|
||||||
|
VALUES ($1, $2, $3, $4, $5, 'voice', $6, $7, $8) RETURNING *",
|
||||||
|
)
|
||||||
|
.bind(finding_id)
|
||||||
|
.bind(session_id)
|
||||||
|
.bind(&session.device_serial)
|
||||||
|
.bind(build_hash)
|
||||||
|
.bind(session.reviewer_id)
|
||||||
|
.bind(audio_ref)
|
||||||
|
.bind(transcript)
|
||||||
|
.bind(time_offset_ms)
|
||||||
|
.fetch_one(&st.db)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
Ok(Json(finding))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Advance a multipart stream, mapping parse errors (malformed/empty body, a
|
||||||
|
/// truncated upload) to 400 rather than 500 — they are client-supplied.
|
||||||
|
async fn next_field(
|
||||||
|
mp: &mut Multipart,
|
||||||
|
) -> Result<Option<axum::extract::multipart::Field<'_>>, AppError> {
|
||||||
|
mp.next_field()
|
||||||
|
.await
|
||||||
|
.map_err(|e| AppError::BadRequest(format!("invalid multipart upload: {e}")))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read a single named file part from a multipart upload, returning its
|
||||||
|
/// content-type (or `default_ct`) and bytes.
|
||||||
|
async fn read_upload(
|
||||||
|
multipart: &mut Multipart,
|
||||||
|
field_name: &str,
|
||||||
|
default_ct: &str,
|
||||||
|
) -> Result<(String, axum::body::Bytes), AppError> {
|
||||||
|
let mut content_type = default_ct.to_string();
|
||||||
|
let mut data: Option<axum::body::Bytes> = None;
|
||||||
|
while let Some(field) = next_field(multipart).await? {
|
||||||
|
if field.name() == Some(field_name) {
|
||||||
|
if let Some(ct) = field.content_type() {
|
||||||
|
content_type = ct.to_string();
|
||||||
|
}
|
||||||
|
data = Some(
|
||||||
|
field
|
||||||
|
.bytes()
|
||||||
|
.await
|
||||||
|
.map_err(|e| AppError::BadRequest(format!("invalid multipart upload: {e}")))?,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let data = data.ok_or_else(|| AppError::BadRequest(format!("missing '{field_name}' field")))?;
|
||||||
|
Ok((content_type, data))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn image_ext(content_type: &str) -> &'static str {
|
||||||
|
match content_type {
|
||||||
|
"image/png" => "png",
|
||||||
|
"image/jpeg" => "jpg",
|
||||||
|
"image/webp" => "webp",
|
||||||
|
_ => "bin",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Transcribe audio without persisting anything — used for live, interim dictation
|
||||||
|
/// while the reviewer is still speaking (§4.5). Returns `{ "text": ... }`.
|
||||||
|
pub async fn transcribe_audio(
|
||||||
|
State(st): State<AppState>,
|
||||||
|
mut multipart: Multipart,
|
||||||
|
) -> Result<Json<Value>, AppError> {
|
||||||
|
let (content_type, data) = read_upload(&mut multipart, "audio", "audio/webm").await?;
|
||||||
|
let text = crate::transcription::transcribe(st.transcribe_url.as_deref(), &data, &content_type)
|
||||||
|
.await?
|
||||||
|
.unwrap_or_default();
|
||||||
|
Ok(Json(json!({ "text": text })))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Create a finding from a one-tap screen snapshot (§4.5): stores the frame and
|
||||||
|
/// anchors a `frame`-kind finding to the session.
|
||||||
|
pub async fn create_frame_finding(
|
||||||
|
State(st): State<AppState>,
|
||||||
|
Path(session_id): Path<Uuid>,
|
||||||
|
mut multipart: Multipart,
|
||||||
|
) -> Result<Json<Finding>, AppError> {
|
||||||
|
let session = load_session(&st.db, session_id).await?;
|
||||||
|
let (content_type, data) = read_upload(&mut multipart, "frame", "image/png").await?;
|
||||||
|
|
||||||
|
let finding_id = Uuid::new_v4();
|
||||||
|
let frame_ref = crate::storage::put(
|
||||||
|
&st.storage_dir,
|
||||||
|
"frames",
|
||||||
|
&format!("{finding_id}.{}", image_ext(&content_type)),
|
||||||
|
&data,
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
let build_hash = build_hash_for(&st.db, &session).await?;
|
||||||
|
let time_offset_ms = (Utc::now() - session.started_at).num_milliseconds();
|
||||||
|
|
||||||
|
let finding = sqlx::query_as::<_, Finding>(
|
||||||
|
"INSERT INTO findings \
|
||||||
|
(id, session_id, device_serial, build_hash, reviewer_id, kind, frame_ref, time_offset_ms) \
|
||||||
|
VALUES ($1, $2, $3, $4, $5, 'frame', $6, $7) RETURNING *",
|
||||||
|
)
|
||||||
|
.bind(finding_id)
|
||||||
|
.bind(session_id)
|
||||||
|
.bind(&session.device_serial)
|
||||||
|
.bind(build_hash)
|
||||||
|
.bind(session.reviewer_id)
|
||||||
|
.bind(frame_ref)
|
||||||
|
.bind(time_offset_ms)
|
||||||
|
.fetch_one(&st.db)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
Ok(Json(finding))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Serve a finding's stored frame image for display in the review surface.
|
||||||
|
pub async fn get_frame(
|
||||||
|
State(st): State<AppState>,
|
||||||
|
Path(id): Path<Uuid>,
|
||||||
|
) -> Result<axum::response::Response, AppError> {
|
||||||
|
let reference: Option<String> = sqlx::query_scalar("SELECT frame_ref FROM findings WHERE id=$1")
|
||||||
|
.bind(id)
|
||||||
|
.fetch_optional(&st.db)
|
||||||
|
.await?
|
||||||
|
.flatten();
|
||||||
|
let reference = reference.ok_or_else(|| AppError::NotFound("no frame for finding".into()))?;
|
||||||
|
let bytes = crate::storage::get(&st.storage_dir, &reference).await?;
|
||||||
|
let content_type = crate::storage::image_content_type(&reference);
|
||||||
|
Ok(([(axum::http::header::CONTENT_TYPE, content_type)], bytes).into_response())
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn load_session(db: &PgPool, id: Uuid) -> Result<Session, AppError> {
|
||||||
|
sqlx::query_as::<_, Session>("SELECT * FROM sessions WHERE id=$1")
|
||||||
|
.bind(id)
|
||||||
|
.fetch_optional(db)
|
||||||
|
.await?
|
||||||
|
.ok_or_else(|| AppError::NotFound("session not found".into()))
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn build_hash_for(db: &PgPool, session: &Session) -> Result<Option<String>, AppError> {
|
||||||
|
match session.build_id {
|
||||||
|
Some(bid) => Ok(sqlx::query_scalar("SELECT hash FROM builds WHERE id=$1")
|
||||||
|
.bind(bid)
|
||||||
|
.fetch_optional(db)
|
||||||
|
.await?),
|
||||||
|
None => Ok(None),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Seed the dev reviewer plus one placeholder build and device so the Track-1
|
||||||
|
/// review surface has something to claim without any streaming code.
|
||||||
|
pub async fn seed_placeholder(db: &PgPool) -> anyhow::Result<()> {
|
||||||
|
sqlx::query(
|
||||||
|
"INSERT INTO users (id, email, display_name, role) VALUES ($1, $2, $3, 'engineer') \
|
||||||
|
ON CONFLICT (email) DO NOTHING",
|
||||||
|
)
|
||||||
|
.bind(DEV_USER_ID)
|
||||||
|
.bind("[email protected]")
|
||||||
|
.bind("Dev Reviewer")
|
||||||
|
.execute(db)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
sqlx::query(
|
||||||
|
"INSERT INTO builds (id, app_identity, version, hash) VALUES ($1, $2, $3, $4) \
|
||||||
|
ON CONFLICT (id) DO NOTHING",
|
||||||
|
)
|
||||||
|
.bind(PLACEHOLDER_BUILD_ID)
|
||||||
|
.bind("com.redclaw.digitalhallmark")
|
||||||
|
.bind("0.0.0-placeholder")
|
||||||
|
.bind("deadbeefcafe")
|
||||||
|
.execute(db)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
// The real rig device (Phase 1: single physical device streamed by streamd).
|
||||||
|
sqlx::query(
|
||||||
|
"INSERT INTO devices (serial, model, status, stream_node, current_build_id) \
|
||||||
|
VALUES ($1, $2, 'free', 'local-streamd', $3) ON CONFLICT (serial) DO NOTHING",
|
||||||
|
)
|
||||||
|
.bind("A11PRO052500527")
|
||||||
|
.bind("LAGENIO A11 Pro")
|
||||||
|
.bind(PLACEHOLDER_BUILD_ID)
|
||||||
|
.execute(db)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
pub mod error;
|
||||||
|
pub mod events;
|
||||||
|
pub mod handlers;
|
||||||
|
pub mod models;
|
||||||
|
pub mod reaper;
|
||||||
|
pub mod storage;
|
||||||
|
pub mod transcription;
|
||||||
|
|
||||||
|
use std::path::PathBuf;
|
||||||
|
|
||||||
|
use axum::{
|
||||||
|
routing::{get, patch, post},
|
||||||
|
Router,
|
||||||
|
};
|
||||||
|
use tower_http::cors::CorsLayer;
|
||||||
|
|
||||||
|
use crate::events::AppEvent;
|
||||||
|
|
||||||
|
/// Shared application state. Storage location and the transcription endpoint live
|
||||||
|
/// here (not read from env inside handlers) so tests can inject isolated values
|
||||||
|
/// and run in parallel without clobbering a process-global.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct AppState {
|
||||||
|
pub db: sqlx::PgPool,
|
||||||
|
pub storage_dir: PathBuf,
|
||||||
|
pub transcribe_url: Option<String>,
|
||||||
|
pub events: tokio::sync::broadcast::Sender<AppEvent>,
|
||||||
|
/// A claim with no heartbeat for this long is treated as disconnected.
|
||||||
|
pub disconnect_timeout_secs: i64,
|
||||||
|
/// A claim with no interaction for this long is treated as dormant (§4.4).
|
||||||
|
pub idle_timeout_secs: i64,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn build_router(state: AppState) -> Router {
|
||||||
|
Router::new()
|
||||||
|
.route("/health", get(handlers::health))
|
||||||
|
.route("/api/events", get(events::events_handler))
|
||||||
|
.route("/api/devices", get(handlers::list_devices))
|
||||||
|
.route("/api/sessions", post(handlers::create_session))
|
||||||
|
.route("/api/sessions/:id/release", post(handlers::release_session))
|
||||||
|
.route("/api/sessions/:id/heartbeat", post(handlers::heartbeat))
|
||||||
|
.route(
|
||||||
|
"/api/sessions/:id/findings",
|
||||||
|
get(handlers::list_findings).post(handlers::create_finding),
|
||||||
|
)
|
||||||
|
.route(
|
||||||
|
"/api/sessions/:id/findings/audio",
|
||||||
|
post(handlers::create_audio_finding),
|
||||||
|
)
|
||||||
|
.route(
|
||||||
|
"/api/sessions/:id/findings/frame",
|
||||||
|
post(handlers::create_frame_finding),
|
||||||
|
)
|
||||||
|
.route("/api/transcribe", post(handlers::transcribe_audio))
|
||||||
|
.route("/api/findings/:id", patch(handlers::update_finding))
|
||||||
|
.route(
|
||||||
|
"/api/findings/:id/frame",
|
||||||
|
post(handlers::attach_frame).get(handlers::get_frame),
|
||||||
|
)
|
||||||
|
// Permissive CORS so the Vite dev server (different origin) can call the API.
|
||||||
|
.layer(CorsLayer::permissive())
|
||||||
|
.with_state(state)
|
||||||
|
}
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
use anyhow::Context;
|
||||||
|
use review_api::{build_router, handlers, AppState};
|
||||||
|
use sqlx::postgres::PgPoolOptions;
|
||||||
|
use std::time::Duration;
|
||||||
|
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
|
||||||
|
|
||||||
|
fn env_secs(key: &str, default: i64) -> i64 {
|
||||||
|
std::env::var(key).ok().and_then(|v| v.parse().ok()).unwrap_or(default)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::main]
|
||||||
|
async fn main() -> anyhow::Result<()> {
|
||||||
|
tracing_subscriber::registry()
|
||||||
|
.with(
|
||||||
|
tracing_subscriber::EnvFilter::try_from_default_env()
|
||||||
|
.unwrap_or_else(|_| "review_api=debug,tower_http=info,info".into()),
|
||||||
|
)
|
||||||
|
.with(tracing_subscriber::fmt::layer())
|
||||||
|
.init();
|
||||||
|
|
||||||
|
let database_url = std::env::var("DATABASE_URL").unwrap_or_else(|_| {
|
||||||
|
"postgres://clawreview:clawreview@localhost:5432/clawreview".to_string()
|
||||||
|
});
|
||||||
|
|
||||||
|
let db = PgPoolOptions::new()
|
||||||
|
.max_connections(10)
|
||||||
|
.acquire_timeout(Duration::from_secs(5))
|
||||||
|
.connect(&database_url)
|
||||||
|
.await
|
||||||
|
.context("failed to connect to Postgres")?;
|
||||||
|
|
||||||
|
sqlx::migrate!()
|
||||||
|
.run(&db)
|
||||||
|
.await
|
||||||
|
.context("database migrations failed")?;
|
||||||
|
handlers::seed_placeholder(&db)
|
||||||
|
.await
|
||||||
|
.context("seeding placeholder fixtures failed")?;
|
||||||
|
|
||||||
|
let state = AppState {
|
||||||
|
db,
|
||||||
|
storage_dir: std::env::var("STORAGE_DIR")
|
||||||
|
.unwrap_or_else(|_| "./data".to_string())
|
||||||
|
.into(),
|
||||||
|
transcribe_url: Some(
|
||||||
|
std::env::var("TRANSCRIBE_URL")
|
||||||
|
.unwrap_or_else(|_| "http://127.0.0.1:8099/transcribe".to_string()),
|
||||||
|
),
|
||||||
|
events: review_api::events::channel(),
|
||||||
|
disconnect_timeout_secs: env_secs("DISCONNECT_TIMEOUT_SECS", 45),
|
||||||
|
idle_timeout_secs: env_secs("IDLE_TIMEOUT_SECS", 600),
|
||||||
|
};
|
||||||
|
|
||||||
|
// Background dormancy reaper: auto-frees disconnected/idle devices (§4.4).
|
||||||
|
review_api::reaper::spawn(state.clone());
|
||||||
|
|
||||||
|
let app = build_router(state);
|
||||||
|
|
||||||
|
let addr = std::env::var("BIND_ADDR").unwrap_or_else(|_| "0.0.0.0:8090".to_string());
|
||||||
|
let listener = tokio::net::TcpListener::bind(&addr).await?;
|
||||||
|
tracing::info!("review-api listening on http://{addr}");
|
||||||
|
axum::serve(listener, app).await?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
use chrono::{DateTime, Utc};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use sqlx::FromRow;
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
#[derive(Debug, Serialize, FromRow)]
|
||||||
|
pub struct Device {
|
||||||
|
pub serial: String,
|
||||||
|
pub model: Option<String>,
|
||||||
|
pub status: String,
|
||||||
|
pub stream_node: Option<String>,
|
||||||
|
pub current_build_id: Option<Uuid>,
|
||||||
|
pub updated_at: DateTime<Utc>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Serialize, FromRow)]
|
||||||
|
pub struct Session {
|
||||||
|
pub id: Uuid,
|
||||||
|
pub reviewer_id: Uuid,
|
||||||
|
pub device_serial: String,
|
||||||
|
pub build_id: Option<Uuid>,
|
||||||
|
pub started_at: DateTime<Utc>,
|
||||||
|
pub ended_at: Option<DateTime<Utc>>,
|
||||||
|
pub heartbeat_at: DateTime<Utc>,
|
||||||
|
pub last_activity_at: DateTime<Utc>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Serialize, FromRow)]
|
||||||
|
pub struct Finding {
|
||||||
|
pub id: Uuid,
|
||||||
|
pub session_id: Uuid,
|
||||||
|
pub device_serial: String,
|
||||||
|
pub build_hash: Option<String>,
|
||||||
|
pub reviewer_id: Uuid,
|
||||||
|
pub kind: String,
|
||||||
|
pub note_text: Option<String>,
|
||||||
|
pub audio_ref: Option<String>,
|
||||||
|
pub transcript: Option<String>,
|
||||||
|
pub frame_ref: Option<String>,
|
||||||
|
pub clip_ref: Option<String>,
|
||||||
|
pub time_offset_ms: Option<i64>,
|
||||||
|
pub created_at: DateTime<Utc>,
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- request bodies ----
|
||||||
|
|
||||||
|
#[derive(Debug, Deserialize)]
|
||||||
|
pub struct CreateSession {
|
||||||
|
pub device_serial: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Deserialize)]
|
||||||
|
pub struct CreateFinding {
|
||||||
|
#[serde(default)]
|
||||||
|
pub kind: Option<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
pub note_text: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Deserialize)]
|
||||||
|
pub struct UpdateFinding {
|
||||||
|
#[serde(default)]
|
||||||
|
pub note_text: Option<String>,
|
||||||
|
}
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
//! Background dormancy reaper (§4.4, device-sharing scenario). Periodically frees
|
||||||
|
//! devices whose active session has either disconnected (stale heartbeat) or gone
|
||||||
|
//! dormant (no interaction for the idle window), and pushes the change to everyone
|
||||||
|
//! over SSE. This makes reclamation automatic rather than only-on-next-claim.
|
||||||
|
|
||||||
|
use std::collections::HashSet;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use sqlx::PgPool;
|
||||||
|
use tokio::sync::broadcast::Sender;
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
use crate::events::AppEvent;
|
||||||
|
use crate::AppState;
|
||||||
|
|
||||||
|
/// End every reclaimable session and free its device, returning the freed serials.
|
||||||
|
/// A session is reclaimable when `heartbeat_at` is older than `disconnect_secs`
|
||||||
|
/// (browser gone) OR `last_activity_at` is older than `idle_secs` (dormant).
|
||||||
|
pub async fn reap_dormant(
|
||||||
|
db: &PgPool,
|
||||||
|
events: &Sender<AppEvent>,
|
||||||
|
disconnect_secs: i64,
|
||||||
|
idle_secs: i64,
|
||||||
|
) -> anyhow::Result<Vec<String>> {
|
||||||
|
let reaped: Vec<(Uuid, String)> = sqlx::query_as(
|
||||||
|
"UPDATE sessions SET ended_at=now() \
|
||||||
|
WHERE ended_at IS NULL \
|
||||||
|
AND (heartbeat_at < now() - make_interval(secs => $1) \
|
||||||
|
OR last_activity_at < now() - make_interval(secs => $2)) \
|
||||||
|
RETURNING id, device_serial",
|
||||||
|
)
|
||||||
|
.bind(disconnect_secs as f64)
|
||||||
|
.bind(idle_secs as f64)
|
||||||
|
.fetch_all(db)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
// Free each affected device that now has no active session, and announce it.
|
||||||
|
let mut freed = Vec::new();
|
||||||
|
let serials: HashSet<String> = reaped.iter().map(|(_, s)| s.clone()).collect();
|
||||||
|
for serial in serials {
|
||||||
|
let updated = sqlx::query(
|
||||||
|
"UPDATE devices SET status='free', updated_at=now() \
|
||||||
|
WHERE serial=$1 AND status='claimed' \
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM sessions WHERE device_serial=$1 AND ended_at IS NULL)",
|
||||||
|
)
|
||||||
|
.bind(&serial)
|
||||||
|
.execute(db)
|
||||||
|
.await?;
|
||||||
|
if updated.rows_affected() > 0 {
|
||||||
|
let _ = events.send(AppEvent::DeviceStatus {
|
||||||
|
serial: serial.clone(),
|
||||||
|
status: "free".into(),
|
||||||
|
});
|
||||||
|
freed.push(serial);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for (session_id, _) in reaped {
|
||||||
|
let _ = events.send(AppEvent::SessionEnded { session_id });
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(freed)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Spawn the periodic reaper loop. Runs for the life of the process.
|
||||||
|
pub fn spawn(state: AppState) {
|
||||||
|
tokio::spawn(async move {
|
||||||
|
let mut ticker = tokio::time::interval(Duration::from_secs(15));
|
||||||
|
loop {
|
||||||
|
ticker.tick().await;
|
||||||
|
if let Err(e) = reap_dormant(
|
||||||
|
&state.db,
|
||||||
|
&state.events,
|
||||||
|
state.disconnect_timeout_secs,
|
||||||
|
state.idle_timeout_secs,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
tracing::warn!(error = ?e, "dormancy reaper failed");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
/// Persist a blob under `base_dir/category/key` and return its storage reference.
|
||||||
|
/// The local filesystem stands in for Cloudflare R2 (§5) in dev; swap this impl
|
||||||
|
/// for an R2 client without touching callers — the returned reference string is
|
||||||
|
/// what lives in Postgres.
|
||||||
|
pub async fn put(
|
||||||
|
base_dir: &Path,
|
||||||
|
category: &str,
|
||||||
|
key: &str,
|
||||||
|
bytes: &[u8],
|
||||||
|
) -> anyhow::Result<String> {
|
||||||
|
let dir = base_dir.join(category);
|
||||||
|
tokio::fs::create_dir_all(&dir).await?;
|
||||||
|
tokio::fs::write(dir.join(key), bytes).await?;
|
||||||
|
Ok(format!("local://{category}/{key}"))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read back a blob previously stored with `put`, given its `local://…` reference.
|
||||||
|
pub async fn get(base_dir: &Path, reference: &str) -> anyhow::Result<Vec<u8>> {
|
||||||
|
let rel = reference
|
||||||
|
.strip_prefix("local://")
|
||||||
|
.ok_or_else(|| anyhow::anyhow!("not a local reference: {reference}"))?;
|
||||||
|
Ok(tokio::fs::read(base_dir.join(rel)).await?)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Guess an image content-type from a stored reference's extension.
|
||||||
|
pub fn image_content_type(reference: &str) -> &'static str {
|
||||||
|
if reference.ends_with(".png") {
|
||||||
|
"image/png"
|
||||||
|
} else if reference.ends_with(".jpg") || reference.ends_with(".jpeg") {
|
||||||
|
"image/jpeg"
|
||||||
|
} else if reference.ends_with(".webp") {
|
||||||
|
"image/webp"
|
||||||
|
} else {
|
||||||
|
"application/octet-stream"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
use serde_json::Value;
|
||||||
|
|
||||||
|
/// Hand audio to the self-hosted Whisper-class service (§6.4) over HTTP, kept off
|
||||||
|
/// the real-time streaming path. Returns `Ok(None)` when no endpoint is configured
|
||||||
|
/// so a finding can still be saved without a transcript; returns `Ok(Some(text))`
|
||||||
|
/// when the service responds with `{ "text": ... }`.
|
||||||
|
pub async fn transcribe(
|
||||||
|
endpoint: Option<&str>,
|
||||||
|
bytes: &[u8],
|
||||||
|
content_type: &str,
|
||||||
|
) -> anyhow::Result<Option<String>> {
|
||||||
|
let Some(url) = endpoint else {
|
||||||
|
return Ok(None);
|
||||||
|
};
|
||||||
|
|
||||||
|
let resp = reqwest::Client::new()
|
||||||
|
.post(url)
|
||||||
|
.header("content-type", content_type)
|
||||||
|
.body(bytes.to_vec())
|
||||||
|
.send()
|
||||||
|
.await?
|
||||||
|
.error_for_status()?;
|
||||||
|
|
||||||
|
let body: Value = resp.json().await?;
|
||||||
|
Ok(Some(
|
||||||
|
body.get("text")
|
||||||
|
.and_then(Value::as_str)
|
||||||
|
.unwrap_or_default()
|
||||||
|
.to_string(),
|
||||||
|
))
|
||||||
|
}
|
||||||
@@ -0,0 +1,525 @@
|
|||||||
|
//! Integration tests for the review API. Each test runs against its own ephemeral
|
||||||
|
//! Postgres database (`#[sqlx::test]`) and drives the real Axum router over HTTP.
|
||||||
|
|
||||||
|
use review_api::{build_router, handlers, AppState};
|
||||||
|
use serde_json::Value;
|
||||||
|
use sqlx::PgPool;
|
||||||
|
use std::path::PathBuf;
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
const DEVICE: &str = "A11PRO052500527";
|
||||||
|
|
||||||
|
struct TestApp {
|
||||||
|
base: String,
|
||||||
|
client: reqwest::Client,
|
||||||
|
storage_dir: PathBuf,
|
||||||
|
pool: PgPool,
|
||||||
|
events: tokio::sync::broadcast::Sender<review_api::events::AppEvent>,
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn spawn(pool: PgPool, transcribe_url: Option<String>) -> TestApp {
|
||||||
|
handlers::seed_placeholder(&pool).await.unwrap();
|
||||||
|
let storage_dir = std::env::temp_dir().join(format!("clawreview-test-{}", Uuid::new_v4()));
|
||||||
|
let events = review_api::events::channel();
|
||||||
|
let state = AppState {
|
||||||
|
db: pool.clone(),
|
||||||
|
storage_dir: storage_dir.clone(),
|
||||||
|
transcribe_url,
|
||||||
|
events: events.clone(),
|
||||||
|
disconnect_timeout_secs: 45,
|
||||||
|
idle_timeout_secs: 600,
|
||||||
|
};
|
||||||
|
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let addr = listener.local_addr().unwrap();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
axum::serve(listener, build_router(state)).await.unwrap();
|
||||||
|
});
|
||||||
|
TestApp {
|
||||||
|
base: format!("http://{addr}"),
|
||||||
|
client: reqwest::Client::new(),
|
||||||
|
storage_dir,
|
||||||
|
pool,
|
||||||
|
events,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A throwaway server that mimics the transcription service (§6.4).
|
||||||
|
async fn spawn_fake_transcriber(text: &'static str) -> String {
|
||||||
|
let app = axum::Router::new().route(
|
||||||
|
"/transcribe",
|
||||||
|
axum::routing::post(move || async move {
|
||||||
|
axum::Json(serde_json::json!({ "text": text }))
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let addr = listener.local_addr().unwrap();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
axum::serve(listener, app).await.unwrap();
|
||||||
|
});
|
||||||
|
format!("http://{addr}/transcribe")
|
||||||
|
}
|
||||||
|
|
||||||
|
impl TestApp {
|
||||||
|
async fn claim(&self) -> String {
|
||||||
|
let res = self
|
||||||
|
.client
|
||||||
|
.post(format!("{}/api/sessions", self.base))
|
||||||
|
.json(&serde_json::json!({ "device_serial": DEVICE }))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(res.status(), 200, "claim should succeed");
|
||||||
|
res.json::<Value>().await.unwrap()["id"]
|
||||||
|
.as_str()
|
||||||
|
.unwrap()
|
||||||
|
.to_string()
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn try_claim(&self) -> reqwest::StatusCode {
|
||||||
|
self.client
|
||||||
|
.post(format!("{}/api/sessions", self.base))
|
||||||
|
.json(&serde_json::json!({ "device_serial": DEVICE }))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.status()
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn create_note(&self, session: &str, text: &str) -> Value {
|
||||||
|
self.client
|
||||||
|
.post(format!("{}/api/sessions/{session}/findings", self.base))
|
||||||
|
.json(&serde_json::json!({ "kind": "note", "note_text": text }))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.json::<Value>()
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Directly insert an active claim held by a *different* reviewer, with a given
|
||||||
|
/// heartbeat age in seconds, and mark the device claimed — to exercise the reap.
|
||||||
|
async fn inject_foreign_claim(&self, heartbeat_age_secs: i64) {
|
||||||
|
let other = Uuid::new_v4();
|
||||||
|
sqlx::query("INSERT INTO users (id, email, display_name, role) VALUES ($1, $2, 'Other', 'engineer')")
|
||||||
|
.bind(other)
|
||||||
|
.bind(format!("other-{other}@redclaw.local"))
|
||||||
|
.execute(&self.pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
sqlx::query(
|
||||||
|
"INSERT INTO sessions (id, reviewer_id, device_serial, started_at, heartbeat_at) \
|
||||||
|
VALUES ($1, $2, $3, now(), now() - make_interval(secs => $4))",
|
||||||
|
)
|
||||||
|
.bind(Uuid::new_v4())
|
||||||
|
.bind(other)
|
||||||
|
.bind(DEVICE)
|
||||||
|
.bind(heartbeat_age_secs as f64)
|
||||||
|
.execute(&self.pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
sqlx::query("UPDATE devices SET status='claimed' WHERE serial=$1")
|
||||||
|
.bind(DEVICE)
|
||||||
|
.execute(&self.pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn same_reviewer_reclaims_on_refresh(pool: PgPool) {
|
||||||
|
let app = spawn(pool, None).await;
|
||||||
|
let first = app.claim().await;
|
||||||
|
|
||||||
|
// A refresh re-claims as the same reviewer: it takes over rather than conflicting.
|
||||||
|
assert_eq!(app.try_claim().await, 200, "same reviewer should reclaim");
|
||||||
|
|
||||||
|
// Exactly one active session remains for the device.
|
||||||
|
let active: i64 =
|
||||||
|
sqlx::query_scalar("SELECT count(*) FROM sessions WHERE device_serial=$1 AND ended_at IS NULL")
|
||||||
|
.bind(DEVICE)
|
||||||
|
.fetch_one(&app.pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(active, 1);
|
||||||
|
// The original session was ended by the takeover.
|
||||||
|
let ended: bool =
|
||||||
|
sqlx::query_scalar("SELECT ended_at IS NOT NULL FROM sessions WHERE id=$1::uuid")
|
||||||
|
.bind(&first)
|
||||||
|
.fetch_one(&app.pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert!(ended, "the prior session should be ended after takeover");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn reaper_frees_dormant_devices(pool: PgPool) {
|
||||||
|
use review_api::events::AppEvent;
|
||||||
|
let app = spawn(pool, None).await;
|
||||||
|
let _ = app.claim().await; // fresh claim
|
||||||
|
|
||||||
|
// Simulate dormancy: no interaction for 15 minutes (idle window is 10).
|
||||||
|
sqlx::query(
|
||||||
|
"UPDATE sessions SET last_activity_at = now() - make_interval(secs => 900) \
|
||||||
|
WHERE device_serial=$1 AND ended_at IS NULL",
|
||||||
|
)
|
||||||
|
.bind(DEVICE)
|
||||||
|
.execute(&app.pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let mut rx = app.events.subscribe();
|
||||||
|
let freed = review_api::reaper::reap_dormant(&app.pool, &app.events, 45, 600)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(freed, vec![DEVICE.to_string()]);
|
||||||
|
|
||||||
|
let status: String = sqlx::query_scalar("SELECT status FROM devices WHERE serial=$1")
|
||||||
|
.bind(DEVICE)
|
||||||
|
.fetch_one(&app.pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(status, "free", "dormant device should be auto-freed");
|
||||||
|
|
||||||
|
// A "device free" event is broadcast to all subscribers.
|
||||||
|
let mut saw_free = false;
|
||||||
|
while let Ok(ev) = rx.try_recv() {
|
||||||
|
if matches!(ev, AppEvent::DeviceStatus { ref status, .. } if status == "free") {
|
||||||
|
saw_free = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert!(saw_free, "freeing a dormant device should broadcast device_status free");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn reaper_leaves_active_sessions(pool: PgPool) {
|
||||||
|
let app = spawn(pool, None).await;
|
||||||
|
let _ = app.claim().await; // fresh heartbeat + activity
|
||||||
|
|
||||||
|
let freed = review_api::reaper::reap_dormant(&app.pool, &app.events, 45, 600)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert!(freed.is_empty(), "an active session must not be reaped");
|
||||||
|
|
||||||
|
let status: String = sqlx::query_scalar("SELECT status FROM devices WHERE serial=$1")
|
||||||
|
.bind(DEVICE)
|
||||||
|
.fetch_one(&app.pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(status, "claimed");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn active_heartbeat_resets_the_dormancy_clock(pool: PgPool) {
|
||||||
|
let app = spawn(pool, None).await;
|
||||||
|
let session = app.claim().await;
|
||||||
|
|
||||||
|
// Go dormant…
|
||||||
|
sqlx::query(
|
||||||
|
"UPDATE sessions SET last_activity_at = now() - make_interval(secs => 900) WHERE id=$1::uuid",
|
||||||
|
)
|
||||||
|
.bind(&session)
|
||||||
|
.execute(&app.pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
// …then interact: an active heartbeat bumps last_activity_at.
|
||||||
|
let res = app
|
||||||
|
.client
|
||||||
|
.post(format!("{}/api/sessions/{session}/heartbeat?active=true", app.base))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(res.status(), 204);
|
||||||
|
|
||||||
|
let freed = review_api::reaper::reap_dormant(&app.pool, &app.events, 45, 600)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert!(freed.is_empty(), "a freshly-active session must not be reaped");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn another_reviewers_live_claim_blocks(pool: PgPool) {
|
||||||
|
let app = spawn(pool, None).await;
|
||||||
|
app.inject_foreign_claim(0).await; // fresh heartbeat → genuinely in use
|
||||||
|
assert_eq!(app.try_claim().await, 409, "a live foreign claim must block");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn stale_claim_is_reclaimable(pool: PgPool) {
|
||||||
|
let app = spawn(pool, None).await;
|
||||||
|
app.inject_foreign_claim(120).await; // heartbeat 2min old → abandoned
|
||||||
|
assert_eq!(app.try_claim().await, 200, "a stale claim must be reclaimable");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn heartbeat_keeps_session_then_404s_after_release(pool: PgPool) {
|
||||||
|
let app = spawn(pool, None).await;
|
||||||
|
let session = app.claim().await;
|
||||||
|
|
||||||
|
let beat = app
|
||||||
|
.client
|
||||||
|
.post(format!("{}/api/sessions/{session}/heartbeat", app.base))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(beat.status(), 204);
|
||||||
|
|
||||||
|
app.client
|
||||||
|
.post(format!("{}/api/sessions/{session}/release", app.base))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
// After release the session is no longer active, so heartbeats 404.
|
||||||
|
let beat_after = app
|
||||||
|
.client
|
||||||
|
.post(format!("{}/api/sessions/{session}/heartbeat", app.base))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(beat_after.status(), 404);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn claim_broadcasts_a_device_status_event(pool: PgPool) {
|
||||||
|
use review_api::events::AppEvent;
|
||||||
|
let app = spawn(pool, None).await;
|
||||||
|
let mut rx = app.events.subscribe(); // subscribe before claiming
|
||||||
|
|
||||||
|
let _ = app.claim().await;
|
||||||
|
|
||||||
|
let ev = tokio::time::timeout(std::time::Duration::from_secs(2), rx.recv())
|
||||||
|
.await
|
||||||
|
.expect("an event should be broadcast")
|
||||||
|
.expect("recv ok");
|
||||||
|
match ev {
|
||||||
|
AppEvent::DeviceStatus { serial, status } => {
|
||||||
|
assert_eq!(serial, DEVICE);
|
||||||
|
assert_eq!(status, "claimed");
|
||||||
|
}
|
||||||
|
other => panic!("expected DeviceStatus claimed, got {other:?}"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn note_is_anchored_and_autosaves(pool: PgPool) {
|
||||||
|
let app = spawn(pool, None).await;
|
||||||
|
let session = app.claim().await;
|
||||||
|
|
||||||
|
let finding = app.create_note(&session, "draft").await;
|
||||||
|
let fid = finding["id"].as_str().unwrap();
|
||||||
|
assert_eq!(finding["device_serial"], DEVICE);
|
||||||
|
assert_eq!(finding["build_hash"], "deadbeefcafe");
|
||||||
|
assert!(finding["time_offset_ms"].as_i64().unwrap() >= 0);
|
||||||
|
|
||||||
|
// PATCH (autosave) updates the text, leaving anchoring intact.
|
||||||
|
let patched: Value = app
|
||||||
|
.client
|
||||||
|
.patch(format!("{}/api/findings/{fid}", app.base))
|
||||||
|
.json(&serde_json::json!({ "note_text": "tap latency ~1.8s" }))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.json()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(patched["note_text"], "tap latency ~1.8s");
|
||||||
|
assert_eq!(patched["build_hash"], "deadbeefcafe");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn frame_attaches_to_finding_and_persists(pool: PgPool) {
|
||||||
|
let app = spawn(pool, None).await;
|
||||||
|
let session = app.claim().await;
|
||||||
|
let finding = app.create_note(&session, "result screen").await;
|
||||||
|
let fid = finding["id"].as_str().unwrap().to_string();
|
||||||
|
|
||||||
|
let png = b"\x89PNG\r\n\x1a\n-fake-frame-bytes".to_vec();
|
||||||
|
let form = reqwest::multipart::Form::new().part(
|
||||||
|
"frame",
|
||||||
|
reqwest::multipart::Part::bytes(png.clone())
|
||||||
|
.file_name("frame.png")
|
||||||
|
.mime_str("image/png")
|
||||||
|
.unwrap(),
|
||||||
|
);
|
||||||
|
let res = app
|
||||||
|
.client
|
||||||
|
.post(format!("{}/api/findings/{fid}/frame", app.base))
|
||||||
|
.multipart(form)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(res.status(), 200);
|
||||||
|
let body: Value = res.json().await.unwrap();
|
||||||
|
assert_eq!(body["frame_ref"], format!("local://frames/{fid}.png"));
|
||||||
|
|
||||||
|
// The blob is actually on disk.
|
||||||
|
let path = app.storage_dir.join("frames").join(format!("{fid}.png"));
|
||||||
|
let stored = std::fs::read(&path).expect("frame file should exist");
|
||||||
|
assert_eq!(stored, png);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn frame_on_missing_finding_is_404(pool: PgPool) {
|
||||||
|
let app = spawn(pool, None).await;
|
||||||
|
let form = reqwest::multipart::Form::new().part(
|
||||||
|
"frame",
|
||||||
|
reqwest::multipart::Part::bytes(b"x".to_vec())
|
||||||
|
.file_name("f.png")
|
||||||
|
.mime_str("image/png")
|
||||||
|
.unwrap(),
|
||||||
|
);
|
||||||
|
let res = app
|
||||||
|
.client
|
||||||
|
.post(format!("{}/api/findings/{}/frame", app.base, Uuid::new_v4()))
|
||||||
|
.multipart(form)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(res.status(), 404);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn frame_without_field_is_400(pool: PgPool) {
|
||||||
|
let app = spawn(pool, None).await;
|
||||||
|
let session = app.claim().await;
|
||||||
|
let finding = app.create_note(&session, "n").await;
|
||||||
|
let fid = finding["id"].as_str().unwrap();
|
||||||
|
|
||||||
|
let res = app
|
||||||
|
.client
|
||||||
|
.post(format!("{}/api/findings/{fid}/frame", app.base))
|
||||||
|
.multipart(reqwest::multipart::Form::new()) // no "frame" part
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(res.status(), 400);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn live_transcribe_returns_text_without_persisting(pool: PgPool) {
|
||||||
|
let transcribe_url = spawn_fake_transcriber("live dictation text").await;
|
||||||
|
let app = spawn(pool, Some(transcribe_url)).await;
|
||||||
|
let session = app.claim().await;
|
||||||
|
|
||||||
|
let form = reqwest::multipart::Form::new().part(
|
||||||
|
"audio",
|
||||||
|
reqwest::multipart::Part::bytes(b"-partial-audio-".to_vec())
|
||||||
|
.file_name("live.webm")
|
||||||
|
.mime_str("audio/webm")
|
||||||
|
.unwrap(),
|
||||||
|
);
|
||||||
|
let body: Value = app
|
||||||
|
.client
|
||||||
|
.post(format!("{}/api/transcribe", app.base))
|
||||||
|
.multipart(form)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.json()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(body["text"], "live dictation text");
|
||||||
|
|
||||||
|
// Nothing was persisted as a finding.
|
||||||
|
let findings: Value = app
|
||||||
|
.client
|
||||||
|
.get(format!("{}/api/sessions/{session}/findings", app.base))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.json()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(findings.as_array().unwrap().len(), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn frame_snapshot_creates_finding_and_serves_image(pool: PgPool) {
|
||||||
|
let app = spawn(pool, None).await;
|
||||||
|
let session = app.claim().await;
|
||||||
|
|
||||||
|
let png = b"\x89PNG\r\n\x1a\n-screen-snapshot".to_vec();
|
||||||
|
let form = reqwest::multipart::Form::new().part(
|
||||||
|
"frame",
|
||||||
|
reqwest::multipart::Part::bytes(png.clone())
|
||||||
|
.file_name("screen.png")
|
||||||
|
.mime_str("image/png")
|
||||||
|
.unwrap(),
|
||||||
|
);
|
||||||
|
let finding: Value = app
|
||||||
|
.client
|
||||||
|
.post(format!("{}/api/sessions/{session}/findings/frame", app.base))
|
||||||
|
.multipart(form)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.json()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(finding["kind"], "frame");
|
||||||
|
assert_eq!(finding["device_serial"], DEVICE);
|
||||||
|
let fid = finding["id"].as_str().unwrap();
|
||||||
|
|
||||||
|
// The frame is served back with an image content-type and the exact bytes.
|
||||||
|
let res = app
|
||||||
|
.client
|
||||||
|
.get(format!("{}/api/findings/{fid}/frame", app.base))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(res.status(), 200);
|
||||||
|
assert_eq!(res.headers()["content-type"], "image/png");
|
||||||
|
assert_eq!(res.bytes().await.unwrap().to_vec(), png);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn get_frame_404_when_finding_has_none(pool: PgPool) {
|
||||||
|
let app = spawn(pool, None).await;
|
||||||
|
let session = app.claim().await;
|
||||||
|
let note = app.create_note(&session, "no frame here").await;
|
||||||
|
let fid = note["id"].as_str().unwrap();
|
||||||
|
let res = app
|
||||||
|
.client
|
||||||
|
.get(format!("{}/api/findings/{fid}/frame", app.base))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(res.status(), 404);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn voice_finding_stores_audio_and_transcript(pool: PgPool) {
|
||||||
|
let transcribe_url = spawn_fake_transcriber("nfc read resolved in about two seconds").await;
|
||||||
|
let app = spawn(pool, Some(transcribe_url)).await;
|
||||||
|
let session = app.claim().await;
|
||||||
|
|
||||||
|
let audio = b"-fake-webm-opus-bytes-".to_vec();
|
||||||
|
let form = reqwest::multipart::Form::new().part(
|
||||||
|
"audio",
|
||||||
|
reqwest::multipart::Part::bytes(audio.clone())
|
||||||
|
.file_name("rec.webm")
|
||||||
|
.mime_str("audio/webm")
|
||||||
|
.unwrap(),
|
||||||
|
);
|
||||||
|
let res = app
|
||||||
|
.client
|
||||||
|
.post(format!("{}/api/sessions/{session}/findings/audio", app.base))
|
||||||
|
.multipart(form)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(res.status(), 200);
|
||||||
|
let body: Value = res.json().await.unwrap();
|
||||||
|
assert_eq!(body["kind"], "voice");
|
||||||
|
assert_eq!(body["transcript"], "nfc read resolved in about two seconds");
|
||||||
|
assert_eq!(body["audio_ref"], format!("local://audio/{}.webm", body["id"].as_str().unwrap()));
|
||||||
|
|
||||||
|
// Audio blob persisted.
|
||||||
|
let path = app
|
||||||
|
.storage_dir
|
||||||
|
.join("audio")
|
||||||
|
.join(format!("{}.webm", body["id"].as_str().unwrap()));
|
||||||
|
assert_eq!(std::fs::read(&path).unwrap(), audio);
|
||||||
|
}
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
[package]
|
||||||
|
name = "stream-node"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2021"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
tokio = { version = "1", features = ["rt-multi-thread", "macros", "net", "io-util", "process", "time", "sync"] }
|
||||||
|
anyhow = "1"
|
||||||
|
bytes = "1"
|
||||||
|
# 0.20.x is an incomplete sans-IO rewrite; 0.11 is the last stable classic API
|
||||||
|
# (MediaEngine / TrackLocalStaticSample / write_sample). See §9 risk #5.
|
||||||
|
webrtc = "0.11"
|
||||||
|
axum = { version = "0.7", features = ["ws"] }
|
||||||
|
serde = { version = "1", features = ["derive"] }
|
||||||
|
serde_json = "1"
|
||||||
|
tracing = "0.1"
|
||||||
|
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||||
|
|
||||||
|
[[bin]]
|
||||||
|
name = "scrcpy-probe"
|
||||||
|
path = "src/bin/probe.rs"
|
||||||
|
|
||||||
|
[[bin]]
|
||||||
|
name = "streamd"
|
||||||
|
path = "src/bin/streamd.rs"
|
||||||
|
|
||||||
|
[dev-dependencies]
|
||||||
|
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json"] }
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
//! Single-device spike probe (§10 Phase 0, Track 2): start a real scrcpy session,
|
||||||
|
//! read the H.264 stream off the device, dump the first access unit for validation,
|
||||||
|
//! and exercise the control socket with a tap. Run with a device attached:
|
||||||
|
//! cargo run -p stream-node --bin scrcpy-probe -- [serial] [server-jar]
|
||||||
|
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use anyhow::Result;
|
||||||
|
use stream_node::control::{KeyAction, MotionAction};
|
||||||
|
use stream_node::coords::VideoSize;
|
||||||
|
use stream_node::scrcpy::{StreamSession, CODEC_ID_H264};
|
||||||
|
use stream_node::{key_message, touch_message, PointerEvent};
|
||||||
|
|
||||||
|
#[tokio::main]
|
||||||
|
async fn main() -> Result<()> {
|
||||||
|
let mut args = std::env::args().skip(1);
|
||||||
|
let serial = args.next().unwrap_or_else(|| "A11PRO052500527".to_string());
|
||||||
|
let jar = args
|
||||||
|
.next()
|
||||||
|
.unwrap_or_else(|| "/opt/homebrew/share/scrcpy/scrcpy-server".to_string());
|
||||||
|
|
||||||
|
println!("starting scrcpy session on {serial} (jar {jar})");
|
||||||
|
let mut session = StreamSession::start(&serial, &jar, 0).await?;
|
||||||
|
|
||||||
|
println!("device name : {}", session.device_name);
|
||||||
|
println!(
|
||||||
|
"codec : 0x{:08x} ({})",
|
||||||
|
session.codec_id,
|
||||||
|
if session.codec_id == CODEC_ID_H264 { "h264" } else { "?" }
|
||||||
|
);
|
||||||
|
|
||||||
|
let mut config_unit: Vec<u8> = Vec::new();
|
||||||
|
let mut first_keyframe: Vec<u8> = Vec::new();
|
||||||
|
let (mut n_config, mut n_key, mut n_other, mut bytes) = (0u32, 0u32, 0u32, 0usize);
|
||||||
|
|
||||||
|
// A static screen emits few frames, so cap the wait per packet rather than
|
||||||
|
// requiring a fixed count. Stop once we have the config unit + a keyframe.
|
||||||
|
for _ in 0..200 {
|
||||||
|
let pkt = match tokio::time::timeout(Duration::from_secs(3), session.next_packet()).await {
|
||||||
|
Ok(pkt) => pkt?,
|
||||||
|
Err(_) => {
|
||||||
|
println!("(no new frame for 3s — stopping read)");
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
bytes += pkt.data.len();
|
||||||
|
if pkt.header.config {
|
||||||
|
n_config += 1;
|
||||||
|
if config_unit.is_empty() {
|
||||||
|
config_unit = pkt.data.clone();
|
||||||
|
}
|
||||||
|
} else if pkt.header.keyframe {
|
||||||
|
n_key += 1;
|
||||||
|
if first_keyframe.is_empty() {
|
||||||
|
first_keyframe = pkt.data.clone();
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
n_other += 1;
|
||||||
|
}
|
||||||
|
if n_config >= 1 && n_key >= 1 && n_other >= 3 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
println!("video size : {}x{} (from session packet)", session.width(), session.height());
|
||||||
|
println!(
|
||||||
|
"packets : {n_config} config, {n_key} keyframe, {n_other} other ({bytes} bytes)"
|
||||||
|
);
|
||||||
|
println!(
|
||||||
|
"config unit : {} bytes (SPS/PPS), first keyframe: {} bytes",
|
||||||
|
config_unit.len(),
|
||||||
|
first_keyframe.len()
|
||||||
|
);
|
||||||
|
|
||||||
|
let video = VideoSize::new(session.width().max(1), session.height().max(1));
|
||||||
|
|
||||||
|
// Write config + first keyframe as a standalone Annex-B access unit to validate.
|
||||||
|
let mut au = config_unit.clone();
|
||||||
|
au.extend_from_slice(&first_keyframe);
|
||||||
|
std::fs::write("/tmp/au.h264", &au)?;
|
||||||
|
println!("wrote /tmp/au.h264 ({} bytes) — validate with ffprobe", au.len());
|
||||||
|
|
||||||
|
// Exercise the control socket: a center tap (down then up) via the input encoder.
|
||||||
|
println!("sending a center tap over the control socket…");
|
||||||
|
let down = touch_message(
|
||||||
|
&PointerEvent { action: MotionAction::Down, norm_x: 0.5, norm_y: 0.5 },
|
||||||
|
video,
|
||||||
|
);
|
||||||
|
session.send_control(&down).await?;
|
||||||
|
tokio::time::sleep(Duration::from_millis(60)).await;
|
||||||
|
let up = touch_message(
|
||||||
|
&PointerEvent { action: MotionAction::Up, norm_x: 0.5, norm_y: 0.5 },
|
||||||
|
video,
|
||||||
|
);
|
||||||
|
session.send_control(&up).await?;
|
||||||
|
|
||||||
|
// And a HOME key press to show key injection works.
|
||||||
|
session.send_control(&key_message(KeyAction::Down, 3, 0)).await?; // KEYCODE_HOME = 3
|
||||||
|
session.send_control(&key_message(KeyAction::Up, 3, 0)).await?;
|
||||||
|
println!("control messages sent without error");
|
||||||
|
|
||||||
|
session.shutdown().await?;
|
||||||
|
println!("session shut down cleanly");
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -0,0 +1,215 @@
|
|||||||
|
//! streamd: the browser-facing edge of the single-device spike (§4.4 signaling).
|
||||||
|
//! Serves the WebRTC client, brokers SDP (offer → answer), runs a scrcpy session,
|
||||||
|
//! pumps its H.264 into the peer connection, and forwards browser input events to
|
||||||
|
//! the device's control socket. One viewer / one session — enough for the spike.
|
||||||
|
|
||||||
|
use std::net::SocketAddr;
|
||||||
|
use std::sync::atomic::{AtomicU32, Ordering};
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use anyhow::Result;
|
||||||
|
use axum::{
|
||||||
|
extract::{
|
||||||
|
ws::{Message, WebSocket, WebSocketUpgrade},
|
||||||
|
Query, State,
|
||||||
|
},
|
||||||
|
http::{header, StatusCode},
|
||||||
|
response::{Html, IntoResponse, Response},
|
||||||
|
routing::{get, post},
|
||||||
|
Json, Router,
|
||||||
|
};
|
||||||
|
use serde::Deserialize;
|
||||||
|
use serde_json::json;
|
||||||
|
use tokio::io::AsyncWriteExt;
|
||||||
|
use tokio::net::TcpStream;
|
||||||
|
use tokio::sync::Mutex;
|
||||||
|
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
|
||||||
|
|
||||||
|
use stream_node::control::{KeyAction, MotionAction};
|
||||||
|
use stream_node::coords::VideoSize;
|
||||||
|
use stream_node::scrcpy::{ServerHandle, StreamSession};
|
||||||
|
use stream_node::webrtc_bridge::{pump_video, SharedSize, VideoBridge};
|
||||||
|
use stream_node::{key_message, touch_message, PointerEvent};
|
||||||
|
|
||||||
|
const PAGE: &str = include_str!("streamd_page.html");
|
||||||
|
|
||||||
|
struct AppState {
|
||||||
|
serial: String,
|
||||||
|
jar: String,
|
||||||
|
control: Mutex<Option<TcpStream>>,
|
||||||
|
size: SharedSize,
|
||||||
|
guard: Mutex<Option<ServerHandle>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::main]
|
||||||
|
async fn main() -> Result<()> {
|
||||||
|
tracing_subscriber::registry()
|
||||||
|
.with(
|
||||||
|
tracing_subscriber::EnvFilter::try_from_default_env()
|
||||||
|
.unwrap_or_else(|_| "stream_node=info,info".into()),
|
||||||
|
)
|
||||||
|
.with(tracing_subscriber::fmt::layer())
|
||||||
|
.init();
|
||||||
|
|
||||||
|
let state = Arc::new(AppState {
|
||||||
|
serial: std::env::var("DEVICE_SERIAL").unwrap_or_else(|_| "A11PRO052500527".into()),
|
||||||
|
jar: std::env::var("SCRCPY_SERVER")
|
||||||
|
.unwrap_or_else(|_| "/opt/homebrew/share/scrcpy/scrcpy-server".into()),
|
||||||
|
control: Mutex::new(None),
|
||||||
|
size: Arc::new((AtomicU32::new(720), AtomicU32::new(1600))),
|
||||||
|
guard: Mutex::new(None),
|
||||||
|
});
|
||||||
|
|
||||||
|
let app = Router::new()
|
||||||
|
.route("/", get(|| async { Html(PAGE) }))
|
||||||
|
.route("/offer", post(offer))
|
||||||
|
.route("/input", get(input_ws))
|
||||||
|
.route("/thumbnail", get(thumbnail))
|
||||||
|
.with_state(state);
|
||||||
|
|
||||||
|
let addr: SocketAddr = std::env::var("BIND_ADDR")
|
||||||
|
.unwrap_or_else(|_| "0.0.0.0:8095".into())
|
||||||
|
.parse()?;
|
||||||
|
let listener = tokio::net::TcpListener::bind(addr).await?;
|
||||||
|
tracing::info!("streamd listening on http://{addr}");
|
||||||
|
axum::serve(listener, app).await?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct OfferBody {
|
||||||
|
sdp: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct ThumbQuery {
|
||||||
|
serial: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Capture a still of a device's current screen via `adb exec-out screencap -p`.
|
||||||
|
/// Works whether or not the device is being streamed, so the device gallery can show
|
||||||
|
/// a live-ish thumbnail. The serial is passed as an argv (no shell), so it can't inject.
|
||||||
|
async fn thumbnail(Query(q): Query<ThumbQuery>) -> Response {
|
||||||
|
let out = tokio::process::Command::new("adb")
|
||||||
|
.args(["-s", &q.serial, "exec-out", "screencap", "-p"])
|
||||||
|
.output()
|
||||||
|
.await;
|
||||||
|
match out {
|
||||||
|
Ok(o) if o.status.success() && !o.stdout.is_empty() => (
|
||||||
|
[
|
||||||
|
(header::CONTENT_TYPE, "image/png"),
|
||||||
|
(header::CACHE_CONTROL, "no-store"),
|
||||||
|
],
|
||||||
|
o.stdout,
|
||||||
|
)
|
||||||
|
.into_response(),
|
||||||
|
Ok(o) => (
|
||||||
|
StatusCode::SERVICE_UNAVAILABLE,
|
||||||
|
format!("screencap failed: {}", String::from_utf8_lossy(&o.stderr)),
|
||||||
|
)
|
||||||
|
.into_response(),
|
||||||
|
Err(e) => (StatusCode::SERVICE_UNAVAILABLE, format!("adb error: {e}")).into_response(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn offer(State(st): State<Arc<AppState>>, Json(body): Json<OfferBody>) -> impl IntoResponse {
|
||||||
|
match start_stream(&st, body.sdp).await {
|
||||||
|
Ok(answer) => Json(json!({ "sdp": answer })).into_response(),
|
||||||
|
Err(e) => {
|
||||||
|
tracing::error!(error = ?e, "failed to start stream");
|
||||||
|
(StatusCode::INTERNAL_SERVER_ERROR, format!("{e:#}")).into_response()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn start_stream(st: &Arc<AppState>, offer_sdp: String) -> Result<String> {
|
||||||
|
// Build the peer connection and answer first, so a bad offer never spins up a
|
||||||
|
// scrcpy session on the device.
|
||||||
|
let bridge = VideoBridge::new(&["stun:stun.l.google.com:19302".to_string()]).await?;
|
||||||
|
let answer = bridge.answer(offer_sdp).await?;
|
||||||
|
|
||||||
|
// Tear down any previous session before starting a new one.
|
||||||
|
if let Some(prev) = st.guard.lock().await.take() {
|
||||||
|
let _ = prev.shutdown().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
let session = StreamSession::start(&st.serial, &st.jar, 0).await?;
|
||||||
|
let (mut video, control, guard) = session.split();
|
||||||
|
*st.control.lock().await = Some(control);
|
||||||
|
*st.guard.lock().await = Some(guard);
|
||||||
|
|
||||||
|
let track = bridge.track();
|
||||||
|
|
||||||
|
let size = st.size.clone();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
// Hold the bridge so its peer connection outlives this scope.
|
||||||
|
let _bridge = bridge;
|
||||||
|
if let Err(e) = pump_video(&mut video, track, size).await {
|
||||||
|
tracing::warn!(error = ?e, "video pump ended");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
Ok(answer)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn input_ws(State(st): State<Arc<AppState>>, ws: WebSocketUpgrade) -> impl IntoResponse {
|
||||||
|
ws.on_upgrade(move |socket| handle_input(st, socket))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
#[serde(tag = "kind", rename_all = "lowercase")]
|
||||||
|
enum InputEvent {
|
||||||
|
Touch { action: String, x: f64, y: f64 },
|
||||||
|
Key { action: String, keycode: u32 },
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn handle_input(st: Arc<AppState>, mut socket: WebSocket) {
|
||||||
|
while let Some(Ok(msg)) = socket.recv().await {
|
||||||
|
let text = match msg {
|
||||||
|
Message::Text(t) => t,
|
||||||
|
Message::Close(_) => break,
|
||||||
|
_ => continue,
|
||||||
|
};
|
||||||
|
let Ok(event) = serde_json::from_str::<InputEvent>(&text) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
let Some(bytes) = build_control_message(&st, &event) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
let mut guard = st.control.lock().await;
|
||||||
|
if let Some(control) = guard.as_mut() {
|
||||||
|
let _ = control.write_all(&bytes).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn build_control_message(st: &AppState, event: &InputEvent) -> Option<Vec<u8>> {
|
||||||
|
match event {
|
||||||
|
InputEvent::Touch { action, x, y } => {
|
||||||
|
let action = match action.as_str() {
|
||||||
|
"down" => MotionAction::Down,
|
||||||
|
"up" => MotionAction::Up,
|
||||||
|
"move" => MotionAction::Move,
|
||||||
|
_ => return None,
|
||||||
|
};
|
||||||
|
let w = st.size.0.load(Ordering::Relaxed).max(1);
|
||||||
|
let h = st.size.1.load(Ordering::Relaxed).max(1);
|
||||||
|
Some(touch_message(
|
||||||
|
&PointerEvent {
|
||||||
|
action,
|
||||||
|
norm_x: *x,
|
||||||
|
norm_y: *y,
|
||||||
|
},
|
||||||
|
VideoSize::new(w, h),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
InputEvent::Key { action, keycode } => {
|
||||||
|
let action = match action.as_str() {
|
||||||
|
"down" => KeyAction::Down,
|
||||||
|
"up" => KeyAction::Up,
|
||||||
|
_ => return None,
|
||||||
|
};
|
||||||
|
Some(key_message(action, *keycode, 0))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>ClawReview — live device</title>
|
||||||
|
<style>
|
||||||
|
body { margin: 0; background: #09090b; color: #e4e4e7; font-family: system-ui, sans-serif;
|
||||||
|
display: flex; flex-direction: column; align-items: center; height: 100vh; }
|
||||||
|
#status { padding: 6px 12px; font-size: 13px; color: #a1a1aa; }
|
||||||
|
#stage { flex: 1; display: flex; align-items: center; gap: 12px; }
|
||||||
|
video { height: 88vh; background: #000; border-radius: 10px; touch-action: none; cursor: crosshair; }
|
||||||
|
.keys { display: flex; flex-direction: column; gap: 8px; }
|
||||||
|
button { background: #18181b; color: #e4e4e7; border: 1px solid #3f3f46; border-radius: 8px;
|
||||||
|
padding: 8px 12px; font-size: 13px; cursor: pointer; }
|
||||||
|
button:hover { background: #27272a; }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div id="status">connecting…</div>
|
||||||
|
<div id="stage">
|
||||||
|
<video id="v" autoplay playsinline muted></video>
|
||||||
|
<div class="keys">
|
||||||
|
<button data-key="3">HOME</button>
|
||||||
|
<button data-key="4">BACK</button>
|
||||||
|
<button data-key="187">RECENTS</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<script>
|
||||||
|
const statusEl = document.getElementById('status');
|
||||||
|
const video = document.getElementById('v');
|
||||||
|
let ws, down = false;
|
||||||
|
|
||||||
|
function setStatus(t) { statusEl.textContent = t; }
|
||||||
|
|
||||||
|
function iceComplete(pc) {
|
||||||
|
return new Promise((res) => {
|
||||||
|
if (pc.iceGatheringState === 'complete') return res();
|
||||||
|
pc.addEventListener('icegatheringstatechange', () => {
|
||||||
|
if (pc.iceGatheringState === 'complete') res();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async function start() {
|
||||||
|
const pc = new RTCPeerConnection();
|
||||||
|
pc.addTransceiver('video', { direction: 'recvonly' });
|
||||||
|
pc.ontrack = (e) => { video.srcObject = e.streams[0]; setStatus('streaming'); };
|
||||||
|
pc.oniceconnectionstatechange = () => setStatus('ice: ' + pc.iceConnectionState);
|
||||||
|
|
||||||
|
await pc.setLocalDescription(await pc.createOffer());
|
||||||
|
await iceComplete(pc);
|
||||||
|
|
||||||
|
const res = await fetch('/offer', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'content-type': 'application/json' },
|
||||||
|
body: JSON.stringify({ sdp: pc.localDescription.sdp }),
|
||||||
|
});
|
||||||
|
if (!res.ok) { setStatus('offer failed: ' + (await res.text())); return; }
|
||||||
|
const { sdp } = await res.json();
|
||||||
|
await pc.setRemoteDescription({ type: 'answer', sdp });
|
||||||
|
|
||||||
|
const proto = location.protocol === 'https:' ? 'wss' : 'ws';
|
||||||
|
ws = new WebSocket(`${proto}://${location.host}/input`);
|
||||||
|
wireInput();
|
||||||
|
}
|
||||||
|
|
||||||
|
function norm(e) {
|
||||||
|
const r = video.getBoundingClientRect();
|
||||||
|
return { x: (e.clientX - r.left) / r.width, y: (e.clientY - r.top) / r.height };
|
||||||
|
}
|
||||||
|
function sendTouch(action, e) {
|
||||||
|
if (!ws || ws.readyState !== 1) return;
|
||||||
|
const { x, y } = norm(e);
|
||||||
|
if (x < 0 || x > 1 || y < 0 || y > 1) return;
|
||||||
|
ws.send(JSON.stringify({ kind: 'touch', action, x, y }));
|
||||||
|
}
|
||||||
|
function sendKey(keycode) {
|
||||||
|
if (!ws || ws.readyState !== 1) return;
|
||||||
|
ws.send(JSON.stringify({ kind: 'key', action: 'down', keycode }));
|
||||||
|
ws.send(JSON.stringify({ kind: 'key', action: 'up', keycode }));
|
||||||
|
}
|
||||||
|
function wireInput() {
|
||||||
|
video.addEventListener('pointerdown', (e) => { down = true; video.setPointerCapture(e.pointerId); sendTouch('down', e); });
|
||||||
|
video.addEventListener('pointermove', (e) => { if (down) sendTouch('move', e); });
|
||||||
|
video.addEventListener('pointerup', (e) => { down = false; sendTouch('up', e); });
|
||||||
|
video.addEventListener('pointercancel', () => { down = false; });
|
||||||
|
document.querySelectorAll('button[data-key]').forEach((b) =>
|
||||||
|
b.addEventListener('click', () => sendKey(parseInt(b.dataset.key, 10))));
|
||||||
|
}
|
||||||
|
|
||||||
|
start().catch((e) => setStatus('error: ' + e));
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,171 @@
|
|||||||
|
//! scrcpy control-protocol message encoding (§4.3, §6.1). Byte layouts match the
|
||||||
|
//! scrcpy v4.0 server's `ControlMessageReader` (client serializer `control_msg.c`):
|
||||||
|
//! all multi-byte integers are big-endian.
|
||||||
|
|
||||||
|
/// Control message type tags, in the scrcpy v4.0 enum order.
|
||||||
|
#[repr(u8)]
|
||||||
|
#[derive(Clone, Copy)]
|
||||||
|
pub enum MsgType {
|
||||||
|
InjectKeycode = 0,
|
||||||
|
InjectTouchEvent = 2,
|
||||||
|
BackOrScreenOn = 4,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Android `KeyEvent` actions.
|
||||||
|
#[repr(u8)]
|
||||||
|
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
|
||||||
|
pub enum KeyAction {
|
||||||
|
Down = 0,
|
||||||
|
Up = 1,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Android `MotionEvent` actions (the subset the input path uses).
|
||||||
|
#[repr(u8)]
|
||||||
|
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
|
||||||
|
pub enum MotionAction {
|
||||||
|
Down = 0,
|
||||||
|
Up = 1,
|
||||||
|
Move = 2,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// scrcpy's synthetic pointer ids (Android `int64`, so two's-complement of -1/-2).
|
||||||
|
pub const POINTER_ID_MOUSE: u64 = u64::MAX; // -1
|
||||||
|
pub const POINTER_ID_GENERIC_FINGER: u64 = u64::MAX - 1; // -2
|
||||||
|
|
||||||
|
/// scrcpy `sc_float_to_u16fp`: map a [0,1] float to a u16 fixed-point value.
|
||||||
|
fn float_to_u16fp(value: f32) -> u16 {
|
||||||
|
debug_assert!((0.0..=1.0).contains(&value));
|
||||||
|
let u = (value * 65536.0) as u32; // f * 2^16
|
||||||
|
if u >= 0xffff {
|
||||||
|
0xffff
|
||||||
|
} else {
|
||||||
|
u as u16
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// INJECT_TOUCH_EVENT — 32 bytes.
|
||||||
|
#[allow(clippy::too_many_arguments)]
|
||||||
|
pub fn encode_touch(
|
||||||
|
action: MotionAction,
|
||||||
|
pointer_id: u64,
|
||||||
|
x: i32,
|
||||||
|
y: i32,
|
||||||
|
screen_width: u16,
|
||||||
|
screen_height: u16,
|
||||||
|
pressure: f32,
|
||||||
|
action_button: u32,
|
||||||
|
buttons: u32,
|
||||||
|
) -> Vec<u8> {
|
||||||
|
let mut buf = Vec::with_capacity(32);
|
||||||
|
buf.push(MsgType::InjectTouchEvent as u8);
|
||||||
|
buf.push(action as u8);
|
||||||
|
buf.extend_from_slice(&pointer_id.to_be_bytes());
|
||||||
|
buf.extend_from_slice(&x.to_be_bytes());
|
||||||
|
buf.extend_from_slice(&y.to_be_bytes());
|
||||||
|
buf.extend_from_slice(&screen_width.to_be_bytes());
|
||||||
|
buf.extend_from_slice(&screen_height.to_be_bytes());
|
||||||
|
buf.extend_from_slice(&float_to_u16fp(pressure).to_be_bytes());
|
||||||
|
buf.extend_from_slice(&action_button.to_be_bytes());
|
||||||
|
buf.extend_from_slice(&buttons.to_be_bytes());
|
||||||
|
debug_assert_eq!(buf.len(), 32);
|
||||||
|
buf
|
||||||
|
}
|
||||||
|
|
||||||
|
/// INJECT_KEYCODE — 14 bytes.
|
||||||
|
pub fn encode_keycode(action: KeyAction, keycode: u32, repeat: u32, metastate: u32) -> Vec<u8> {
|
||||||
|
let mut buf = Vec::with_capacity(14);
|
||||||
|
buf.push(MsgType::InjectKeycode as u8);
|
||||||
|
buf.push(action as u8);
|
||||||
|
buf.extend_from_slice(&keycode.to_be_bytes());
|
||||||
|
buf.extend_from_slice(&repeat.to_be_bytes());
|
||||||
|
buf.extend_from_slice(&metastate.to_be_bytes());
|
||||||
|
debug_assert_eq!(buf.len(), 14);
|
||||||
|
buf
|
||||||
|
}
|
||||||
|
|
||||||
|
/// BACK_OR_SCREEN_ON — 2 bytes (carries a key action).
|
||||||
|
pub fn encode_back_or_screen_on(action: KeyAction) -> Vec<u8> {
|
||||||
|
vec![MsgType::BackOrScreenOn as u8, action as u8]
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn touch_down_center_exact_bytes() {
|
||||||
|
// 540,960 on a 1080x1920 frame, finger pointer, full pressure.
|
||||||
|
let bytes = encode_touch(
|
||||||
|
MotionAction::Down,
|
||||||
|
POINTER_ID_GENERIC_FINGER,
|
||||||
|
540,
|
||||||
|
960,
|
||||||
|
1080,
|
||||||
|
1920,
|
||||||
|
1.0,
|
||||||
|
0,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
bytes,
|
||||||
|
vec![
|
||||||
|
0x02, // type INJECT_TOUCH_EVENT
|
||||||
|
0x00, // action DOWN
|
||||||
|
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFE, // pointer_id = -2
|
||||||
|
0x00, 0x00, 0x02, 0x1C, // x = 540
|
||||||
|
0x00, 0x00, 0x03, 0xC0, // y = 960
|
||||||
|
0x04, 0x38, // width = 1080
|
||||||
|
0x07, 0x80, // height = 1920
|
||||||
|
0xFF, 0xFF, // pressure 1.0 -> 0xffff
|
||||||
|
0x00, 0x00, 0x00, 0x00, // action_button
|
||||||
|
0x00, 0x00, 0x00, 0x00, // buttons
|
||||||
|
]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn touch_up_has_zero_pressure_encoding() {
|
||||||
|
let bytes = encode_touch(
|
||||||
|
MotionAction::Up,
|
||||||
|
POINTER_ID_GENERIC_FINGER,
|
||||||
|
0,
|
||||||
|
0,
|
||||||
|
1080,
|
||||||
|
1920,
|
||||||
|
0.0,
|
||||||
|
0,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
assert_eq!(bytes[1], 0x01); // action UP
|
||||||
|
assert_eq!(&bytes[22..24], &[0x00, 0x00]); // pressure 0.0
|
||||||
|
assert_eq!(bytes.len(), 32);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn keycode_back_exact_bytes() {
|
||||||
|
// KEYCODE_BACK = 4.
|
||||||
|
let bytes = encode_keycode(KeyAction::Down, 4, 0, 0);
|
||||||
|
assert_eq!(
|
||||||
|
bytes,
|
||||||
|
vec![
|
||||||
|
0x00, // type INJECT_KEYCODE
|
||||||
|
0x00, // action DOWN
|
||||||
|
0x00, 0x00, 0x00, 0x04, // keycode 4
|
||||||
|
0x00, 0x00, 0x00, 0x00, // repeat
|
||||||
|
0x00, 0x00, 0x00, 0x00, // metastate
|
||||||
|
]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn back_or_screen_on_is_two_bytes() {
|
||||||
|
assert_eq!(encode_back_or_screen_on(KeyAction::Up), vec![0x04, 0x01]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pressure_clamps_at_one() {
|
||||||
|
assert_eq!(float_to_u16fp(1.0), 0xffff);
|
||||||
|
assert_eq!(float_to_u16fp(0.0), 0x0000);
|
||||||
|
assert_eq!(float_to_u16fp(0.5), 0x8000);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
//! Coordinate mapping for the input back-channel (§4.3, §9 risk #2 — taps landing
|
||||||
|
//! in the wrong place). The browser sends *normalized* pointer positions in [0,1]
|
||||||
|
//! (the "normalized events" of §4.3); the stream node scales them against the
|
||||||
|
//! current video frame size. Normalizing on the wire — rather than baking in a
|
||||||
|
//! resolution on the frontend — means an orientation/size change can't make a
|
||||||
|
//! stale device size land taps in the wrong spot: the live frame size is authoritative.
|
||||||
|
|
||||||
|
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||||
|
pub struct VideoSize {
|
||||||
|
pub width: u32,
|
||||||
|
pub height: u32,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl VideoSize {
|
||||||
|
pub fn new(width: u32, height: u32) -> Self {
|
||||||
|
Self { width, height }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Map a normalized position to device pixels, clamped to `[0, width-1] × [0, height-1]`.
|
||||||
|
/// Out-of-range inputs are clamped rather than rejected so a pointer dragged off the
|
||||||
|
/// edge still produces an in-bounds coordinate.
|
||||||
|
pub fn to_device_pixels(norm_x: f64, norm_y: f64, video: VideoSize) -> (i32, i32) {
|
||||||
|
let x = scale(norm_x, video.width);
|
||||||
|
let y = scale(norm_y, video.height);
|
||||||
|
(x, y)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn scale(norm: f64, extent: u32) -> i32 {
|
||||||
|
if extent == 0 {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
let clamped = norm.clamp(0.0, 1.0);
|
||||||
|
let max = (extent - 1) as f64;
|
||||||
|
(clamped * extent as f64).round().min(max) as i32
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
const FHD: VideoSize = VideoSize {
|
||||||
|
width: 1080,
|
||||||
|
height: 1920,
|
||||||
|
};
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn center_maps_to_mid_pixel() {
|
||||||
|
assert_eq!(to_device_pixels(0.5, 0.5, FHD), (540, 960));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn top_left_corner() {
|
||||||
|
assert_eq!(to_device_pixels(0.0, 0.0, FHD), (0, 0));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn bottom_right_clamps_inside_bounds() {
|
||||||
|
// 1.0 would scale to width/height; must clamp to the last valid pixel.
|
||||||
|
assert_eq!(to_device_pixels(1.0, 1.0, FHD), (1079, 1919));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn out_of_range_is_clamped_not_rejected() {
|
||||||
|
assert_eq!(to_device_pixels(1.5, -0.3, FHD), (1079, 0));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn landscape_size_scales_independently() {
|
||||||
|
let land = VideoSize::new(2340, 1080);
|
||||||
|
assert_eq!(to_device_pixels(0.5, 0.5, land), (1170, 540));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn zero_size_is_safe() {
|
||||||
|
assert_eq!(to_device_pixels(0.5, 0.5, VideoSize::new(0, 0)), (0, 0));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
//! ClawReview stream node (§6.1). This crate currently holds the device-independent
|
||||||
|
//! input back-channel: translating normalized browser pointer/key events into scrcpy
|
||||||
|
//! control-protocol messages. The scrcpy supervisor, H.264 reader, and the webrtc-rs
|
||||||
|
//! peer connection are added once a device is attached for the single-device spike.
|
||||||
|
|
||||||
|
pub mod control;
|
||||||
|
pub mod coords;
|
||||||
|
pub mod scrcpy;
|
||||||
|
pub mod webrtc_bridge;
|
||||||
|
|
||||||
|
use control::{encode_keycode, encode_touch, KeyAction, MotionAction, POINTER_ID_GENERIC_FINGER};
|
||||||
|
use coords::{to_device_pixels, VideoSize};
|
||||||
|
|
||||||
|
/// A normalized pointer event arriving from the browser over the input WebSocket
|
||||||
|
/// (§4.3). Coordinates are in [0,1] relative to the displayed video.
|
||||||
|
pub struct PointerEvent {
|
||||||
|
pub action: MotionAction,
|
||||||
|
pub norm_x: f64,
|
||||||
|
pub norm_y: f64,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Translate a normalized pointer event into a scrcpy touch control message for the
|
||||||
|
/// current live video frame size — the mirrored, backend half of the §4.3 scaling.
|
||||||
|
pub fn touch_message(event: &PointerEvent, video: VideoSize) -> Vec<u8> {
|
||||||
|
let (x, y) = to_device_pixels(event.norm_x, event.norm_y, video);
|
||||||
|
let pressure = if event.action == MotionAction::Up { 0.0 } else { 1.0 };
|
||||||
|
encode_touch(
|
||||||
|
event.action,
|
||||||
|
POINTER_ID_GENERIC_FINGER,
|
||||||
|
x,
|
||||||
|
y,
|
||||||
|
video.width as u16,
|
||||||
|
video.height as u16,
|
||||||
|
pressure,
|
||||||
|
0,
|
||||||
|
0,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Translate a key press/release into a scrcpy keycode control message.
|
||||||
|
pub fn key_message(action: KeyAction, keycode: u32, metastate: u32) -> Vec<u8> {
|
||||||
|
encode_keycode(action, keycode, 0, metastate)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn center_tap_down_matches_protocol_encoding() {
|
||||||
|
let event = PointerEvent {
|
||||||
|
action: MotionAction::Down,
|
||||||
|
norm_x: 0.5,
|
||||||
|
norm_y: 0.5,
|
||||||
|
};
|
||||||
|
let msg = touch_message(&event, VideoSize::new(1080, 1920));
|
||||||
|
let expected = encode_touch(
|
||||||
|
MotionAction::Down,
|
||||||
|
POINTER_ID_GENERIC_FINGER,
|
||||||
|
540,
|
||||||
|
960,
|
||||||
|
1080,
|
||||||
|
1920,
|
||||||
|
1.0,
|
||||||
|
0,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
assert_eq!(msg, expected);
|
||||||
|
assert_eq!(msg.len(), 32);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn up_event_uses_zero_pressure() {
|
||||||
|
let event = PointerEvent {
|
||||||
|
action: MotionAction::Up,
|
||||||
|
norm_x: 0.5,
|
||||||
|
norm_y: 0.5,
|
||||||
|
};
|
||||||
|
let msg = touch_message(&event, VideoSize::new(1080, 1920));
|
||||||
|
assert_eq!(&msg[22..24], &[0x00, 0x00]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn key_message_sets_zero_repeat() {
|
||||||
|
let msg = key_message(KeyAction::Down, 4, 0);
|
||||||
|
assert_eq!(&msg[6..10], &[0, 0, 0, 0]); // repeat
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,355 @@
|
|||||||
|
//! scrcpy v4.0 server supervisor and video-stream demuxer (§4.2, §6.1).
|
||||||
|
//!
|
||||||
|
//! Lifecycle: push the server jar, `adb forward` a TCP port to the device's
|
||||||
|
//! `localabstract:scrcpy_<scid>`, launch the server via `app_process`, then connect
|
||||||
|
//! the video socket (first — carries a dummy byte, the 64-byte device name, and the
|
||||||
|
//! 12-byte codec header) followed by the control socket. Frames arrive as
|
||||||
|
//! `[u64 pts+flags][u32 len][payload]`.
|
||||||
|
|
||||||
|
use std::process::Stdio;
|
||||||
|
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||||
|
|
||||||
|
use anyhow::{bail, Context, Result};
|
||||||
|
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||||
|
use tokio::net::{TcpListener, TcpStream};
|
||||||
|
use tokio::process::{Child, Command};
|
||||||
|
|
||||||
|
pub const SERVER_DEVICE_PATH: &str = "/data/local/tmp/scrcpy-server.jar";
|
||||||
|
pub const SCRCPY_VERSION: &str = "4.0";
|
||||||
|
/// scrcpy codec id for H.264, the ASCII bytes "h264".
|
||||||
|
pub const CODEC_ID_H264: u32 = 0x6832_3634;
|
||||||
|
|
||||||
|
const DEVICE_NAME_LEN: usize = 64;
|
||||||
|
const CODEC_ID_SIZE: usize = 4;
|
||||||
|
const PACKET_HEADER_SIZE: usize = 12;
|
||||||
|
const SESSION_PACKET_FLAG: u8 = 0x80;
|
||||||
|
const FLAG_CONFIG: u64 = 1 << 62;
|
||||||
|
const FLAG_KEY_FRAME: u64 = 1 << 61;
|
||||||
|
const PTS_MASK: u64 = FLAG_KEY_FRAME - 1;
|
||||||
|
|
||||||
|
/// A 12-byte header with the MSB set is a "session" packet: video metadata
|
||||||
|
/// (width/height) with no payload, rather than a media frame.
|
||||||
|
pub fn is_session_packet(header: &[u8; PACKET_HEADER_SIZE]) -> bool {
|
||||||
|
header[0] & SESSION_PACKET_FLAG != 0
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Width and height carried by a session packet (bytes 4..8 and 8..12).
|
||||||
|
pub fn parse_session_packet(header: &[u8; PACKET_HEADER_SIZE]) -> (u32, u32) {
|
||||||
|
let width = u32::from_be_bytes(header[4..8].try_into().unwrap());
|
||||||
|
let height = u32::from_be_bytes(header[8..12].try_into().unwrap());
|
||||||
|
(width, height)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub struct PacketHeader {
|
||||||
|
pub config: bool,
|
||||||
|
pub keyframe: bool,
|
||||||
|
pub pts: u64,
|
||||||
|
pub len: u32,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn parse_packet_header(buf: &[u8; PACKET_HEADER_SIZE]) -> PacketHeader {
|
||||||
|
let pts_flags = u64::from_be_bytes(buf[0..8].try_into().unwrap());
|
||||||
|
let len = u32::from_be_bytes(buf[8..12].try_into().unwrap());
|
||||||
|
PacketHeader {
|
||||||
|
config: pts_flags & FLAG_CONFIG != 0,
|
||||||
|
keyframe: pts_flags & FLAG_KEY_FRAME != 0,
|
||||||
|
pts: pts_flags & PTS_MASK,
|
||||||
|
len,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A live H.264 access unit read off the video socket.
|
||||||
|
pub struct VideoPacket {
|
||||||
|
pub header: PacketHeader,
|
||||||
|
pub data: Vec<u8>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The video half of a session: reads H.264 media access units off the socket,
|
||||||
|
/// tracking the current frame dimensions from session packets.
|
||||||
|
pub struct VideoHalf {
|
||||||
|
stream: TcpStream,
|
||||||
|
/// Current video dimensions, set from session packets (0 until the first one).
|
||||||
|
pub width: u32,
|
||||||
|
pub height: u32,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl VideoHalf {
|
||||||
|
/// Read the next H.264 media access unit. Session packets (size metadata) are
|
||||||
|
/// consumed internally (updating `width`/`height`) and skipped.
|
||||||
|
pub async fn next_packet(&mut self) -> Result<VideoPacket> {
|
||||||
|
loop {
|
||||||
|
let mut header = [0u8; PACKET_HEADER_SIZE];
|
||||||
|
self.stream.read_exact(&mut header).await.context("packet header")?;
|
||||||
|
|
||||||
|
if is_session_packet(&header) {
|
||||||
|
let (w, h) = parse_session_packet(&header);
|
||||||
|
self.width = w;
|
||||||
|
self.height = h;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let parsed = parse_packet_header(&header);
|
||||||
|
let mut data = vec![0u8; parsed.len as usize];
|
||||||
|
self.stream.read_exact(&mut data).await.context("packet payload")?;
|
||||||
|
return Ok(VideoPacket { header: parsed, data });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Handle to the server process and reverse tunnel, kept for clean teardown after
|
||||||
|
/// a session is split into its independently-owned video and control halves.
|
||||||
|
pub struct ServerHandle {
|
||||||
|
serial: String,
|
||||||
|
socket_name: String,
|
||||||
|
server: Child,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ServerHandle {
|
||||||
|
pub async fn shutdown(mut self) -> Result<()> {
|
||||||
|
let _ = self.server.kill().await;
|
||||||
|
let _ = adb(&[
|
||||||
|
"-s",
|
||||||
|
&self.serial,
|
||||||
|
"reverse",
|
||||||
|
"--remove",
|
||||||
|
&format!("localabstract:{}", self.socket_name),
|
||||||
|
])
|
||||||
|
.await;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A running scrcpy session: the server process plus the connected video and
|
||||||
|
/// control sockets, with the negotiated device name and codec metadata.
|
||||||
|
pub struct StreamSession {
|
||||||
|
pub serial: String,
|
||||||
|
pub scid: u32,
|
||||||
|
pub socket_name: String,
|
||||||
|
pub device_name: String,
|
||||||
|
pub codec_id: u32,
|
||||||
|
pub video: VideoHalf,
|
||||||
|
pub control: TcpStream,
|
||||||
|
server: Child,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl StreamSession {
|
||||||
|
/// Push the server, launch it, and connect the sockets for a video+control
|
||||||
|
/// session (audio disabled — video-only per v1). Uses a reverse tunnel exactly
|
||||||
|
/// like the scrcpy client: the host listens, the server connects out (video
|
||||||
|
/// socket first, then control). No dummy byte in reverse mode.
|
||||||
|
pub async fn start(serial: &str, server_jar: &str, max_size: u32) -> Result<Self> {
|
||||||
|
push_server(serial, server_jar).await?;
|
||||||
|
|
||||||
|
let scid = new_scid();
|
||||||
|
let socket_name = format!("scrcpy_{scid:08x}");
|
||||||
|
|
||||||
|
let listener = TcpListener::bind("127.0.0.1:0")
|
||||||
|
.await
|
||||||
|
.context("bind reverse-tunnel listener")?;
|
||||||
|
let port = listener.local_addr()?.port();
|
||||||
|
|
||||||
|
adb(&[
|
||||||
|
"-s",
|
||||||
|
serial,
|
||||||
|
"reverse",
|
||||||
|
&format!("localabstract:{socket_name}"),
|
||||||
|
&format!("tcp:{port}"),
|
||||||
|
])
|
||||||
|
.await
|
||||||
|
.context("adb reverse")?;
|
||||||
|
|
||||||
|
// Match the scrcpy client's minimal invocation; extra args abort the server.
|
||||||
|
let mut server_args: Vec<String> = vec![
|
||||||
|
"-s".into(),
|
||||||
|
serial.into(),
|
||||||
|
"shell".into(),
|
||||||
|
format!("CLASSPATH={SERVER_DEVICE_PATH}"),
|
||||||
|
"app_process".into(),
|
||||||
|
"/".into(),
|
||||||
|
"com.genymobile.scrcpy.Server".into(),
|
||||||
|
SCRCPY_VERSION.into(),
|
||||||
|
format!("scid={scid:08x}"),
|
||||||
|
"log_level=info".into(),
|
||||||
|
"audio=false".into(),
|
||||||
|
];
|
||||||
|
if max_size > 0 {
|
||||||
|
server_args.push(format!("max_size={max_size}"));
|
||||||
|
}
|
||||||
|
|
||||||
|
let server = Command::new("adb")
|
||||||
|
.args(&server_args)
|
||||||
|
.stdout(Stdio::inherit())
|
||||||
|
.stderr(Stdio::inherit())
|
||||||
|
.spawn()
|
||||||
|
.context("spawn scrcpy server")?;
|
||||||
|
|
||||||
|
// The server connects the video socket first.
|
||||||
|
let mut video = accept(&listener).await.context("accept video socket")?;
|
||||||
|
|
||||||
|
let mut name_buf = [0u8; DEVICE_NAME_LEN];
|
||||||
|
video.read_exact(&mut name_buf).await.context("read device name")?;
|
||||||
|
let device_name = String::from_utf8_lossy(&name_buf)
|
||||||
|
.trim_end_matches('\0')
|
||||||
|
.trim()
|
||||||
|
.to_string();
|
||||||
|
|
||||||
|
let mut codec_buf = [0u8; CODEC_ID_SIZE];
|
||||||
|
video.read_exact(&mut codec_buf).await.context("read codec id")?;
|
||||||
|
let codec_id = u32::from_be_bytes(codec_buf);
|
||||||
|
|
||||||
|
// The control socket connects second.
|
||||||
|
let control = accept(&listener).await.context("accept control socket")?;
|
||||||
|
|
||||||
|
Ok(Self {
|
||||||
|
serial: serial.to_string(),
|
||||||
|
scid,
|
||||||
|
socket_name,
|
||||||
|
device_name,
|
||||||
|
codec_id,
|
||||||
|
video: VideoHalf {
|
||||||
|
stream: video,
|
||||||
|
width: 0,
|
||||||
|
height: 0,
|
||||||
|
},
|
||||||
|
control,
|
||||||
|
server,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read the next H.264 media access unit from the video socket.
|
||||||
|
pub async fn next_packet(&mut self) -> Result<VideoPacket> {
|
||||||
|
self.video.next_packet().await
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn width(&self) -> u32 {
|
||||||
|
self.video.width
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn height(&self) -> u32 {
|
||||||
|
self.video.height
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Send a scrcpy control message (from `crate::control`) to the device.
|
||||||
|
pub async fn send_control(&mut self, message: &[u8]) -> Result<()> {
|
||||||
|
self.control.write_all(message).await.context("write control")?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Split into independently-owned halves so the video pump and the input
|
||||||
|
/// handler can run concurrently, plus a handle for teardown.
|
||||||
|
pub fn split(self) -> (VideoHalf, TcpStream, ServerHandle) {
|
||||||
|
(
|
||||||
|
self.video,
|
||||||
|
self.control,
|
||||||
|
ServerHandle {
|
||||||
|
serial: self.serial,
|
||||||
|
socket_name: self.socket_name,
|
||||||
|
server: self.server,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Stop the server and remove the reverse tunnel.
|
||||||
|
pub async fn shutdown(self) -> Result<()> {
|
||||||
|
let (_video, _control, handle) = self.split();
|
||||||
|
handle.shutdown().await
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Accept the next inbound socket from the server, with a startup timeout.
|
||||||
|
async fn accept(listener: &TcpListener) -> Result<TcpStream> {
|
||||||
|
let (stream, _) = tokio::time::timeout(Duration::from_secs(10), listener.accept())
|
||||||
|
.await
|
||||||
|
.context("timed out waiting for scrcpy server to connect")??;
|
||||||
|
stream.set_nodelay(true).ok();
|
||||||
|
Ok(stream)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn push_server(serial: &str, server_jar: &str) -> Result<()> {
|
||||||
|
adb(&["-s", serial, "push", server_jar, SERVER_DEVICE_PATH])
|
||||||
|
.await
|
||||||
|
.context("adb push scrcpy-server")?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn adb(args: &[&str]) -> Result<()> {
|
||||||
|
let status = Command::new("adb")
|
||||||
|
.args(args)
|
||||||
|
.stdout(Stdio::null())
|
||||||
|
.stderr(Stdio::inherit())
|
||||||
|
.status()
|
||||||
|
.await
|
||||||
|
.context("running adb")?;
|
||||||
|
if !status.success() {
|
||||||
|
bail!("adb {:?} failed with {status}", args);
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn new_scid() -> u32 {
|
||||||
|
let nanos = SystemTime::now()
|
||||||
|
.duration_since(UNIX_EPOCH)
|
||||||
|
.map(|d| d.subsec_nanos())
|
||||||
|
.unwrap_or(1);
|
||||||
|
(nanos & 0x7fff_ffff).max(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn session_packet_carries_dimensions_and_is_flagged() {
|
||||||
|
let mut buf = [0u8; 12];
|
||||||
|
buf[0] = 0x80; // session-packet flag (MSB set)
|
||||||
|
buf[4..8].copy_from_slice(&720u32.to_be_bytes());
|
||||||
|
buf[8..12].copy_from_slice(&1600u32.to_be_bytes());
|
||||||
|
assert!(is_session_packet(&buf));
|
||||||
|
assert_eq!(parse_session_packet(&buf), (720, 1600));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn media_packet_is_not_a_session_packet() {
|
||||||
|
let mut buf = [0u8; 12];
|
||||||
|
buf[0..8].copy_from_slice(&(1u64 << 61).to_be_bytes()); // keyframe, MSB clear
|
||||||
|
assert!(!is_session_packet(&buf));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn config_packet_flag_is_bit_62() {
|
||||||
|
let mut buf = [0u8; 12];
|
||||||
|
let pts_flags: u64 = 1 << 62; // CONFIG, pts 0
|
||||||
|
buf[0..8].copy_from_slice(&pts_flags.to_be_bytes());
|
||||||
|
buf[8..12].copy_from_slice(&37u32.to_be_bytes());
|
||||||
|
let h = parse_packet_header(&buf);
|
||||||
|
assert!(h.config);
|
||||||
|
assert!(!h.keyframe);
|
||||||
|
assert_eq!(h.len, 37);
|
||||||
|
assert_eq!(h.pts, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn keyframe_flag_is_bit_61_and_pts_masked() {
|
||||||
|
let mut buf = [0u8; 12];
|
||||||
|
let pts: u64 = 123_456;
|
||||||
|
let pts_flags: u64 = (1 << 61) | pts; // KEY_FRAME + pts
|
||||||
|
buf[0..8].copy_from_slice(&pts_flags.to_be_bytes());
|
||||||
|
buf[8..12].copy_from_slice(&9000u32.to_be_bytes());
|
||||||
|
let h = parse_packet_header(&buf);
|
||||||
|
assert!(h.keyframe);
|
||||||
|
assert!(!h.config);
|
||||||
|
assert_eq!(h.pts, pts);
|
||||||
|
assert_eq!(h.len, 9000);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn plain_frame_has_no_flags() {
|
||||||
|
let mut buf = [0u8; 12];
|
||||||
|
buf[0..8].copy_from_slice(&500u64.to_be_bytes());
|
||||||
|
buf[8..12].copy_from_slice(&1234u32.to_be_bytes());
|
||||||
|
let h = parse_packet_header(&buf);
|
||||||
|
assert!(!h.config && !h.keyframe);
|
||||||
|
assert_eq!(h.pts, 500);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,155 @@
|
|||||||
|
//! webrtc-rs bridge (§4.2, §4.6): owns a peer connection with a single H.264 video
|
||||||
|
//! track and feeds it the scrcpy access units. SDP is exchanged vanilla-ICE (gather
|
||||||
|
//! to completion, then hand over the full description) — simplest for the spike;
|
||||||
|
//! trickle/TURN come with the relay path (Phase 3).
|
||||||
|
|
||||||
|
use std::sync::atomic::{AtomicU32, Ordering};
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use anyhow::{Context, Result};
|
||||||
|
use bytes::Bytes;
|
||||||
|
use webrtc::api::interceptor_registry::register_default_interceptors;
|
||||||
|
use webrtc::api::media_engine::{MediaEngine, MIME_TYPE_H264};
|
||||||
|
use webrtc::api::APIBuilder;
|
||||||
|
use webrtc::ice_transport::ice_server::RTCIceServer;
|
||||||
|
use webrtc::interceptor::registry::Registry;
|
||||||
|
use webrtc::media::Sample;
|
||||||
|
use webrtc::peer_connection::configuration::RTCConfiguration;
|
||||||
|
use webrtc::peer_connection::sdp::session_description::RTCSessionDescription;
|
||||||
|
use webrtc::peer_connection::RTCPeerConnection;
|
||||||
|
use webrtc::rtp_transceiver::rtp_codec::RTCRtpCodecCapability;
|
||||||
|
use webrtc::track::track_local::track_local_static_sample::TrackLocalStaticSample;
|
||||||
|
|
||||||
|
use crate::scrcpy::VideoHalf;
|
||||||
|
|
||||||
|
/// Shared current video dimensions (width, height), published by the video pump so
|
||||||
|
/// the input handler can scale normalized coordinates against the live frame size.
|
||||||
|
pub type SharedSize = Arc<(AtomicU32, AtomicU32)>;
|
||||||
|
|
||||||
|
pub struct VideoBridge {
|
||||||
|
pub pc: Arc<RTCPeerConnection>,
|
||||||
|
track: Arc<TrackLocalStaticSample>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl VideoBridge {
|
||||||
|
/// Build a peer connection with a single sendonly H.264 track.
|
||||||
|
pub async fn new(stun_urls: &[String]) -> Result<Self> {
|
||||||
|
let mut media = MediaEngine::default();
|
||||||
|
media.register_default_codecs().context("register codecs")?;
|
||||||
|
|
||||||
|
let mut registry = Registry::new();
|
||||||
|
registry = register_default_interceptors(registry, &mut media)
|
||||||
|
.context("register interceptors")?;
|
||||||
|
|
||||||
|
let api = APIBuilder::new()
|
||||||
|
.with_media_engine(media)
|
||||||
|
.with_interceptor_registry(registry)
|
||||||
|
.build();
|
||||||
|
|
||||||
|
let config = RTCConfiguration {
|
||||||
|
ice_servers: if stun_urls.is_empty() {
|
||||||
|
vec![]
|
||||||
|
} else {
|
||||||
|
vec![RTCIceServer {
|
||||||
|
urls: stun_urls.to_vec(),
|
||||||
|
..Default::default()
|
||||||
|
}]
|
||||||
|
},
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
|
||||||
|
let pc = Arc::new(api.new_peer_connection(config).await.context("new peer connection")?);
|
||||||
|
|
||||||
|
let track = Arc::new(TrackLocalStaticSample::new(
|
||||||
|
RTCRtpCodecCapability {
|
||||||
|
mime_type: MIME_TYPE_H264.to_owned(),
|
||||||
|
..Default::default()
|
||||||
|
},
|
||||||
|
"video".to_owned(),
|
||||||
|
"scrcpy".to_owned(),
|
||||||
|
));
|
||||||
|
pc.add_track(track.clone()).await.context("add video track")?;
|
||||||
|
|
||||||
|
Ok(Self { pc, track })
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Accept a remote offer and produce an SDP answer (waiting for ICE gathering).
|
||||||
|
pub async fn answer(&self, offer_sdp: String) -> Result<String> {
|
||||||
|
let offer = RTCSessionDescription::offer(offer_sdp).context("parse offer")?;
|
||||||
|
self.pc.set_remote_description(offer).await.context("set remote")?;
|
||||||
|
|
||||||
|
let answer = self.pc.create_answer(None).await.context("create answer")?;
|
||||||
|
let mut gather_complete = self.pc.gathering_complete_promise().await;
|
||||||
|
self.pc.set_local_description(answer).await.context("set local")?;
|
||||||
|
let _ = gather_complete.recv().await;
|
||||||
|
|
||||||
|
let local = self.pc.local_description().await.context("no local description")?;
|
||||||
|
Ok(local.sdp)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn track(&self) -> Arc<TrackLocalStaticSample> {
|
||||||
|
self.track.clone()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Write one H.264 access unit (Annex-B) to the track; webrtc-rs packetizes it.
|
||||||
|
pub async fn write_access_unit(&self, data: Bytes, duration: Duration) -> Result<()> {
|
||||||
|
write_access_unit(&self.track, data, duration).await
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn write_access_unit(
|
||||||
|
track: &TrackLocalStaticSample,
|
||||||
|
data: Bytes,
|
||||||
|
duration: Duration,
|
||||||
|
) -> Result<()> {
|
||||||
|
track
|
||||||
|
.write_sample(&Sample {
|
||||||
|
data,
|
||||||
|
duration,
|
||||||
|
..Default::default()
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.context("write sample")?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Pump scrcpy access units into the track until the video socket closes. SPS/PPS
|
||||||
|
/// (the config packet) is cached and prepended to each keyframe so a freshly-joined
|
||||||
|
/// decoder always has parameter sets. Frame duration comes from the pts delta.
|
||||||
|
pub async fn pump_video(
|
||||||
|
video: &mut VideoHalf,
|
||||||
|
track: Arc<TrackLocalStaticSample>,
|
||||||
|
size: SharedSize,
|
||||||
|
) -> Result<()> {
|
||||||
|
let mut config: Vec<u8> = Vec::new();
|
||||||
|
let mut last_pts: Option<u64> = None;
|
||||||
|
|
||||||
|
loop {
|
||||||
|
let pkt = video.next_packet().await?;
|
||||||
|
size.0.store(video.width, Ordering::Relaxed);
|
||||||
|
size.1.store(video.height, Ordering::Relaxed);
|
||||||
|
|
||||||
|
if pkt.header.config {
|
||||||
|
config = pkt.data;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let duration = match last_pts {
|
||||||
|
Some(prev) if pkt.header.pts > prev => Duration::from_micros(pkt.header.pts - prev),
|
||||||
|
_ => Duration::from_millis(33),
|
||||||
|
};
|
||||||
|
last_pts = Some(pkt.header.pts);
|
||||||
|
|
||||||
|
let data = if pkt.header.keyframe && !config.is_empty() {
|
||||||
|
let mut unit = Vec::with_capacity(config.len() + pkt.data.len());
|
||||||
|
unit.extend_from_slice(&config);
|
||||||
|
unit.extend_from_slice(&pkt.data);
|
||||||
|
Bytes::from(unit)
|
||||||
|
} else {
|
||||||
|
Bytes::from(pkt.data)
|
||||||
|
};
|
||||||
|
|
||||||
|
write_access_unit(&track, data, duration).await?;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
//! End-to-end test of the full phone→browser path: spawn streamd, act as the
|
||||||
|
//! browser (webrtc-rs receiver), POST a real offer, and confirm actual RTP video
|
||||||
|
//! arrives from the device. Ignored by default — needs a device attached and adb.
|
||||||
|
//! cargo test -p stream-node --test streamd_e2e -- --ignored
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use tokio::sync::mpsc;
|
||||||
|
use webrtc::api::interceptor_registry::register_default_interceptors;
|
||||||
|
use webrtc::api::media_engine::MediaEngine;
|
||||||
|
use webrtc::api::APIBuilder;
|
||||||
|
use webrtc::interceptor::registry::Registry;
|
||||||
|
use webrtc::peer_connection::configuration::RTCConfiguration;
|
||||||
|
use webrtc::peer_connection::sdp::session_description::RTCSessionDescription;
|
||||||
|
use webrtc::rtp_transceiver::rtp_codec::RTPCodecType;
|
||||||
|
|
||||||
|
const BASE: &str = "http://127.0.0.1:18096";
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
#[ignore = "needs a device + adb; spawns streamd"]
|
||||||
|
async fn streamd_streams_real_device_video() {
|
||||||
|
let serial = std::env::var("DEVICE_SERIAL").unwrap_or_else(|_| "A11PRO052500527".into());
|
||||||
|
|
||||||
|
let mut streamd = tokio::process::Command::new(env!("CARGO_BIN_EXE_streamd"))
|
||||||
|
.env("BIND_ADDR", "127.0.0.1:18096")
|
||||||
|
.env("DEVICE_SERIAL", serial)
|
||||||
|
.kill_on_drop(true)
|
||||||
|
.spawn()
|
||||||
|
.expect("spawn streamd");
|
||||||
|
|
||||||
|
wait_until_up().await;
|
||||||
|
|
||||||
|
// Build a receiver peer (the "browser").
|
||||||
|
let mut media = MediaEngine::default();
|
||||||
|
media.register_default_codecs().unwrap();
|
||||||
|
let mut registry = Registry::new();
|
||||||
|
registry = register_default_interceptors(registry, &mut media).unwrap();
|
||||||
|
let api = APIBuilder::new()
|
||||||
|
.with_media_engine(media)
|
||||||
|
.with_interceptor_registry(registry)
|
||||||
|
.build();
|
||||||
|
let pc = Arc::new(api.new_peer_connection(RTCConfiguration::default()).await.unwrap());
|
||||||
|
pc.add_transceiver_from_kind(RTPCodecType::Video, None).await.unwrap();
|
||||||
|
|
||||||
|
let (tx, mut rx) = mpsc::channel::<usize>(4);
|
||||||
|
pc.on_track(Box::new(move |track, _, _| {
|
||||||
|
let tx = tx.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
tokio::spawn(async move {
|
||||||
|
while let Ok((pkt, _)) = track.read_rtp().await {
|
||||||
|
if !pkt.payload.is_empty() {
|
||||||
|
let _ = tx.send(pkt.payload.len()).await;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
})
|
||||||
|
}));
|
||||||
|
|
||||||
|
// Offer (vanilla ICE) → POST to streamd → answer.
|
||||||
|
pc.set_local_description(pc.create_offer(None).await.unwrap()).await.unwrap();
|
||||||
|
let mut gather = pc.gathering_complete_promise().await;
|
||||||
|
let _ = gather.recv().await;
|
||||||
|
let offer_sdp = pc.local_description().await.unwrap().sdp;
|
||||||
|
|
||||||
|
let resp: serde_json::Value = reqwest::Client::new()
|
||||||
|
.post(format!("{BASE}/offer"))
|
||||||
|
.json(&serde_json::json!({ "sdp": offer_sdp }))
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.expect("POST /offer")
|
||||||
|
.json()
|
||||||
|
.await
|
||||||
|
.expect("answer json");
|
||||||
|
let answer_sdp = resp["sdp"].as_str().expect("answer sdp").to_string();
|
||||||
|
pc.set_remote_description(RTCSessionDescription::answer(answer_sdp).unwrap())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let got = tokio::time::timeout(Duration::from_secs(20), rx.recv()).await;
|
||||||
|
let _ = pc.close().await;
|
||||||
|
let _ = streamd.kill().await;
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
matches!(got, Ok(Some(n)) if n > 0),
|
||||||
|
"no RTP video arrived from the device through streamd: {got:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn wait_until_up() {
|
||||||
|
let client = reqwest::Client::new();
|
||||||
|
for _ in 0..100 {
|
||||||
|
if client.get(format!("{BASE}/")).send().await.map(|r| r.status().is_success()).unwrap_or(false) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
tokio::time::sleep(Duration::from_millis(100)).await;
|
||||||
|
}
|
||||||
|
panic!("streamd did not come up");
|
||||||
|
}
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
//! Headless verification of the webrtc-rs media path: the VideoBridge (sender) and a
|
||||||
|
//! second webrtc-rs peer (receiver) negotiate in-process, the bridge writes H.264
|
||||||
|
//! samples, and we confirm RTP actually arrives on the receiver's track — no browser.
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use bytes::Bytes;
|
||||||
|
use stream_node::webrtc_bridge::VideoBridge;
|
||||||
|
use tokio::sync::mpsc;
|
||||||
|
use webrtc::api::interceptor_registry::register_default_interceptors;
|
||||||
|
use webrtc::api::media_engine::MediaEngine;
|
||||||
|
use webrtc::api::APIBuilder;
|
||||||
|
use webrtc::media::Sample;
|
||||||
|
use webrtc::peer_connection::configuration::RTCConfiguration;
|
||||||
|
use webrtc::peer_connection::sdp::session_description::RTCSessionDescription;
|
||||||
|
use webrtc::rtp_transceiver::rtp_codec::RTPCodecType;
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn h264_track_flows_to_a_receiver() {
|
||||||
|
// Sender: our bridge with a sendonly H.264 track.
|
||||||
|
let sender = VideoBridge::new(&[]).await.unwrap();
|
||||||
|
|
||||||
|
// Receiver: a plain webrtc-rs peer that offers a recvonly video transceiver.
|
||||||
|
let mut media = MediaEngine::default();
|
||||||
|
media.register_default_codecs().unwrap();
|
||||||
|
let mut registry = webrtc::interceptor::registry::Registry::new();
|
||||||
|
registry = register_default_interceptors(registry, &mut media).unwrap();
|
||||||
|
let api = APIBuilder::new()
|
||||||
|
.with_media_engine(media)
|
||||||
|
.with_interceptor_registry(registry)
|
||||||
|
.build();
|
||||||
|
let receiver = Arc::new(api.new_peer_connection(RTCConfiguration::default()).await.unwrap());
|
||||||
|
receiver
|
||||||
|
.add_transceiver_from_kind(RTPCodecType::Video, None)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
// Signal "first RTP payload arrived" out of the on_track handler.
|
||||||
|
let (tx, mut rx) = mpsc::channel::<usize>(4);
|
||||||
|
receiver.on_track(Box::new(move |track, _receiver, _transceiver| {
|
||||||
|
let tx = tx.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
tokio::spawn(async move {
|
||||||
|
while let Ok((pkt, _)) = track.read_rtp().await {
|
||||||
|
if !pkt.payload.is_empty() {
|
||||||
|
let _ = tx.send(pkt.payload.len()).await;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
})
|
||||||
|
}));
|
||||||
|
|
||||||
|
// Vanilla-ICE signaling: receiver offers, sender answers.
|
||||||
|
let offer = receiver.create_offer(None).await.unwrap();
|
||||||
|
let mut gather = receiver.gathering_complete_promise().await;
|
||||||
|
receiver.set_local_description(offer).await.unwrap();
|
||||||
|
let _ = gather.recv().await;
|
||||||
|
let offer_sdp = receiver.local_description().await.unwrap().sdp;
|
||||||
|
|
||||||
|
let answer_sdp = sender.answer(offer_sdp).await.unwrap();
|
||||||
|
receiver
|
||||||
|
.set_remote_description(RTCSessionDescription::answer(answer_sdp).unwrap())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
// Feed synthetic Annex-B NALs until the receiver reports a payload (content is
|
||||||
|
// irrelevant for verifying the transport — the H.264 payloader just splits NALs).
|
||||||
|
let track = sender.track();
|
||||||
|
let feeder = tokio::spawn(async move {
|
||||||
|
for _ in 0..400 {
|
||||||
|
let nal = Bytes::from_static(&[0, 0, 0, 1, 0x65, 0x88, 0x84, 0x21, 0x42, 0x63]);
|
||||||
|
let _ = track
|
||||||
|
.write_sample(&Sample {
|
||||||
|
data: nal,
|
||||||
|
duration: Duration::from_millis(33),
|
||||||
|
..Default::default()
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
tokio::time::sleep(Duration::from_millis(20)).await;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
let got = tokio::time::timeout(Duration::from_secs(15), rx.recv()).await;
|
||||||
|
feeder.abort();
|
||||||
|
let _ = receiver.close().await;
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
matches!(got, Ok(Some(n)) if n > 0),
|
||||||
|
"receiver never got an RTP payload over the negotiated H.264 track: {got:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
models/
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
[package]
|
||||||
|
name = "transcription-svc"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2021"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
axum = "0.7"
|
||||||
|
tokio = { version = "1", features = ["full"] }
|
||||||
|
async-trait = "0.1"
|
||||||
|
anyhow = "1"
|
||||||
|
serde_json = "1"
|
||||||
|
uuid = { version = "1", features = ["v4"] }
|
||||||
|
tracing = "0.1"
|
||||||
|
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||||
|
whisper-rs = "0.14"
|
||||||
|
|
||||||
|
[dev-dependencies]
|
||||||
|
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json"] }
|
||||||
Executable
+18
@@ -0,0 +1,18 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Fetch a ggml Whisper model for the transcription service.
|
||||||
|
# Usage: ./scripts/fetch-model.sh [model-name] (default: tiny.en)
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
MODEL="${1:-tiny.en}"
|
||||||
|
DIR="$(cd "$(dirname "$0")/.." && pwd)/models"
|
||||||
|
OUT="$DIR/ggml-${MODEL}.bin"
|
||||||
|
URL="https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-${MODEL}.bin"
|
||||||
|
|
||||||
|
mkdir -p "$DIR"
|
||||||
|
if [ -f "$OUT" ]; then
|
||||||
|
echo "model already present: $OUT"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
echo "downloading $URL"
|
||||||
|
curl -fL --retry 3 -o "$OUT" "$URL"
|
||||||
|
echo "saved $OUT"
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
//! Self-hosted Whisper-class transcription service (§6.4). A separate service from
|
||||||
|
//! the review API so transcription load never touches the real-time path, and on
|
||||||
|
//! RedClaw infrastructure so partner findings never leave RedClaw control.
|
||||||
|
|
||||||
|
pub mod whisper;
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use async_trait::async_trait;
|
||||||
|
use axum::{
|
||||||
|
body::Bytes,
|
||||||
|
extract::State,
|
||||||
|
http::{HeaderMap, StatusCode},
|
||||||
|
routing::{get, post},
|
||||||
|
Json, Router,
|
||||||
|
};
|
||||||
|
use serde_json::{json, Value};
|
||||||
|
|
||||||
|
/// Audio → text. Injected so the HTTP layer is testable without a model, and so
|
||||||
|
/// the real engine (whisper.cpp) can be swapped without touching the service.
|
||||||
|
#[async_trait]
|
||||||
|
pub trait Transcriber: Send + Sync {
|
||||||
|
async fn transcribe(&self, audio: &[u8], content_type: &str) -> anyhow::Result<String>;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct AppState {
|
||||||
|
pub transcriber: Arc<dyn Transcriber>,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn build_router(state: AppState) -> Router {
|
||||||
|
Router::new()
|
||||||
|
.route("/health", get(health))
|
||||||
|
.route("/transcribe", post(transcribe))
|
||||||
|
.with_state(state)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn health() -> Json<Value> {
|
||||||
|
Json(json!({ "status": "ok" }))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Accepts a raw audio body (any container ffmpeg can read) and returns `{ "text": ... }`.
|
||||||
|
async fn transcribe(
|
||||||
|
State(st): State<AppState>,
|
||||||
|
headers: HeaderMap,
|
||||||
|
body: Bytes,
|
||||||
|
) -> Result<Json<Value>, (StatusCode, String)> {
|
||||||
|
if body.is_empty() {
|
||||||
|
return Err((StatusCode::BAD_REQUEST, "empty audio body".to_string()));
|
||||||
|
}
|
||||||
|
let content_type = headers
|
||||||
|
.get("content-type")
|
||||||
|
.and_then(|v| v.to_str().ok())
|
||||||
|
.unwrap_or("application/octet-stream");
|
||||||
|
|
||||||
|
match st.transcriber.transcribe(&body, content_type).await {
|
||||||
|
Ok(text) => Ok(Json(json!({ "text": text }))),
|
||||||
|
Err(e) => Err((
|
||||||
|
StatusCode::INTERNAL_SERVER_ERROR,
|
||||||
|
format!("transcription failed: {e}"),
|
||||||
|
)),
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use anyhow::Context;
|
||||||
|
use transcription_svc::{build_router, whisper::WhisperTranscriber, AppState};
|
||||||
|
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
|
||||||
|
|
||||||
|
#[tokio::main]
|
||||||
|
async fn main() -> anyhow::Result<()> {
|
||||||
|
tracing_subscriber::registry()
|
||||||
|
.with(
|
||||||
|
tracing_subscriber::EnvFilter::try_from_default_env()
|
||||||
|
.unwrap_or_else(|_| "transcription_svc=info,info".into()),
|
||||||
|
)
|
||||||
|
.with(tracing_subscriber::fmt::layer())
|
||||||
|
.init();
|
||||||
|
|
||||||
|
let model = std::env::var("WHISPER_MODEL").unwrap_or_else(|_| "models/ggml-tiny.bin".into());
|
||||||
|
let transcriber = Arc::new(
|
||||||
|
WhisperTranscriber::new(&model)
|
||||||
|
.with_context(|| format!("initializing whisper from {model}"))?,
|
||||||
|
);
|
||||||
|
|
||||||
|
let app = build_router(AppState { transcriber });
|
||||||
|
|
||||||
|
let addr = std::env::var("BIND_ADDR").unwrap_or_else(|_| "0.0.0.0:8099".to_string());
|
||||||
|
let listener = tokio::net::TcpListener::bind(&addr).await?;
|
||||||
|
tracing::info!("transcription-svc listening on http://{addr} (model {model})");
|
||||||
|
axum::serve(listener, app).await?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -0,0 +1,92 @@
|
|||||||
|
//! Real transcription engine: ffmpeg normalizes any input container to 16 kHz mono
|
||||||
|
//! f32 PCM, then whisper.cpp (via `whisper-rs`) runs inference. This is the path the
|
||||||
|
//! deployed service uses; `WHISPER_MODEL` points at a ggml model file.
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use async_trait::async_trait;
|
||||||
|
use anyhow::{bail, Context};
|
||||||
|
use whisper_rs::{FullParams, SamplingStrategy, WhisperContext, WhisperContextParameters};
|
||||||
|
|
||||||
|
use crate::Transcriber;
|
||||||
|
|
||||||
|
pub struct WhisperTranscriber {
|
||||||
|
ctx: Arc<WhisperContext>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl WhisperTranscriber {
|
||||||
|
pub fn new(model_path: &str) -> anyhow::Result<Self> {
|
||||||
|
let ctx = WhisperContext::new_with_params(model_path, WhisperContextParameters::default())
|
||||||
|
.with_context(|| format!("loading whisper model from {model_path}"))?;
|
||||||
|
Ok(Self { ctx: Arc::new(ctx) })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[async_trait]
|
||||||
|
impl Transcriber for WhisperTranscriber {
|
||||||
|
async fn transcribe(&self, audio: &[u8], _content_type: &str) -> anyhow::Result<String> {
|
||||||
|
let samples = decode_to_pcm_f32_16k_mono(audio).await?;
|
||||||
|
if samples.is_empty() {
|
||||||
|
bail!("decoded audio is empty");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Inference is CPU-bound and blocking; move it off the async runtime.
|
||||||
|
let ctx = self.ctx.clone();
|
||||||
|
let text = tokio::task::spawn_blocking(move || run_whisper(&ctx, &samples)).await??;
|
||||||
|
Ok(text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn run_whisper(ctx: &WhisperContext, samples: &[f32]) -> anyhow::Result<String> {
|
||||||
|
let mut state = ctx.create_state().context("creating whisper state")?;
|
||||||
|
|
||||||
|
let mut params = FullParams::new(SamplingStrategy::Greedy { best_of: 1 });
|
||||||
|
let threads = std::thread::available_parallelism()
|
||||||
|
.map(|n| n.get() as i32)
|
||||||
|
.unwrap_or(4);
|
||||||
|
params.set_n_threads(threads);
|
||||||
|
params.set_print_special(false);
|
||||||
|
params.set_print_progress(false);
|
||||||
|
params.set_print_realtime(false);
|
||||||
|
params.set_print_timestamps(false);
|
||||||
|
|
||||||
|
state.full(params, samples).context("whisper inference")?;
|
||||||
|
|
||||||
|
let n = state.full_n_segments().context("counting segments")?;
|
||||||
|
let mut text = String::new();
|
||||||
|
for i in 0..n {
|
||||||
|
text.push_str(&state.full_get_segment_text(i).context("reading segment")?);
|
||||||
|
}
|
||||||
|
Ok(text.trim().to_string())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Decode arbitrary audio bytes to 16 kHz mono f32 PCM via ffmpeg.
|
||||||
|
async fn decode_to_pcm_f32_16k_mono(audio: &[u8]) -> anyhow::Result<Vec<f32>> {
|
||||||
|
let input = std::env::temp_dir().join(format!("clawreview-asr-{}", uuid::Uuid::new_v4()));
|
||||||
|
tokio::fs::write(&input, audio)
|
||||||
|
.await
|
||||||
|
.context("writing temp audio")?;
|
||||||
|
|
||||||
|
let result = tokio::process::Command::new("ffmpeg")
|
||||||
|
.args(["-nostdin", "-loglevel", "error", "-i"])
|
||||||
|
.arg(&input)
|
||||||
|
.args(["-f", "f32le", "-ac", "1", "-ar", "16000", "pipe:1"])
|
||||||
|
.output()
|
||||||
|
.await
|
||||||
|
.context("spawning ffmpeg (is it installed?)")?;
|
||||||
|
|
||||||
|
tokio::fs::remove_file(&input).await.ok();
|
||||||
|
|
||||||
|
if !result.status.success() {
|
||||||
|
bail!(
|
||||||
|
"ffmpeg decode failed: {}",
|
||||||
|
String::from_utf8_lossy(&result.stderr)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(result
|
||||||
|
.stdout
|
||||||
|
.chunks_exact(4)
|
||||||
|
.map(|b| f32::from_le_bytes([b[0], b[1], b[2], b[3]]))
|
||||||
|
.collect())
|
||||||
|
}
|
||||||
BIN
Binary file not shown.
@@ -0,0 +1,69 @@
|
|||||||
|
//! HTTP-contract tests for the transcription service, using a fake transcriber so
|
||||||
|
//! they run fast with no model. The real whisper path is covered in `whisper.rs`.
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use async_trait::async_trait;
|
||||||
|
use serde_json::Value;
|
||||||
|
use transcription_svc::{build_router, AppState, Transcriber};
|
||||||
|
|
||||||
|
struct FakeTranscriber {
|
||||||
|
reply: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[async_trait]
|
||||||
|
impl Transcriber for FakeTranscriber {
|
||||||
|
async fn transcribe(&self, audio: &[u8], _content_type: &str) -> anyhow::Result<String> {
|
||||||
|
assert!(!audio.is_empty(), "handler must not call transcriber on empty body");
|
||||||
|
Ok(self.reply.clone())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn spawn(reply: &str) -> (String, reqwest::Client) {
|
||||||
|
let state = AppState {
|
||||||
|
transcriber: Arc::new(FakeTranscriber {
|
||||||
|
reply: reply.to_string(),
|
||||||
|
}),
|
||||||
|
};
|
||||||
|
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let addr = listener.local_addr().unwrap();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
axum::serve(listener, build_router(state)).await.unwrap();
|
||||||
|
});
|
||||||
|
(format!("http://{addr}"), reqwest::Client::new())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn transcribe_returns_text_json() {
|
||||||
|
let (base, client) = spawn("nfc read resolved quickly").await;
|
||||||
|
let res = client
|
||||||
|
.post(format!("{base}/transcribe"))
|
||||||
|
.header("content-type", "audio/webm")
|
||||||
|
.body(b"-some-audio-".to_vec())
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(res.status(), 200);
|
||||||
|
let body: Value = res.json().await.unwrap();
|
||||||
|
assert_eq!(body["text"], "nfc read resolved quickly");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn empty_body_is_400() {
|
||||||
|
let (base, client) = spawn("unused").await;
|
||||||
|
let res = client
|
||||||
|
.post(format!("{base}/transcribe"))
|
||||||
|
.header("content-type", "audio/webm")
|
||||||
|
.body(Vec::<u8>::new())
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(res.status(), 400);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn health_ok() {
|
||||||
|
let (base, client) = spawn("unused").await;
|
||||||
|
let res = client.get(format!("{base}/health")).send().await.unwrap();
|
||||||
|
assert_eq!(res.status(), 200);
|
||||||
|
}
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
//! End-to-end test of the real whisper.cpp engine: a recorded speech fixture is
|
||||||
|
//! decoded by ffmpeg and transcribed. Ignored by default because it needs a
|
||||||
|
//! downloaded model (`scripts/fetch-model.sh`) and ffmpeg on PATH; run with:
|
||||||
|
//! cargo test -p transcription-svc --test whisper -- --ignored
|
||||||
|
|
||||||
|
use std::path::PathBuf;
|
||||||
|
|
||||||
|
use transcription_svc::{whisper::WhisperTranscriber, Transcriber};
|
||||||
|
|
||||||
|
fn manifest(rel: &str) -> PathBuf {
|
||||||
|
PathBuf::from(env!("CARGO_MANIFEST_DIR")).join(rel)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
#[ignore = "needs a downloaded model and ffmpeg"]
|
||||||
|
async fn transcribes_real_speech() {
|
||||||
|
let model = std::env::var("WHISPER_MODEL")
|
||||||
|
.map(PathBuf::from)
|
||||||
|
.unwrap_or_else(|_| manifest("models/ggml-tiny.en.bin"));
|
||||||
|
assert!(
|
||||||
|
model.exists(),
|
||||||
|
"model not found at {model:?} — run scripts/fetch-model.sh"
|
||||||
|
);
|
||||||
|
|
||||||
|
let transcriber = WhisperTranscriber::new(model.to_str().unwrap()).expect("load model");
|
||||||
|
let audio = std::fs::read(manifest("tests/fixtures/speech.wav")).expect("read fixture");
|
||||||
|
|
||||||
|
let text = transcriber
|
||||||
|
.transcribe(&audio, "audio/wav")
|
||||||
|
.await
|
||||||
|
.expect("transcribe")
|
||||||
|
.to_lowercase();
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
text.contains("fox"),
|
||||||
|
"expected 'fox' in transcript, got: {text:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user