Live Anthropic validation — and the platform-breaking bug it caught

Running the opt-in live suite (CM_LIVE_LLM=1 + ANTHROPIC_API_KEY) against
the real API immediately surfaced a launch blocker: Anthropic (and
OpenAI) restrict tool names to ^[a-zA-Z0-9_-]{1,128}$ — our ENTIRE
registry uses dotted names (clock.now, email.send, shell.exec, ...).
The scripted provider never enforced the pattern, so every real-model
deployment would have 400'd on the first tool call.

- Fix at the provider boundary, where it belongs: wire_tool_name /
  internal_tool_name codec (dots <-> __) applied in BOTH HTTP providers
  at all three sites (tools list, assistant tool_use echo, inbound
  tool_use decode). Internal naming (DB step rows, scenarios, UI traces)
  unchanged. Offline unit test round-trips every registry name through
  the wire pattern
- New live tests, all passing against api.anthropic.com (Haiku 4.5):
  - provider tool ROUND TRIP: real ToolUse arrives, ToolResult ships
    back exactly as a checkpoint would reassemble it, model completes,
    real usage events on the wire
  - full runtime loop: real model calls clock.now, run completes, REAL
    token usage metered, credits decremented
  - the #1-risk validation: a real model's email.send intercepted ->
    suspended -> approved -> checkpoint RESUMED against the live API ->
    completed -> outbox exactly 1 (checkpoint/resume fidelity end to end)
- Stray TC_OPENAI_COMPAT_* envs renamed to CM_OPENAI_COMPAT_*

No credentials stored anywhere; the key was passed via env only.

163 Rust tests (+5 live, key-gated).

Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-06-10 12:55:51 -05:00
co-authored by Claude Fable 5
parent add4f79fed
commit 5407111a89
6 changed files with 342 additions and 12 deletions
+4 -2
View File
@@ -39,7 +39,8 @@ impl OpenAiCompatProvider {
ContentPart::ToolUse { id, name, input } => tool_calls.push(json!({
"id": id,
"type": "function",
"function": {"name": name, "arguments": input.to_string()},
"function": {"name": crate::wire_tool_name(name),
"arguments": input.to_string()},
})),
ContentPart::ToolResult {
tool_use_id,
@@ -85,7 +86,7 @@ impl LlmProvider for OpenAiCompatProvider {
json!({
"type": "function",
"function": {
"name": t.name,
"name": crate::wire_tool_name(&t.name),
"description": t.description,
"parameters": t.input_schema,
},
@@ -168,6 +169,7 @@ impl LlmProvider for OpenAiCompatProvider {
}
}
for (id, name, args) in pending.drain(..) {
let name = crate::internal_tool_name(&name);
if name.is_empty() {
continue;
}