Live Anthropic validation — and the platform-breaking bug it caught
Running the opt-in live suite (CM_LIVE_LLM=1 + ANTHROPIC_API_KEY) against
the real API immediately surfaced a launch blocker: Anthropic (and
OpenAI) restrict tool names to ^[a-zA-Z0-9_-]{1,128}$ — our ENTIRE
registry uses dotted names (clock.now, email.send, shell.exec, ...).
The scripted provider never enforced the pattern, so every real-model
deployment would have 400'd on the first tool call.
- Fix at the provider boundary, where it belongs: wire_tool_name /
internal_tool_name codec (dots <-> __) applied in BOTH HTTP providers
at all three sites (tools list, assistant tool_use echo, inbound
tool_use decode). Internal naming (DB step rows, scenarios, UI traces)
unchanged. Offline unit test round-trips every registry name through
the wire pattern
- New live tests, all passing against api.anthropic.com (Haiku 4.5):
- provider tool ROUND TRIP: real ToolUse arrives, ToolResult ships
back exactly as a checkpoint would reassemble it, model completes,
real usage events on the wire
- full runtime loop: real model calls clock.now, run completes, REAL
token usage metered, credits decremented
- the #1-risk validation: a real model's email.send intercepted ->
suspended -> approved -> checkpoint RESUMED against the live API ->
completed -> outbox exactly 1 (checkpoint/resume fidelity end to end)
- Stray TC_OPENAI_COMPAT_* envs renamed to CM_OPENAI_COMPAT_*
No credentials stored anywhere; the key was passed via env only.
163 Rust tests (+5 live, key-gated).
Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Fable 5
parent
add4f79fed
commit
5407111a89
@@ -32,6 +32,20 @@ pub enum ContentPart {
|
||||
},
|
||||
}
|
||||
|
||||
/// Anthropic and OpenAI both restrict tool names to
|
||||
/// `^[a-zA-Z0-9_-]{1,128}$`; our registry uses dotted names
|
||||
/// (`clock.now`, `email.send`). The HTTP providers encode dots as `__`
|
||||
/// on the wire and decode on receipt — internal naming (DB step rows,
|
||||
/// scenarios, UI traces) never changes. No registry tool may contain
|
||||
/// a literal `__`.
|
||||
pub fn wire_tool_name(internal: &str) -> String {
|
||||
internal.replace('.', "__")
|
||||
}
|
||||
|
||||
pub fn internal_tool_name(wire: &str) -> String {
|
||||
wire.replace("__", ".")
|
||||
}
|
||||
|
||||
impl ContentPart {
|
||||
/// Convenience constructor for plain text parts.
|
||||
pub fn text(value: impl Into<String>) -> ContentPart {
|
||||
@@ -109,3 +123,33 @@ pub type EventStream = BoxStream<'static, Result<LlmEvent, LlmError>>;
|
||||
pub trait LlmProvider: Send + Sync {
|
||||
async fn stream(&self, request: ChatRequest) -> Result<EventStream, LlmError>;
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod wire_name_tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn dotted_registry_names_round_trip_the_wire_codec() {
|
||||
for name in [
|
||||
"clock.now",
|
||||
"email.send",
|
||||
"files.write",
|
||||
"files.list",
|
||||
"files.delete",
|
||||
"routine.schedule",
|
||||
"chat.send",
|
||||
"chat.inbox",
|
||||
"browser.goto",
|
||||
"shell.exec",
|
||||
"slack.post",
|
||||
] {
|
||||
let wire = wire_tool_name(name);
|
||||
assert!(
|
||||
wire.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-'),
|
||||
"{wire} must satisfy the Anthropic/OpenAI name pattern"
|
||||
);
|
||||
assert_eq!(internal_tool_name(&wire), name, "lossless round trip");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user