test(rtx-fsi): coupling-level rescue rung A (RTX_FSI{2,3}_CRESCUE, default off) + TRACE_FROM autopsy window
CI / Build CPU-Only (Explicit) (push) Canceled after 0s
Documentation / Build User Guide (push) Canceled after 0s
Performance Benchmarks / Run Benchmarks (push) Canceled after 0s
CI / Format Check (push) Canceled after 0s
CI / Clippy Check (push) Canceled after 0s
CI / Build (macos-latest) (push) Canceled after 0s
CI / Test (macos-latest) (push) Canceled after 0s
CI / Build (ubuntu-latest) (push) Canceled after 0s
CI / Test (ubuntu-latest) (push) Canceled after 0s
Documentation / Build API Documentation (push) Canceled after 0s
CI / Python Bindings (maturin) (macos-latest) (push) Canceled after 0s
CI / Python Bindings (maturin) (ubuntu-latest) (push) Canceled after 0s
CI / WASM Build + Size Check (push) Canceled after 0s
CI / Distributed Training Tests (push) Canceled after 0s
CI / CI Success (push) Canceled after 0s

Coupling-rescue campaign (omni-cortex docs/coupling_rescue_campaign.md).
Every recorded FSI2 s=1 coupling death is the END of a multi-step
runaway (74-march burst scan: green marches never exceed 2.4x their own
p95 tip jump; 27/28 deaths burst 5-37 steps first). The traced autopsy
of the (1.27, 2.0) death (replay digit-identical, CSV byte-identical)
shows a growing period-2 instability of the CONVERGED coupled scheme at
the tip's max-velocity crossing: increment x33 and converged load
1e4 -> 1e6 N over 200 steps with the coupling converging on 111 of the
first 112 steps.

- march.rs: RTX_{prefix}_TRACE_FROM (print-only autopsy window: per-pass
  residual/load + one line per step with increment, tol_step, retry_at,
  acceptable, outcome, committed tip jump); RTX_{prefix}_CRESCUE (default
  off): on a fatal stall or a committed tip jump > 3x the running p95
  (trailing 2000 non-rescued steps), reject the step and repeat the
  interval as 2/4/8/16/32 coupled substeps of dt/n (Mayr-Wall-Gee
  reduced-step repetition, five repetitions); per-rescue record printed,
  MarchResult.coupling_rescues/_failures/rescue_records, rate cap 20
  rescues per second of march (dies loudly).
- rescue.rs (new): the substep ladder — each substep a complete coupled
  step at dt/n with its own predictor, fresh coupler, C1 velocity
  chaining, tolerances at the substep's increment; the march's own step
  path is NOT routed through it (digit identity by construction).
- rtx-fea NonlinearDynamicStepper::step_with_dt (step at an explicit dt;
  step() delegates float-for-float); Fsi2Harness::advance_subcycled_with
  (explicit fluid dt; advance_subcycled delegates).

Verified: FSI2 + FSI3 committed defaults and the noise probe
digit-identical knob-off vs same-day / 2026-08-31 baselines; knob ON
on the anchor u=1.00 r=1: green, zero rescues, CSV byte-identical to
TWIN-1's u1.00.csv; fmt + clippy clean on the touched files.

Verdict of rung A on the provocation set: REFUTED 3/3 by mechanism —
every rescued interval was carried but reproduced the rejected step's
motion (dt/2..dt/32 give the same jump), so the runaway lives in the
coupled LOAD at the crossing, not in the time integration; the
SUBCYCLE=2 closure marches through the same crossing. The knob stays,
default off, as the instrument that measured this.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01X2GmJXeQ2njUecEKiJZ1G2
This commit is contained in:
Omar Sobh
2026-09-02 21:36:06 -07:00
co-authored by Claude Fable 5.1
parent 0f578087ce
commit e76271ac67
6 changed files with 584 additions and 22 deletions
@@ -0,0 +1,277 @@
//! Coupling-level rescue — rung A of omni-cortex
//! `docs/coupling_rescue_campaign.md`.
//!
//! A coupled step the march would otherwise abandon (a fatal stall:
//! the coupling ended above its acceptance window) or reject (a
//! committed tip jump beyond the march's own running statistic — the
//! runaway signature every recorded death shows for 537 steps before
//! its panic) is REPEATED over the same interval as n = 2, 4, 8, 16,
//! 32 coupled substeps of dt/n: Mayr, Wall and Gee's reduced-step
//! repetition (κ = 0.5, at most five repetitions) applied to the
//! partitioned loop, one layer above the structural Newton rescue.
//!
//! Each substep is a complete coupled step at dt/n: its own predictor
//! from the substep's start state, the fluid subcycled at
//! dt_fluid/n from the fluid state the previous substep committed, the
//! flag stepped at dt/n (`NonlinearDynamicStepper::step_with_dt`, whose
//! Newton and rescue ladder already take dt as a parameter), the C¹
//! interface velocity chained through the substep's own start
//! velocity, and a FRESH coupler (the secant history of the other dt
//! is not this map's). Tolerances follow the march's own rule at the
//! substep's own increment. A substep that ends above its acceptance
//! window fails the rung; the next rung restores the start-of-step
//! fluid snapshot and tries twice as many substeps.
//!
//! **Default OFF.** With the knob off nothing here runs — the march's
//! own step logic is not routed through this module, on purpose: the
//! path the digit-identity protocol certifies stays float-for-float
//! untouched, and this duplicate of it only exists past a trigger.
use std::cell::{Cell, RefCell};
use nalgebra::Vector3;
use rtx_cfd::solvers::incompressible::{EmbeddedPisoSolver, EmbeddedSolverState, FlowField};
use rtx_fea::analysis::{DynamicState, NonlinearDynamicStepper};
use rtx_fea::mesh::NodeId;
use rtx_fsi::{FsiError, IqnIls, Subiterated};
use super::Fsi2Harness;
use super::march::MarchConfig;
/// The substep ladder: five repetitions at κ = 0.5.
pub const LADDER: [usize; 5] = [2, 4, 8, 16, 32];
/// Kinematic trigger: a committed tip jump above this multiple of the
/// march's running p95 of |Δuy| rejects the step. Calibrated on all 74
/// recorded s = 1 marches (`scripts/fsi_burst_scan.py`): every march
/// that reached its horizon stays ≤ 2.4×; 27 of 28 deaths exceed 5×.
pub const TIP_JUMP_FACTOR: f64 = 3.0;
/// The running statistic's window (committed, non-rescued steps); the
/// kinematic trigger is inert until the window is full.
pub const TIP_JUMP_WINDOW: usize = 2000;
/// More rescues than this within one second of march is a runaway the
/// substeps only delay — the march dies loudly instead of hiding it.
pub const RATE_CAP_PER_SECOND: usize = 20;
/// One rescue, as recorded in the march result and printed as it fires.
#[derive(Debug, Clone)]
pub struct RescueRecord {
pub step: usize,
pub t: f64,
/// `"fatal stall"` or `"tip jump"`.
pub trigger: &'static str,
/// The rejected step's residual (fatal stall) or |Δuy| (tip jump).
pub before: f64,
/// Substeps that carried the interval; 0 = every rung failed (the
/// step was then kept as the coupling accepted it, or the march
/// died — see the trigger).
pub n: usize,
/// Coupling passes spent across all rungs tried.
pub passes: usize,
/// Tip uy the rejected step had committed (NaN for a fatal stall,
/// which never commits) and the tip the rescue committed.
pub tip_rejected: f64,
pub tip_rescued: f64,
}
/// What a successful ladder rung hands back to the march.
pub struct RescueOutcome {
pub state: DynamicState,
pub nodal: Vec<(NodeId, Vector3<f64>)>,
pub n: usize,
pub passes: usize,
pub stalls: usize,
pub worst_residual: f64,
pub worst_conservation: f64,
pub skipped: usize,
}
/// Everything one interval repetition needs, borrowed from the march.
pub struct Interval<'a, 'b> {
pub harness: &'a Fsi2Harness,
pub solver: &'a RefCell<EmbeddedPisoSolver>,
pub field: &'a RefCell<FlowField>,
pub flag: &'a RefCell<NonlinearDynamicStepper<'b>>,
pub wetted_dofs: &'a [[usize; 2]],
/// The start-of-step fluid snapshot and field the march already
/// saves for its coupling passes.
pub fluid_saved: &'a EmbeddedSolverState,
pub field_saved: &'a FlowField,
pub start_state: &'a DynamicState,
pub start_nodal: &'a [(NodeId, Vector3<f64>)],
pub config: &'a MarchConfig,
/// The full coupled step the interval spans.
pub dt: f64,
}
fn extract(state: &DynamicState, wetted: &[[usize; 2]]) -> Vec<f64> {
let mut d = vec![0.0; 2 * wetted.len()];
for (k, dofs) in wetted.iter().enumerate() {
d[2 * k] = state.displacement[dofs[0]];
d[2 * k + 1] = state.displacement[dofs[1]];
}
d
}
fn extract_velocity(state: &DynamicState, wetted: &[[usize; 2]]) -> Vec<f64> {
let mut v = vec![0.0; 2 * wetted.len()];
for (k, dofs) in wetted.iter().enumerate() {
v[2 * k] = state.velocity[dofs[0]];
v[2 * k + 1] = state.velocity[dofs[1]];
}
v
}
/// Repeat the interval up the ladder; the first rung whose every
/// substep converges or stalls inside its acceptance window wins.
///
/// # Errors
/// The last rung's coupling error when all five fail. The fluid and
/// field are then left as the last failed rung left them — the caller
/// restores or dies.
pub fn substep_interval(iv: &Interval<'_, '_>) -> Result<RescueOutcome, FsiError> {
let mut last_err = None;
let mut passes_so_far = 0usize;
for &n in &LADDER {
match attempt(iv, n) {
Ok(mut outcome) => {
outcome.passes += passes_so_far;
return Ok(outcome);
}
Err((e, passes)) => {
passes_so_far += passes;
last_err = Some(e);
}
}
}
Err(last_err.expect("ladder is non-empty"))
}
/// One rung: `n` coupled substeps of `dt/n` from the saved start.
fn attempt(iv: &Interval<'_, '_>, n: usize) -> Result<RescueOutcome, (FsiError, usize)> {
let cfg = iv.config;
let dt_sub = iv.dt / n as f64;
let dt_fluid_sub = dt_sub / cfg.subcycle as f64;
iv.solver.borrow_mut().restore(iv.fluid_saved);
*iv.field.borrow_mut() = iv.field_saved.clone();
let mut state = iv.start_state.clone();
let mut nodal: Vec<(NodeId, Vector3<f64>)> = iv.start_nodal.to_vec();
let mut passes = 0usize;
let mut stalls = 0usize;
let mut worst_residual = 0.0f64;
let mut worst_conservation = 0.0f64;
let mut skipped = 0usize;
for _ in 0..n {
let d_n = extract(&state, iv.wetted_dofs);
let v_n: Option<Vec<f64>> = cfg
.c1_interface
.then(|| extract_velocity(&state, iv.wetted_dofs));
let d_predicted: Vec<f64> = if cfg.predictor == "kinematic" {
let v = extract_velocity(&state, iv.wetted_dofs);
d_n.iter().zip(&v).map(|(d, v)| d + dt_sub * v).collect()
} else {
let mut flag = iv.flag.borrow_mut();
flag.set_nodal_forces(&nodal);
let (predicted, _) = flag
.step_with_dt(&state, dt_sub)
.expect("structure-alone predictor at the substep dt");
extract(&predicted, iv.wetted_dofs)
};
let sub_saved = iv.solver.borrow().snapshot();
let sub_field = iv.field.borrow().clone();
type PassResult = (
FlowField,
DynamicState,
Vec<(NodeId, Vector3<f64>)>,
f64,
usize,
);
let latest: RefCell<Option<PassResult>> = RefCell::new(None);
let pass_count = Cell::new(0usize);
let state_ref = &state;
let pass = |d_candidate: &[f64]| -> Vec<f64> {
pass_count.set(pass_count.get() + 1);
let mut solver_ref = iv.solver.borrow_mut();
solver_ref.restore(&sub_saved);
let mut trial = sub_field.clone();
iv.harness.advance_subcycled_with(
&mut solver_ref,
&mut trial,
&d_n,
d_candidate,
cfg.subcycle,
v_n.as_deref(),
dt_fluid_sub,
);
let (nodal_c, conservation, skipped_c) =
iv.harness.sample_load(&solver_ref, &trial, d_candidate);
let mut flag = iv.flag.borrow_mut();
flag.set_nodal_forces(&nodal_c);
let (candidate, _) = flag
.step_with_dt(state_ref, dt_sub)
.expect("flag step at the substep dt");
let d_new = extract(&candidate, iv.wetted_dofs);
*latest.borrow_mut() = Some((trial, candidate, nodal_c, conservation, skipped_c));
d_new
};
let increment: f64 = d_predicted
.iter()
.zip(&d_n)
.map(|(a, b)| (a - b) * (a - b))
.sum::<f64>()
.sqrt();
let tol_step = cfg.tol_floor.max(cfg.rtol * increment);
let acceptable = (cfg.stall_accept * tol_step).max(0.1 * increment);
let outcome = if cfg.coupler == "iqn" {
let mut iqn = IqnIls::new(cfg.max_subiterations, tol_step)
.map_err(|e| (e, passes))?
.with_reuse(cfg.reuse)
.with_initial_relaxation(cfg.initial_relaxation)
.map_err(|e| (e, passes))?;
iqn.solve(&d_predicted, pass)
} else {
Subiterated::aitken(cfg.max_subiterations, tol_step)
.map_err(|e| (e, passes))?
.solve(&d_predicted, pass)
};
passes += pass_count.get();
match outcome {
Ok(c) => worst_residual = worst_residual.max(c.residual),
Err(
FsiError::CouplingNotConverged { residual, .. }
| FsiError::CouplingDiverged { residual, .. },
) if residual < acceptable => {
stalls += 1;
worst_residual = worst_residual.max(residual);
}
Err(e) => return Err((e, passes)),
}
let (new_field, new_state, new_nodal, conservation, skipped_c) =
latest.borrow_mut().take().expect("pass ran");
// As in the march: the solver's state after the last pass IS
// the response to the accepted interface; commit it directly.
*iv.field.borrow_mut() = new_field;
state = new_state;
nodal = new_nodal;
worst_conservation = worst_conservation.max(conservation);
skipped += skipped_c;
}
Ok(RescueOutcome {
state,
nodal,
n,
passes,
stalls,
worst_residual,
worst_conservation,
skipped,
})
}