test(rtx-fsi): coupling-level rescue rung A (RTX_FSI{2,3}_CRESCUE, default off) + TRACE_FROM autopsy window
CI / Build CPU-Only (Explicit) (push) Canceled after 0s
Documentation / Build User Guide (push) Canceled after 0s
Performance Benchmarks / Run Benchmarks (push) Canceled after 0s
CI / Format Check (push) Canceled after 0s
CI / Clippy Check (push) Canceled after 0s
CI / Build (macos-latest) (push) Canceled after 0s
CI / Test (macos-latest) (push) Canceled after 0s
CI / Build (ubuntu-latest) (push) Canceled after 0s
CI / Test (ubuntu-latest) (push) Canceled after 0s
Documentation / Build API Documentation (push) Canceled after 0s
CI / Python Bindings (maturin) (macos-latest) (push) Canceled after 0s
CI / Python Bindings (maturin) (ubuntu-latest) (push) Canceled after 0s
CI / WASM Build + Size Check (push) Canceled after 0s
CI / Distributed Training Tests (push) Canceled after 0s
CI / CI Success (push) Canceled after 0s
CI / Build CPU-Only (Explicit) (push) Canceled after 0s
Documentation / Build User Guide (push) Canceled after 0s
Performance Benchmarks / Run Benchmarks (push) Canceled after 0s
CI / Format Check (push) Canceled after 0s
CI / Clippy Check (push) Canceled after 0s
CI / Build (macos-latest) (push) Canceled after 0s
CI / Test (macos-latest) (push) Canceled after 0s
CI / Build (ubuntu-latest) (push) Canceled after 0s
CI / Test (ubuntu-latest) (push) Canceled after 0s
Documentation / Build API Documentation (push) Canceled after 0s
CI / Python Bindings (maturin) (macos-latest) (push) Canceled after 0s
CI / Python Bindings (maturin) (ubuntu-latest) (push) Canceled after 0s
CI / WASM Build + Size Check (push) Canceled after 0s
CI / Distributed Training Tests (push) Canceled after 0s
CI / CI Success (push) Canceled after 0s
Coupling-rescue campaign (omni-cortex docs/coupling_rescue_campaign.md).
Every recorded FSI2 s=1 coupling death is the END of a multi-step
runaway (74-march burst scan: green marches never exceed 2.4x their own
p95 tip jump; 27/28 deaths burst 5-37 steps first). The traced autopsy
of the (1.27, 2.0) death (replay digit-identical, CSV byte-identical)
shows a growing period-2 instability of the CONVERGED coupled scheme at
the tip's max-velocity crossing: increment x33 and converged load
1e4 -> 1e6 N over 200 steps with the coupling converging on 111 of the
first 112 steps.
- march.rs: RTX_{prefix}_TRACE_FROM (print-only autopsy window: per-pass
residual/load + one line per step with increment, tol_step, retry_at,
acceptable, outcome, committed tip jump); RTX_{prefix}_CRESCUE (default
off): on a fatal stall or a committed tip jump > 3x the running p95
(trailing 2000 non-rescued steps), reject the step and repeat the
interval as 2/4/8/16/32 coupled substeps of dt/n (Mayr-Wall-Gee
reduced-step repetition, five repetitions); per-rescue record printed,
MarchResult.coupling_rescues/_failures/rescue_records, rate cap 20
rescues per second of march (dies loudly).
- rescue.rs (new): the substep ladder — each substep a complete coupled
step at dt/n with its own predictor, fresh coupler, C1 velocity
chaining, tolerances at the substep's increment; the march's own step
path is NOT routed through it (digit identity by construction).
- rtx-fea NonlinearDynamicStepper::step_with_dt (step at an explicit dt;
step() delegates float-for-float); Fsi2Harness::advance_subcycled_with
(explicit fluid dt; advance_subcycled delegates).
Verified: FSI2 + FSI3 committed defaults and the noise probe
digit-identical knob-off vs same-day / 2026-08-31 baselines; knob ON
on the anchor u=1.00 r=1: green, zero rescues, CSV byte-identical to
TWIN-1's u1.00.csv; fmt + clippy clean on the touched files.
Verdict of rung A on the provocation set: REFUTED 3/3 by mechanism —
every rescued interval was carried but reproduced the rejected step's
motion (dt/2..dt/32 give the same jump), so the runaway lives in the
coupled LOAD at the crossing, not in the time integration; the
SUBCYCLE=2 closure marches through the same crossing. The knob stays,
default off, as the instrument that measured this.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01X2GmJXeQ2njUecEKiJZ1G2
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
0f578087ce
commit
e76271ac67
@@ -101,6 +101,27 @@ pub struct MarchConfig {
|
||||
/// Print every pass's interface residual for the first `trace_steps`
|
||||
/// coupled steps (diagnostics; 0 = off).
|
||||
pub trace_steps: usize,
|
||||
/// Trace window for a death autopsy (`RTX_{prefix}_TRACE_FROM`,
|
||||
/// default `usize::MAX` = off): from this coupled step onward,
|
||||
/// print every pass like `trace_steps` does PLUS one line per step
|
||||
/// with the predictor increment, the step tolerance, the two
|
||||
/// acceptance thresholds, the coupling outcome and the committed
|
||||
/// tip jump. Reporting-only — the knob touches no float on the
|
||||
/// solver path, so a traced replay is digit-identical to the
|
||||
/// untraced march (the property every autopsy so far has leaned
|
||||
/// on). Motivation: every recorded coupling death is the END of a
|
||||
/// multi-step runaway (committed tip jumps 10–70× the march's own
|
||||
/// p95 for 5–37 steps before the panic; `scripts/fsi_burst_scan.py`
|
||||
/// in omni-cortex), and `trace_steps` only sees the release.
|
||||
pub trace_from: usize,
|
||||
/// Coupling-level rescue (`RTX_{prefix}_CRESCUE`, default off =
|
||||
/// bit-identical): a step whose coupling ends above its acceptance
|
||||
/// window (today's death), or whose committed tip jump exceeds
|
||||
/// 3× the march's running p95 of |Δuy| (the runaway signature
|
||||
/// every recorded death carries for 5–37 steps before its panic),
|
||||
/// is repeated as 2, 4, 8, 16, 32 coupled substeps of dt/n — see
|
||||
/// `rescue.rs` and omni-cortex `docs/coupling_rescue_campaign.md`.
|
||||
pub coupling_rescue: bool,
|
||||
/// C^1 interface motion (constant acceleration across the step from
|
||||
/// the previous end velocity) instead of a constant velocity with a
|
||||
/// jump at the step boundary. See `Fsi2Harness::advance_subcycled`.
|
||||
@@ -151,6 +172,11 @@ impl MarchConfig {
|
||||
ffld_dir: std::env::var(key("FFLD")).ok().or(defaults.ffld_dir),
|
||||
initial_relaxation: num("OMEGA0", defaults.initial_relaxation),
|
||||
trace_steps: num("TRACE", defaults.trace_steps as f64) as usize,
|
||||
trace_from: std::env::var(key("TRACE_FROM"))
|
||||
.ok()
|
||||
.and_then(|v| v.parse::<usize>().ok())
|
||||
.unwrap_or(defaults.trace_from),
|
||||
coupling_rescue: num("CRESCUE", f64::from(u8::from(defaults.coupling_rescue))) != 0.0,
|
||||
c1_interface: num("C1", f64::from(u8::from(defaults.c1_interface))) != 0.0,
|
||||
predictor: std::env::var(key("PREDICTOR")).unwrap_or(defaults.predictor),
|
||||
quiescent_release: num("QUIESCENT", f64::from(u8::from(defaults.quiescent_release)))
|
||||
@@ -211,6 +237,13 @@ pub struct MarchResult {
|
||||
/// defeated the plain SVK Newton, the failure that used to kill the
|
||||
/// march outright.
|
||||
pub newton_rescues: (usize, usize),
|
||||
/// Coupling-level rescues (steps repeated as substeps — see
|
||||
/// `rescue.rs`); zero with the knob off and on every healthy march.
|
||||
pub coupling_rescues: usize,
|
||||
/// Kinematic-trigger rescues whose whole ladder failed (the step
|
||||
/// was kept as the coupling accepted it).
|
||||
pub coupling_rescue_failures: usize,
|
||||
pub rescue_records: Vec<super::rescue::RescueRecord>,
|
||||
pub final_state_finite: bool,
|
||||
pub elapsed: f64,
|
||||
}
|
||||
@@ -293,6 +326,8 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
|
||||
ref ffld_dir,
|
||||
initial_relaxation,
|
||||
trace_steps,
|
||||
trace_from,
|
||||
coupling_rescue,
|
||||
c1_interface,
|
||||
ref predictor,
|
||||
quiescent_release,
|
||||
@@ -424,6 +459,13 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
|
||||
// trajectory.
|
||||
let mut interval_drag: Vec<f64> = Vec::new();
|
||||
let mut interval_lift: Vec<f64> = Vec::new();
|
||||
// Coupling-level rescue bookkeeping (all inert with the knob off).
|
||||
let mut coupling_rescues = 0usize;
|
||||
let mut coupling_rescue_failures = 0usize;
|
||||
let mut rescue_records: Vec<super::rescue::RescueRecord> = Vec::new();
|
||||
let mut rescue_steps: Vec<usize> = Vec::new();
|
||||
let mut jump_window: std::collections::VecDeque<f64> = std::collections::VecDeque::new();
|
||||
let mut running_p95: Option<f64> = None;
|
||||
let mut csv = csv_path
|
||||
.as_ref()
|
||||
.map(|p| std::fs::File::create(p).expect("csv path"));
|
||||
@@ -470,6 +512,9 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
|
||||
let fluid_saved = solver.borrow().snapshot();
|
||||
let field_saved = field.borrow().clone();
|
||||
t_save += save_start.elapsed().as_secs_f64();
|
||||
// The rescue repeats the interval from the committed start state.
|
||||
let step_start_state = coupling_rescue.then(|| flag_state.clone());
|
||||
let step_start_nodal = coupling_rescue.then(|| committed_nodal.clone());
|
||||
type PassResult = (
|
||||
FlowField,
|
||||
DynamicState,
|
||||
@@ -502,7 +547,7 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
|
||||
let (nodal, conservation, skipped) =
|
||||
harness.sample_load(&solver_ref, &trial_field, d_candidate);
|
||||
t_sample.set(t_sample.get() + sample_start.elapsed().as_secs_f64());
|
||||
if step < trace_steps {
|
||||
if step < trace_steps || step >= trace_from {
|
||||
let load: f64 = nodal.iter().map(|(_, f)| f.norm()).sum();
|
||||
let peak = nodal.iter().map(|(_, f)| f.norm()).fold(0.0, f64::max);
|
||||
println!(
|
||||
@@ -517,7 +562,7 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
|
||||
let (candidate_state, _) = flag_ref.step(&flag_state).unwrap();
|
||||
t_structure.set(t_structure.get() + structure_start.elapsed().as_secs_f64());
|
||||
let d_new = extract(&candidate_state);
|
||||
if step < trace_steps {
|
||||
if step < trace_steps || step >= trace_from {
|
||||
let residual: f64 = d_new
|
||||
.iter()
|
||||
.zip(d_candidate)
|
||||
@@ -586,6 +631,62 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
|
||||
outcome = iqn_ref.solve(&d_predicted, pass);
|
||||
}
|
||||
}
|
||||
let trace_line = (step >= trace_from).then(|| {
|
||||
let verdict = match &outcome {
|
||||
Ok(c) => format!("converged in {} (residual {:.3e})", c.iterations, c.residual),
|
||||
Err(rtx_fsi::FsiError::CouplingNotConverged {
|
||||
iterations,
|
||||
residual,
|
||||
..
|
||||
}) => format!(
|
||||
"NOT converged after {iterations} (residual {residual:.3e}) -> {}",
|
||||
if *residual < acceptable { "accepted stall" } else { "DEATH" }
|
||||
),
|
||||
Err(rtx_fsi::FsiError::CouplingDiverged {
|
||||
iterations,
|
||||
residual,
|
||||
}) => format!(
|
||||
"DIVERGED after {iterations} (residual {residual:.3e}) -> {}",
|
||||
if *residual < acceptable { "accepted stall" } else { "DEATH" }
|
||||
),
|
||||
Err(e) => format!("error {e:?}"),
|
||||
};
|
||||
format!(
|
||||
" TRACE step {step} t = {:.5}: increment {increment:.3e}, tol_step {tol_step:.3e}, \
|
||||
retry_at {retry_at:.3e}, acceptable {acceptable:.3e}, retries so far {retried_steps}, \
|
||||
stalls so far {stalled_steps}, Newton rescues {:?}; {verdict}",
|
||||
t_release + (step + 1) as f64 * dt,
|
||||
flag.borrow().rescue_counts(),
|
||||
)
|
||||
});
|
||||
let outcome_ok = outcome.is_ok();
|
||||
// The coupling-level rescue's ladder over this step's interval
|
||||
// (never called with the knob off; the saved start state exists
|
||||
// only with it on).
|
||||
let run_ladder = || -> Result<super::rescue::RescueOutcome, rtx_fsi::FsiError> {
|
||||
let iv = super::rescue::Interval {
|
||||
harness: &harness,
|
||||
solver: &solver,
|
||||
field: &field,
|
||||
flag: &flag,
|
||||
wetted_dofs: &wetted_dofs,
|
||||
fluid_saved: &fluid_saved,
|
||||
field_saved: &field_saved,
|
||||
start_state: step_start_state.as_ref().expect("rescue start state"),
|
||||
start_nodal: step_start_nodal.as_deref().expect("rescue start load"),
|
||||
config,
|
||||
dt,
|
||||
};
|
||||
super::rescue::substep_interval(&iv)
|
||||
};
|
||||
let mut pending: Option<(&'static str, f64, super::rescue::RescueOutcome)> = None;
|
||||
let mut tip_rejected = f64::NAN;
|
||||
if let (Some(line), Err(_)) = (&trace_line, &outcome) {
|
||||
// A stall's record prints here (accepted or fatal — a death
|
||||
// panics below, before the commit); a converged step's
|
||||
// record prints after the commit with its tip jump.
|
||||
println!("{line}");
|
||||
}
|
||||
match outcome {
|
||||
Ok(converged) => {
|
||||
total_subiterations += converged.iterations;
|
||||
@@ -607,26 +708,153 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
|
||||
total_subiterations += iterations;
|
||||
max_subiterations = max_subiterations.max(iterations);
|
||||
}
|
||||
Err(e) => panic!(
|
||||
"{} coupling failed at step {step}: {e:?} (Newton rescues so far: {:?})",
|
||||
case.name,
|
||||
flag.borrow().rescue_counts()
|
||||
),
|
||||
Err(e) => {
|
||||
assert!(
|
||||
coupling_rescue,
|
||||
"{} coupling failed at step {step}: {e:?} (Newton rescues so far: {:?})",
|
||||
case.name,
|
||||
flag.borrow().rescue_counts()
|
||||
);
|
||||
let residual = match &e {
|
||||
rtx_fsi::FsiError::CouplingNotConverged { residual, .. }
|
||||
| rtx_fsi::FsiError::CouplingDiverged { residual, .. } => *residual,
|
||||
_ => f64::NAN,
|
||||
};
|
||||
match run_ladder() {
|
||||
Ok(o) => pending = Some(("fatal stall", residual, o)),
|
||||
Err(e2) => panic!(
|
||||
"{} coupling failed at step {step}: {e:?}; the coupling-level rescue's \
|
||||
substep ladder {:?} failed too: {e2:?} (coupling rescues so far {}, \
|
||||
Newton rescues {:?})",
|
||||
case.name,
|
||||
super::rescue::LADDER,
|
||||
coupling_rescues,
|
||||
flag.borrow().rescue_counts()
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
let prev_uy: Option<f64> = uy_series.last().copied();
|
||||
if pending.is_none() {
|
||||
// `latest` holds the response to the accepted interface (the
|
||||
// last pass) — commit it directly; the fluid, mask and flag
|
||||
// are consistent with that interface without an extra pass.
|
||||
let (new_field, new_flag_state, nodal, conservation, skipped) =
|
||||
latest.borrow_mut().take().expect("pass ran");
|
||||
*field.borrow_mut() = new_field;
|
||||
flag_state = new_flag_state;
|
||||
committed_nodal = nodal;
|
||||
worst_conservation = worst_conservation.max(conservation);
|
||||
total_skipped += skipped;
|
||||
|
||||
// Kinematic trigger: the coupling accepted this step, but the
|
||||
// committed tip moved beyond the march's own running statistic.
|
||||
if let (true, Some(p95), Some(prev)) = (coupling_rescue, running_p95, prev_uy) {
|
||||
let uy_new = flag_state.displacement[a_dofs[1]];
|
||||
let jump = (uy_new - prev).abs();
|
||||
if jump > super::rescue::TIP_JUMP_FACTOR * p95 {
|
||||
let accepted_solver = solver.borrow().snapshot();
|
||||
let accepted_field = field.borrow().clone();
|
||||
match run_ladder() {
|
||||
Ok(o) => {
|
||||
tip_rejected = uy_new;
|
||||
pending = Some(("tip jump", jump, o));
|
||||
}
|
||||
Err(e2) => {
|
||||
// Keep the step the coupling accepted; the
|
||||
// ladder left the fluid mid-failure.
|
||||
solver.borrow_mut().restore(&accepted_solver);
|
||||
*field.borrow_mut() = accepted_field;
|
||||
coupling_rescue_failures += 1;
|
||||
println!(
|
||||
" COUPLING RESCUE FAILED at step {step} t = {:.4}: tip jump \
|
||||
{jump:.3e} > {} x running p95 {p95:.3e}; ladder {:?} ended \
|
||||
{e2:?}; keeping the accepted step",
|
||||
t_release + (step + 1) as f64 * dt,
|
||||
super::rescue::TIP_JUMP_FACTOR,
|
||||
super::rescue::LADDER
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// A fatal stall never commits its last pass.
|
||||
drop(latest.borrow_mut().take());
|
||||
}
|
||||
let was_rescued = pending.is_some();
|
||||
if let Some((trigger, before, o)) = pending {
|
||||
flag_state = o.state;
|
||||
committed_nodal = o.nodal;
|
||||
worst_conservation = worst_conservation.max(o.worst_conservation);
|
||||
total_skipped += o.skipped;
|
||||
total_subiterations += o.passes;
|
||||
stalled_steps += o.stalls;
|
||||
worst_stall = worst_stall.max(o.worst_residual);
|
||||
// The main coupler's secant columns predate the rejected step.
|
||||
if let Some(iqn_ref) = iqn.as_mut() {
|
||||
iqn_ref.reset_history();
|
||||
}
|
||||
coupling_rescues += 1;
|
||||
rescue_steps.push(step);
|
||||
let record = super::rescue::RescueRecord {
|
||||
step,
|
||||
t: t_release + (step + 1) as f64 * dt,
|
||||
trigger,
|
||||
before,
|
||||
n: o.n,
|
||||
passes: o.passes,
|
||||
tip_rejected,
|
||||
tip_rescued: flag_state.displacement[a_dofs[1]],
|
||||
};
|
||||
println!(
|
||||
" COUPLING RESCUE at step {step} t = {:.4}: {trigger} ({before:.3e}) carried by \
|
||||
n = {} substeps in {} passes ({} substep stalls, worst {:.3e}); tip {:+.4e} -> \
|
||||
{:+.4e}",
|
||||
record.t,
|
||||
o.n,
|
||||
o.passes,
|
||||
o.stalls,
|
||||
o.worst_residual,
|
||||
tip_rejected,
|
||||
record.tip_rescued
|
||||
);
|
||||
rescue_records.push(record);
|
||||
let per_second = (1.0 / dt).round() as usize;
|
||||
let recent = rescue_steps
|
||||
.iter()
|
||||
.filter(|&&s| step - s < per_second)
|
||||
.count();
|
||||
assert!(
|
||||
recent <= super::rescue::RATE_CAP_PER_SECOND,
|
||||
"{} coupling rescue rate cap: {recent} rescues within one second of march at \
|
||||
step {step} — a runaway the substeps only delay is not to be hidden",
|
||||
case.name
|
||||
);
|
||||
}
|
||||
// `latest` holds the response to the accepted interface (the last
|
||||
// pass) — commit it directly; the fluid, mask and flag are
|
||||
// consistent with that interface without an extra pass.
|
||||
let (new_field, new_flag_state, nodal, conservation, skipped) =
|
||||
latest.borrow_mut().take().expect("pass ran");
|
||||
*field.borrow_mut() = new_field;
|
||||
flag_state = new_flag_state;
|
||||
committed_nodal = nodal;
|
||||
worst_conservation = worst_conservation.max(conservation);
|
||||
total_skipped += skipped;
|
||||
|
||||
let t = t_release + (step + 1) as f64 * dt;
|
||||
let ux = flag_state.displacement[a_dofs[0]];
|
||||
let uy = flag_state.displacement[a_dofs[1]];
|
||||
// Running p95 of the committed tip jump over the trailing window
|
||||
// of non-rescued steps (the kinematic trigger's yardstick).
|
||||
if coupling_rescue && !was_rescued {
|
||||
if let Some(prev) = prev_uy {
|
||||
jump_window.push_back((uy - prev).abs());
|
||||
if jump_window.len() > super::rescue::TIP_JUMP_WINDOW {
|
||||
jump_window.pop_front();
|
||||
}
|
||||
if jump_window.len() == super::rescue::TIP_JUMP_WINDOW {
|
||||
let mut sorted: Vec<f64> = jump_window.iter().copied().collect();
|
||||
sorted.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
running_p95 = Some(sorted[(0.95 * sorted.len() as f64) as usize]);
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Some(line) = trace_line.as_ref().filter(|_| outcome_ok) {
|
||||
let jump = uy - uy_series.last().copied().unwrap_or(uy);
|
||||
println!("{line}; committed tip jump {jump:+.3e} (uy {uy:+.4e})");
|
||||
}
|
||||
times.push(t);
|
||||
ux_series.push(ux);
|
||||
uy_series.push(uy);
|
||||
@@ -712,6 +940,9 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
|
||||
skipped: total_skipped,
|
||||
spiked: harness.spiked_total.get(),
|
||||
newton_rescues: flag.borrow().rescue_counts(),
|
||||
coupling_rescues,
|
||||
coupling_rescue_failures,
|
||||
rescue_records,
|
||||
final_state_finite: flag_state.displacement.iter().all(|v| v.is_finite()),
|
||||
elapsed: start.elapsed().as_secs_f64(),
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user