test(rtx-fsi): coupling rescue rung C (CRESCUE_COARSE, burst-local s=2 coarsening) + INCTRACE per-step increment dump — both default off
CI / Test (macos-latest) (push) Canceled after 0s
CI / Test (ubuntu-latest) (push) Canceled after 0s
CI / Build CPU-Only (Explicit) (push) Canceled after 0s
Documentation / Build User Guide (push) Canceled after 0s
Performance Benchmarks / Run Benchmarks (push) Canceled after 0s
CI / Format Check (push) Canceled after 0s
CI / Clippy Check (push) Canceled after 0s
CI / Build (macos-latest) (push) Canceled after 0s
CI / Build (ubuntu-latest) (push) Canceled after 0s
CI / Python Bindings (maturin) (macos-latest) (push) Canceled after 0s
CI / Python Bindings (maturin) (ubuntu-latest) (push) Canceled after 0s
CI / WASM Build + Size Check (push) Canceled after 0s
CI / Distributed Training Tests (push) Canceled after 0s
CI / CI Success (push) Canceled after 0s
Documentation / Build API Documentation (push) Canceled after 0s

Coupling-rescue campaign, continued (omni-cortex
docs/coupling_rescue_campaign.md §11). Rung A was refuted 4/4 by
mechanism (the substeps reproduce the rejected motion — the runaway is
in the converged coupled load at the crossing). Diagnostics on the same
death: SUBCYCLE=2 marches GREEN to t=16 (zero bursts), HYST=0.25 dies
EARLIER. So:

- RTX_{prefix}_CRESCUE_COARSE=<M> (with CRESCUE=1): on a trigger,
  reject the step and take 2dt coupled steps with the fluid subcycled
  at 2x (fluid dt unchanged = the s=2 interpolated closure) for M
  coupled steps, then resume; episodes counted, cap 5 per second of
  march (loud). rescue.rs: coarse_step / attempt_with generalisation;
  march loop is now a while loop (a coarse step consumes two indices,
  the series carries a linear midpoint). VERDICT: refuted 2/2 — the
  coarse steps themselves cannot close once the state is 10x wild;
  both rungs act too late (the kinematic trigger is the limitation).
- RTX_{prefix}_INCTRACE=<csv>: reporting-only per-step dump (step, t,
  predictor increment, tol_step, passes, residual, stalled, tip jump)
  — rung A''s calibration data (healthy anchor vs death).

Verified: the FSI2 committed default digit-identical knob-off after
each change (same-day baseline); fmt + clippy clean on the touched
files.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01X2GmJXeQ2njUecEKiJZ1G2
This commit is contained in:
Omar Sobh
2026-09-02 22:09:49 -07:00
co-authored by Claude Fable 5.1
parent e76271ac67
commit 7e0f159097
4 changed files with 270 additions and 17 deletions
@@ -122,6 +122,20 @@ pub struct MarchConfig {
/// is repeated as 2, 4, 8, 16, 32 coupled substeps of dt/n — see
/// `rescue.rs` and omni-cortex `docs/coupling_rescue_campaign.md`.
pub coupling_rescue: bool,
/// Rung C (`RTX_{prefix}_CRESCUE_COARSE`, default 0 = off; needs
/// `coupling_rescue`): on a trigger, instead of the substep ladder,
/// reject the step and enter a coarse EPISODE of this many coupled
/// steps taken as 2dt steps with the fluid subcycled at 2× (the s = 2
/// interpolated closure, measured to march through the crossing
/// where s = 1 dies), then resume. Reporting carries a linear
/// midpoint for the skipped row.
pub coarse_episode: usize,
/// Per-step increment dump (`RTX_{prefix}_INCTRACE=<csv>`, off by
/// default): one line per coupled step — step, t, predictor
/// increment, tol_step, passes, final residual, stalled (0/1),
/// committed tip jump. Reporting-only (rung A's calibration data:
/// the healthy distribution of the increment vs a death's).
pub inc_trace: Option<String>,
/// C^1 interface motion (constant acceleration across the step from
/// the previous end velocity) instead of a constant velocity with a
/// jump at the step boundary. See `Fsi2Harness::advance_subcycled`.
@@ -177,6 +191,8 @@ impl MarchConfig {
.and_then(|v| v.parse::<usize>().ok())
.unwrap_or(defaults.trace_from),
coupling_rescue: num("CRESCUE", f64::from(u8::from(defaults.coupling_rescue))) != 0.0,
coarse_episode: num("CRESCUE_COARSE", defaults.coarse_episode as f64) as usize,
inc_trace: std::env::var(key("INCTRACE")).ok().or(defaults.inc_trace),
c1_interface: num("C1", f64::from(u8::from(defaults.c1_interface))) != 0.0,
predictor: std::env::var(key("PREDICTOR")).unwrap_or(defaults.predictor),
quiescent_release: num("QUIESCENT", f64::from(u8::from(defaults.quiescent_release)))
@@ -328,6 +344,8 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
trace_steps,
trace_from,
coupling_rescue,
coarse_episode,
ref inc_trace,
c1_interface,
ref predictor,
quiescent_release,
@@ -466,6 +484,19 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
let mut rescue_steps: Vec<usize> = Vec::new();
let mut jump_window: std::collections::VecDeque<f64> = std::collections::VecDeque::new();
let mut running_p95: Option<f64> = None;
// Rung C: coupled steps left in the current coarse episode, and the
// step index of every episode start (for the loud cap).
let mut coarse_remaining = 0usize;
let mut coarse_episodes: Vec<usize> = Vec::new();
let mut inc_trace_file = inc_trace.as_ref().map(|p| {
let mut w = std::io::BufWriter::new(std::fs::File::create(p).expect("inctrace path"));
writeln!(
w,
"step,t,increment,tol_step,passes,residual,stalled,tip_jump"
)
.unwrap();
w
});
let mut csv = csv_path
.as_ref()
.map(|p| std::fs::File::create(p).expect("csv path"));
@@ -487,7 +518,140 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
let mut t_save = 0.0f64;
let phase_start = std::time::Instant::now();
for step in 0..coupled_steps {
let mut step = 0usize;
// Rung C's coarse step over [step, step + 2) with its two-row series
// bookkeeping (a linear midpoint, then the end state). A macro rather
// than a closure so it can mutate the march's state beside the
// closures that borrow it. Never expands with the knobs off.
macro_rules! coarse_now {
($reason:expr, $before:expr, $tip_rejected:expr, $new_episode:expr,
$fluid_saved:expr, $field_saved:expr, $start_state:expr, $start_nodal:expr) => {{
let iv = super::rescue::Interval {
harness: &harness,
solver: &solver,
field: &field,
flag: &flag,
wetted_dofs: &wetted_dofs,
fluid_saved: $fluid_saved,
field_saved: $field_saved,
start_state: $start_state,
start_nodal: $start_nodal,
config,
dt,
};
let start_ux = $start_state.displacement[a_dofs[0]];
let start_uy = $start_state.displacement[a_dofs[1]];
match super::rescue::coarse_step(&iv, 2) {
Ok(o) => {
flag_state = o.state;
committed_nodal = o.nodal;
worst_conservation = worst_conservation.max(o.worst_conservation);
total_skipped += o.skipped;
total_subiterations += o.passes;
stalled_steps += o.stalls;
worst_stall = worst_stall.max(o.worst_residual);
if let Some(iqn_ref) = iqn.as_mut() {
iqn_ref.reset_history();
}
let t_end_c = t_release + (step + 2) as f64 * dt;
if $new_episode {
coarse_episodes.push(step);
coupling_rescues += 1;
let record = super::rescue::RescueRecord {
step,
t: t_end_c,
trigger: $reason,
before: $before,
n: 0,
passes: o.passes,
tip_rejected: $tip_rejected,
tip_rescued: flag_state.displacement[a_dofs[1]],
};
println!(
" COARSE EPISODE at step {step} t = {t_end_c:.4}: {} ({:.3e}); first 2dt \
step in {} passes ({} stalls, worst {:.3e}); tip {:+.4e} -> {:+.4e}; \
episode {} coupled steps",
$reason, $before, o.passes, o.stalls, o.worst_residual, $tip_rejected,
record.tip_rescued, coarse_episode
);
rescue_records.push(record);
let per_second = (1.0 / dt).round() as usize;
let recent = coarse_episodes
.iter()
.filter(|&&s| step - s < per_second)
.count();
assert!(
recent <= super::rescue::COARSE_EPISODE_CAP_PER_SECOND,
"{} coarse-episode cap: {recent} episodes within one second of march \
at step {step} — a runaway the coarsening only delays",
case.name
);
}
let end_ux = flag_state.displacement[a_dofs[0]];
let end_uy = flag_state.displacement[a_dofs[1]];
let (drag_now, lift_now) =
harness.measure_force(&solver.borrow(), &field.borrow());
for k in 0..2usize {
let s = step + k;
let t = t_release + (s + 1) as f64 * dt;
let (ux, uy) = if k == 0 {
(0.5 * (start_ux + end_ux), 0.5 * (start_uy + end_uy))
} else {
(end_ux, end_uy)
};
times.push(t);
ux_series.push(ux);
uy_series.push(uy);
interval_drag.push(drag_now);
interval_lift.push(lift_now);
if (s + 1) % 10 == 0 {
let drag = median(&mut interval_drag);
let lift = median(&mut interval_lift);
interval_drag.clear();
interval_lift.clear();
force_times.push(t);
drag_series.push(drag);
lift_series.push(lift);
if let Some(file) = csv.as_mut() {
writeln!(file, "{t:.6},{ux:.6e},{uy:.6e},{drag:.6e},{lift:.6e}")
.unwrap();
}
} else if let Some(file) = csv.as_mut() {
writeln!(file, "{t:.6},{ux:.6e},{uy:.6e},,").unwrap();
}
if (s + 1) % 1000 == 0 {
let window = &uy_series[uy_series.len().saturating_sub(1000)..];
let (w_mid, w_amp) = mid_amp(window);
println!(
" t = {t:.3} s ({s} steps, coarse): uy(A) = {uy:.3e} (window mid \
{w_mid:.3e} amp {w_amp:.3e}), {:.1} subit/step, {:.0} s wall",
total_subiterations as f64 / (s + 1) as f64,
phase_start.elapsed().as_secs_f64()
);
}
}
}
Err(e) => panic!(
"{} coarse step failed at step {step} ({}): {e:?} (coarse episodes so far {})",
case.name,
$reason,
coarse_episodes.len()
),
}
}};
}
while step < coupled_steps {
// Rung C: inside a coarse episode, keep taking 2dt steps.
if coarse_remaining > 0 && step + 1 < coupled_steps {
let fs = solver.borrow().snapshot();
let ff = field.borrow().clone();
let ss = flag_state.clone();
let sn = committed_nodal.clone();
coarse_now!("episode", f64::NAN, f64::NAN, false, &fs, &ff, &ss, &sn);
coarse_remaining = coarse_remaining.saturating_sub(2);
step += 2;
continue;
}
let d_n = extract(&flag_state);
let v_n: Option<Vec<f64>> = c1_interface.then(|| extract_velocity(&flag_state));
@@ -660,6 +824,24 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
)
});
let outcome_ok = outcome.is_ok();
// Rung A calibration record (reporting-only); the tip jump is
// appended after the commit.
let inc_record: Option<(usize, f64, bool)> =
inc_trace_file.as_ref().map(|_| match &outcome {
Ok(c) => (c.iterations, c.residual, false),
Err(
rtx_fsi::FsiError::CouplingNotConverged {
iterations,
residual,
..
}
| rtx_fsi::FsiError::CouplingDiverged {
iterations,
residual,
},
) => (*iterations, *residual, true),
Err(_) => (0, f64::NAN, true),
});
// The coupling-level rescue's ladder over this step's interval
// (never called with the knob off; the saved start state exists
// only with it on).
@@ -681,6 +863,9 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
};
let mut pending: Option<(&'static str, f64, super::rescue::RescueOutcome)> = None;
let mut tip_rejected = f64::NAN;
// Rung C: (trigger, its magnitude, the rejected tip) when a coarse
// episode is to start on this step.
let mut do_coarse: Option<(&'static str, f64, f64)> = None;
if let (Some(line), Err(_)) = (&trace_line, &outcome) {
// A stall's record prints here (accepted or fatal — a death
// panics below, before the commit); a converged step's
@@ -720,22 +905,25 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
| rtx_fsi::FsiError::CouplingDiverged { residual, .. } => *residual,
_ => f64::NAN,
};
match run_ladder() {
Ok(o) => pending = Some(("fatal stall", residual, o)),
Err(e2) => panic!(
"{} coupling failed at step {step}: {e:?}; the coupling-level rescue's \
substep ladder {:?} failed too: {e2:?} (coupling rescues so far {}, \
Newton rescues {:?})",
case.name,
super::rescue::LADDER,
coupling_rescues,
flag.borrow().rescue_counts()
),
if coarse_episode > 0 && step + 1 < coupled_steps {
do_coarse = Some(("fatal stall", residual, f64::NAN));
} else {
match run_ladder() {
Ok(o) => pending = Some(("fatal stall", residual, o)),
Err(e2) => panic!(
"{} coupling failed at step {step}: {e:?}; the coupling-level \
rescue's substep ladder {:?} failed too: {e2:?} (Newton rescues \
{:?})",
case.name,
super::rescue::LADDER,
flag.borrow().rescue_counts()
),
}
}
}
}
let prev_uy: Option<f64> = uy_series.last().copied();
if pending.is_none() {
if pending.is_none() && do_coarse.is_none() {
// `latest` holds the response to the accepted interface (the
// last pass) — commit it directly; the fluid, mask and flag
// are consistent with that interface without an extra pass.
@@ -752,7 +940,12 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
if let (true, Some(p95), Some(prev)) = (coupling_rescue, running_p95, prev_uy) {
let uy_new = flag_state.displacement[a_dofs[1]];
let jump = (uy_new - prev).abs();
if jump > super::rescue::TIP_JUMP_FACTOR * p95 {
if jump > super::rescue::TIP_JUMP_FACTOR * p95
&& coarse_episode > 0
&& step + 1 < coupled_steps
{
do_coarse = Some(("tip jump", jump, uy_new));
} else if jump > super::rescue::TIP_JUMP_FACTOR * p95 {
let accepted_solver = solver.borrow().snapshot();
let accepted_field = field.borrow().clone();
match run_ladder() {
@@ -782,6 +975,23 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
// A fatal stall never commits its last pass.
drop(latest.borrow_mut().take());
}
if let Some((reason, before, tip_rejected_c)) = do_coarse {
// Rung C: reject the step (the coarse step restores the saved
// start) and open an episode of 2dt steps.
coarse_now!(
reason,
before,
tip_rejected_c,
true,
&fluid_saved,
&field_saved,
step_start_state.as_ref().expect("rescue start state"),
step_start_nodal.as_deref().expect("rescue start load")
);
coarse_remaining = coarse_episode.saturating_sub(2);
step += 2;
continue;
}
let was_rescued = pending.is_some();
if let Some((trigger, before, o)) = pending {
flag_state = o.state;
@@ -855,6 +1065,16 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
let jump = uy - uy_series.last().copied().unwrap_or(uy);
println!("{line}; committed tip jump {jump:+.3e} (uy {uy:+.4e})");
}
if let (Some(w), Some((passes, residual, stalled))) = (inc_trace_file.as_mut(), inc_record)
{
let jump = uy - uy_series.last().copied().unwrap_or(uy);
writeln!(
w,
"{step},{t:.6},{increment:.6e},{tol_step:.6e},{passes},{residual:.6e},{},{jump:.6e}",
u8::from(stalled)
)
.unwrap();
}
times.push(t);
ux_series.push(ux);
uy_series.push(uy);
@@ -903,6 +1123,7 @@ pub fn run_march(case: BenchmarkCase, config: &MarchConfig) -> MarchResult {
phase_start.elapsed().as_secs_f64()
);
}
step += 1;
}
let coupled_elapsed = phase_start.elapsed().as_secs_f64();