osobhandClaude Opus 4.8
fc895a2de5
CI / Build CPU-Only (Explicit) (pull_request) Has been cancelled
Performance Benchmarks / Run Benchmarks (pull_request) Has been cancelled
CI / Format Check (pull_request) Has been cancelled
CI / Clippy Check (pull_request) Has been cancelled
CI / Build (macos-latest) (pull_request) Has been cancelled
CI / Build (ubuntu-latest) (pull_request) Has been cancelled
CI / Test (macos-latest) (pull_request) Has been cancelled
CI / Test (ubuntu-latest) (pull_request) Has been cancelled
CI / CI Success (pull_request) Has been cancelled
Documentation / Build API Documentation (pull_request) Has been cancelled
Documentation / Build User Guide (pull_request) Has been cancelled
The decorator autograd (`Autodiff<B>`) had never been gradient-checked against a real tensor backend — the entire test suite runs on a shape-only `MockBackend` whose ops return their input, so they validate graph structure but never gradient values. Running it through `CpuBackend` for the first time (new `tests/tape_cpu_gradcheck.rs`, finite-difference checks) surfaced three bugs that made the tape unusable for training; this fixes all three. 1. Double-free / UB in the dimension-erasure cast. The backward ops cast a tensor to its runtime const-generic dimension via `mem::transmute_copy::<_, TensorPrimitive<N>>(&src)` in ~100 sites. That bit-copies the owned `Vec` without forgetting the source, so two values own one buffer → double-free on any heap-backed backend (and Stacked-Borrows UB from the typed pun). Replaced every site with a single `into_dim` helper that is now **fully safe** — it round-trips through `to_data`/`from_data` and rebuilds the shape with `array::from_fn`, no `unsafe` at all. (This is why the whole repo previously bypassed the tape with analytic backward.) 2. Fan-out gradients were silently dropped. `accumulate_gradients` was a stub that returned one path and discarded the other, and `AutodiffTensor::clone` minted a fresh `TensorId`. Together, reusing a tensor (residuals, `mul(s, s)`, shared Q/K/V — universal in transformers) split its gradient across two ids and summed neither, yielding a fraction of the true value. `accumulate_gradients` now sums via `B::add`; `clone` preserves the id so fan-out paths collide on one sink. 3. Softmax backward panicked. `SoftmaxBackward` / `stable_softmax_backward` subtracted a keep-dim row-sum from the full-shape grad, but the elementwise backends assert equal shapes (no broadcasting). Added `broadcast_along_dim` to tile the row-sum to full width first. Verified: `tape_cpu_gradcheck` (matmul, fan-out add·mul, softmax) passes with rel-err < 2e-2 vs central differences; full `rtx-autograd` suite green (263 passed, 0 failed); lib clippy `-D warnings` clean. Known follow-up (out of scope): `cargo miri test` still aborts on a Stacked-Borrows / integer-to-pointer violation inside `rtx-backend-cpu`'s buffer internals — a grad-free `from_data`+`add`+`sum` probe reproduces the identical error, so it is pre-existing backend UB, not an autograd issue. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
56 lines
1.2 KiB
TOML
56 lines
1.2 KiB
TOML
[package]
|
|
name = "rtx-autograd"
|
|
version = "1.0.0"
|
|
edition.workspace = true
|
|
rust-version = "1.92"
|
|
authors.workspace = true
|
|
license.workspace = true
|
|
repository.workspace = true
|
|
description = "Automatic differentiation with zero-overhead inference via Autodiff<B> decorator pattern"
|
|
|
|
[dependencies]
|
|
# Backend abstraction (for decorator-pattern autodiff)
|
|
rtx-backend = { path = "../rtx-backend" }
|
|
|
|
# Tensor operations dependency
|
|
rtx-tensor = { path = "../rtx-tensor" }
|
|
|
|
# Core utilities
|
|
thiserror.workspace = true
|
|
tracing.workspace = true
|
|
|
|
# Numeric computing
|
|
ndarray = "0.15"
|
|
|
|
# Collections for graph operations
|
|
indexmap = "2.0"
|
|
once_cell = "1.19"
|
|
parking_lot.workspace = true
|
|
|
|
[dev-dependencies]
|
|
# Testing framework
|
|
proptest.workspace = true
|
|
criterion.workspace = true
|
|
|
|
# Additional testing utilities
|
|
approx = "0.5"
|
|
rand = "0.8"
|
|
|
|
# Real CPU backend for numerical gradient-checking the tape (no cycle:
|
|
# rtx-backend-cpu depends only on rtx-backend / rtx-tensor, not rtx-autograd).
|
|
rtx-backend-cpu = { path = "../rtx-backend-cpu" }
|
|
|
|
[features]
|
|
default = []
|
|
disabled_tests = []
|
|
|
|
[lib]
|
|
name = "rtx_autograd"
|
|
path = "src/lib.rs"
|
|
|
|
[[bench]]
|
|
name = "gradient_benchmarks"
|
|
harness = false
|
|
[lints]
|
|
workspace = true
|