rustytorch/docs/archive/legacy/phase6.md

RustyTorch++ — Phase 6 Plan (Self‑Optimizing Platform & Governance)

Objective (6–12 months): Evolve RustyTorch++ from a high‑performance framework into a self‑optimizing platform with a unified data+compute graph, telemetry‑driven auto‑tuning, and end‑to‑end governance/provenance. Cement reliability, determinism, and compliance while enabling automated improvements via agents.

Reference paths

RUSTYTORCH_ROOT: /home/osobh/projects/rustytorch

RUSTG_COMPILER: /home/osobh/projects/rust/rustg (RTX 5090, sm_120)

STRATOSWARM_ROOT: /home/osobh/projects/stratoswarm

Focus is core platform evolution; app layers remain out of scope this phase.

1. Scope & Non‑Goals

In scope

Unified Data + Compute Graph: integrate Arrow/Polars ETL ops into the same IR/graph as model ops; enable cross‑stage fusion where semantics allow.

Self‑Optimizing Runtime: telemetry‑guided rewrites, auto‑tuning of kernels/launch params/parallel strategies with strict guardrails.

Governance & Provenance: cryptographically signed artifacts (code→data→model→deployment), SBOMs, dataset lineage, and reproducible builds.

Zero‑Copy IO Pathways: GPUDirect Storage/RDMA hooks; page‑pinned staging; NVMe spill for KV/activations where applicable.

Heterogeneous Execution: initial policy engine to place ops across GPU/CPU (and stage stubs for future accelerators).

Agent‑in‑the‑Loop Evolution: safe sandbox loops (propose→test→select→merge) with evolution controller + conductor + security gates.

Out of scope (Phase 7+)

Multi‑tenant billing, global routing, and enterprise policy portals (product layer)

Fully automated cross‑org model exchange marketplaces

2. Success Criteria (Exit / Phase‑Gate)

Unified Graph: ETL→Model→Post steps can be compiled as one graph; measurable end‑to‑end wall‑time reduction ≥15% on canonical pipelines.

Auto‑Tuning Wins: telemetry‑driven optimizations deliver ≥10–25% throughput improvement over Phase‑5 baselines without human intervention, with strict rollback.

Governance: every artifact (dataset snapshot, code hash, kernel bundle, checkpoint, container) is signed; repro build can reconstruct a result with the same metrics (± tolerance).

Zero‑Copy Paths: verified GPUDirect Storage/RDMA data path with ≥20% lower CPU load and ≤10% latency variance.

Hetero Execution: policy engine places at least 3 op families on optimal devices with no accuracy drift and ≤5% overhead when policies disable.

Agent Safety: evolution loops run continuously with no production regressions; all changes attributable and reversible.

3. Architecture Work

A. Unified Data + Compute Graph

Extend IR (rtx-ir) with ETL dialect: columnar ops (filter, project, join, groupby, tokenize) + cost model.

Graph rewrite passes to pushdown ETL operations and fuse around model input/output transforms.

IO nodes for Parquet/Arrow; streamable datasets; deterministic shuffles; seed propagation.

B. Self‑Optimizing Runtime

Telemetry taps: per‑op latency, bandwidth, occupancy, cache hits, queue depths; aggregate per SKU.

Optimization agents propose: kernel param updates, fusion rewrites, graph scheduling tweaks, parallel policy changes.

Safety envelope: fixed budgets, canary benches, A/B gates, automatic rollback, signed PRs only.

C. Governance & Provenance

Provenance spec: { code_sha, data_sha, schema, env, driver, rustg_sha, kernel_cache_ver, config }.

SBOM generation; cosign signatures for artifacts; SLSA level targets for CI.

Dataset lineage store; data access policies; redaction tools for traces.

D. Zero‑Copy IO

GPUDirect Storage path (where supported): NVMe→GPU DMA with staging fallbacks.

RDMA datapath for distributed shards; bounded buffers; backpressure signals.

KV/offload paging integrated with these paths; autotuned block sizes.

E. Heterogeneous Execution

Policy engine: rules based on tensor size, op type, current load; warm/cold device migration with cost modeling.

Residency annotations in IR; copy elision where legal.

F. Agent Evolution Loop

Evolution Controller orchestrates: propose→validate→select→merge.

Conductor defines SLAs; Security enforces scans and secrets hygiene; Governance signs bundles.

4. Benchmarks & Baselines

Pipelines

Text ETL (tokenization, filtering) + LLM training step

Vision ETL (decode, resize, crop, normalize) + ViT training step

Metrics

End‑to‑end wall‑time; CPU utilization; GPU util; DRAM/PCIe/NVLink GB/s; cache hit‑rates; determinism checks.

Determinism

Seeded shuffles; dataset snapshot pins; equal outputs within tolerances; graph + kernel cache hashes logged.

5. CI Gates (Phase 6)

E2E Gate: ≥15% wall‑time reduction vs Phase‑5 pipeline; regression guard ≤5%/week.

Autotune Gate: optimization proposals must beat baselines by configured thresholds; otherwise auto‑revert.

Governance Gate: SBOM/sign passes; provenance manifest attached to releases; reproducer script verified.

Zero‑Copy Gate: data path tests pass; CPU load ↓ ≥20% on supported hosts.

Safety Gate: evolution controller runs only in sandbox until green; canary then promote.

6. Observability & Telemetry

Unified trace spans across ETL+Model; correlation IDs per run.

Metrics: e2e_ms, etl_ms, model_ms, gds_enabled, rdma_enabled, cpu_user_pct, gpu_util, tokens_per_s, cache_hit_pct, occ_pct.

Dashboards: pipeline breakdowns; optimization suggestions & impact; provenance views.

7. Deliverables

Extended rtx-ir with ETL dialect + passes; pushdown & fusion rules.

Runtime hooks for zero‑copy IO (GDS/RDMA) + fallbacks; tuning knobs.

Auto‑tuning agents integrated with safe promotion; profile DB expansion.

Governance pipeline: SBOM, signatures, provenance manifests, dataset lineage store.

Benchmarks & dashboards for ETL+Model pipelines; CI gates configured.

Docs: docs/unified_graph.md, docs/governance.md, docs/zero_copy_io.md, docs/evolution.md.

8. RACI — Phase 6 Agent Ownership

Conductor (Orchestrator) — Accountable: phase delivery, dependency orchestration, gate readiness.

Data/ETL Graph Agent — Responsible: ETL dialect & pushdown; dataset sharding; deterministic transforms.

IR Rewrite & Fusion Agent — Responsible: cross‑stage fusion patterns; legality checks.

Auto‑Tuner Agent — Responsible: telemetry‑driven proposals; cache/profile DB; safe promotion.

Rust Engineer — Responsible: zero‑copy IO hooks; residency/placement plumbing; error paths.

Distributed Strategy Agent — Consulted: RDMA data paths; placement policies.

Performance Engineer — Accountable: pipeline benches; KPI dashboards; regression alerts.

Governance & Provenance Agent — Accountable: SBOM/signing; manifest/verifier; dataset lineage.

Security Engineer & Auditor — Accountable: scans; redaction; secret hygiene; policy enforcement.

Agent Organizer — Accountable: routing policies; freeze switch on gate failure.

9. Risks & Mitigations

Semantic drift when fusing ETL+Model → legality proofs; golden tests; opt‑out kill switch.

Auto‑tuner regressions → canary A/B; rollback; budget caps; sandbox isolation.

GDS/RDMA environment drift → capability detection; test matrix by driver/kernel; graceful fallbacks.

Provenance gaps → mandatory manifests; CI fails on missing lineage/signatures.

Policy misplacement → conservative defaults; cost model learning; operator allow/deny lists.

10. Timeline (suggested)

Months 1–2: ETL dialect & pushdown passes; unified tracing; baseline E2E benches.

Months 3–4: Zero‑copy IO hooks; telemetry taps; auto‑tuner integration (sandbox only).

Months 5–6: Governance pipeline; safe promotion of optimizations; CI gates; docs; phase‑gate review.

11. Phase‑7 Handover Seeds

Enterprise policy plane (RBAC, retention, PII tooling) and compliance dashboards.

Global cache distribution and profile sharing across Stratoswarm clusters.

Heterogeneous expansion (TPU/FPGA/NPUs) with placement rules & kernels.

— End of Phase 6 Plan —