Manufactured Rank Is Real but Unrealized

An optimization-barrier analysis of nonlinear low-rank adapters (e.g. sine-LoRA's ΔW = γ·sin(ωBA), loran): the rank they manufacture beyond the nominal ceiling is real, but under standard training of an adapter it goes largely unrealized, and the part that is realized carries no measurable task signal.

Three findings.

1. Optimization barrier. Producing genuine extra rank needs the fold argument |ωBA| in the nonlinear range, where the loss curvature scales as Θ(ω²). Standard first-order training (AdamW, fixed LR) in practice stays in the near-linear regime where sin(ωBA) ≈ ωBA, so little supra-nominal rank is realized.
2. Inert at convergence (adaptation). A within-checkpoint causal test, Rank-Content Ablation (RCA), finds the realized rank carries no measurable task signal: truncating a trained update back to its nominal rank is lossless (retained-gain fraction η ≈ 1.00), and the manufactured tail in isolation sits at the no-adapter floor (η ≈ 0). This holds across the full folding spectrum, replicates on a second published method (loran), and under matched per-arm tuning there is no detectable advantage over a well-tuned linear adapter.
3. Load-bearing in signal fitting. Run from scratch in the regime the construction comes from (3D occupancy), the same probe inverts: truncating the manufactured rank collapses the fit (η ≈ 0), while the plain low-rank control stays lossless. The inert-versus-load-bearing boundary tracks whether training reaches the folding regime and whether the task rewards rank, rather than the construction itself.

Paper

The preprint is on dojo7.com and in this repo as paper/main.pdf. A companion post, Manufactured Rank Is Real but Unrealised, tells the story behind it.

Method: Rank-Content Ablation (RCA)

Given a trained adapter (or a realized layer weight), materialize ΔW, take its SVD, and re-evaluate modified updates on held-out data with the trained head fixed:

• I1 truncate to nominal rank r (raw + Frobenius-rescaled)
• I1b matched-energy random-direction removal (the content-vs-energy control)
• I3 tail-only (the manufactured rank in isolation)
• I2 tail-spectrum placebo grafted onto a plain-LoRA checkpoint

η = (m(ΔW̃) − m₀) / (m(ΔW) − m₀) is the fraction of over-floor gain retained. The same probe (sinelora.rank_metrics.spectral_variant) is applied to trained adapters and, in the signal-fitting tier, to realized layer weights.

Repository layout

src/sinelora/      adapters.py (arms a1–a10), train_glue.py, train_commonsense.py,
                   inr.py (signal-fitting: weight-replacement low-rank layers + MLP),
                   analyze.py (CR2 + TOST + precision gate), rank_metrics.py, conventions.py
scripts/           intervene_probe.py (I1/I1b/I3), intervene_i2.py (I2),
                   run_all_interventions.py, run_prereg_stats.py, make_figures.py,
                   run_loran_extension.py, run_mora_control.py, train_memorize.py;
                   signal-fitting: occupancy_fit.py, inr_fit.py, signalfit_pilot.py,
                   assemble_signalfit_omega.py, run_signalfit.py, analyze_signalfit.py,
                   make_signalfit_fig.py
paper/             main.tex, references.bib, tab_prereg.tex, tmlr.sty, figs/
specs/             05_final_prereg_spec.json  (frozen pre-registration)
DEVIATIONS.md      contemporaneous deviation log (D1–D10), incl. the signal-fitting tier

Reproduce

uv sync --frozen                         # pinned env (uv.lock); torch 2.5.1+cu124

# train one GLUE adapter (DeBERTaV3-base, FP16 tier)
uv run python -m sinelora.train_glue --task cola --arm a5 --omega 100 \
  --lr 6.9e-5 --seed 13 --model microsoft/deberta-v3-base --out results/run

# RCA battery on a trained checkpoint
uv run python scripts/intervene_probe.py --run-dir results/run

# pre-registered CR2 stats -> paper/tab_prereg.tex
uv run python scripts/run_prereg_stats.py

# signal-fitting tier: 3D occupancy fit + RCA on a Stanford mesh
uv run python scripts/occupancy_fit.py --mesh data/meshes/Armadillo.ply.gz \
  --out results/occ/armadillo --omega 100 --k 1 --siren-init --arms full,lowrank,sine,loran

# figures
uv run python scripts/make_figures.py
uv run python scripts/make_signalfit_fig.py

The commands above run a single GPU job each. The cloud fleet drivers that parallelized the sweeps across RunPod instances assume a private provisioning harness and are not included in this release; they are described in DEVIATIONS.md (D7).

Tiers, pinning, and artifacts

• Primary (intervention) tier: DeBERTaV3-base on GLUE (RTE/CoLA/MRPC/STS-B), FP16.
• Confirmation tier: Llama-3.2-1B QLoRA (NF4) on commonsense, via the ungated mirror NousResearch/Llama-3.2-1B @ a9745ffc3556f145a830ac0c203509ba860582a3 (deviation D9); evaluated with lm-eval==0.4.12. RCA is not run here (quantization perturbs the spectrum).
• Signal-fitting tier: 3D binary occupancy on Stanford shapes (Thai Statue, Armadillo, Dragon), coordinate MLP, each construction trained from scratch under its faithful recipe (sine uses SIREN folding-regime init); pre-registered in DEVIATIONS.md (D10). The Stanford meshes are downloaded, not redistributed.
• Analysis is pre-registered (specs/05_final_prereg_spec.json); all deviations are logged in DEVIATIONS.md.
• results/ and data/ are git-ignored. A slim artifact (per-(arm,task,seed,ω) scores, per-checkpoint η, and spectral/rank summaries) is built by python scripts/build_artifact.py and deposited on Zenodo (10.5281/zenodo.20887092); the raw tree regenerates from the commands above.

License

Code (src/, scripts/, tests/, build/config files) is licensed under the Apache License 2.0 (see LICENSE). The paper text and figures (paper/) and the deposited data artifact are licensed under CC BY 4.0 (see LICENSE-DATA). Please cite via CITATION.cff.