NEURIPS 2026 / EFFICIENT LLM FINE-TUNING

ENGINE.

Endogenous Variational MultiScale Optimization for Zeroth-Order LLM Fine-Tuning

Low-dimensional sampling.
Full-space curvature.

A multiscale approach that brings full-space curvature information into efficient, low-rank updates.

Read the paper Explore the code

ONE MODEL. TWO SCALES.01 / 02

Zhuoli Ouyang^1,2,†Changxi Chi^1,3Siyuan Li^1,3Tailin Wu^1,✉

¹ Westlake University² Southern University of Science and Technology³ Zhejiang University

† Work done as an intern at Westlake University · ✉ Correspondence: wutailin@westlake.edu.cn

FORWARD QUERIES2.61×

maximum speed-up over LOREN

PEAK MEMORY+0%

additional peak memory over LOREN

EVALUATION16

model–task settings across four LLMs

Maximum query speed-up on LLaMA-3-8B / CB, measured at LOREN’s best accuracy. Memory comparison follows the paper’s reported configurations.

01 / THE IDEA

Think across scales.

Can we keep the efficiency of low-dimensional sampling while still using curvature information from the full parameter space?

Zeroth-order optimization fine-tunes language models using only forward passes. Full-space sampling is informative but costly in variance. Low-rank sampling is efficient, but its preconditioned update can miss curvature effects from the full space.

ENGINE connects these two scales through Variational MultiScale theory from computational mechanics. An endogenous V-cycle alternates low-rank and full-space optimization, while a subgrid correction brings fine-scale curvature feedback into the coarse update.

02 / THE METHOD

Small updates.
A wider perspective.

The curvature operator evolves with the model.
ENGINE refreshes the connection between scales as training progresses.

COARSE PHASE

Spend queries in a smaller space.

Represent the update with low-rank factors B and A. Sample in r(m + n) coordinates instead of mn full-space coordinates, and maintain a compact curvature model for each branch.

ΔW = BA · d_c = r(m + n)

The dimension-dependent variance bound decreases under comparable gradient and preconditioning scales.

ENGINE overview showing the curvature gap, alternating coarse and fine phases, and the evolving curvature operator — FIGURE 01The two directions compared by the correction are ℛ(Σ_fG) and Σ_cℛ(G), both in the coarse space.Click to enlarge ↗

03 / THE PRINCIPLE

Reduce variance.
Correct bias.

Two complementary mechanisms improve the update.

Low-dimensional sampling controls variance.

The coarse parameter dimension is d_c = r(m + n), compared with d_f = mn. Under comparable gradient and preconditioning scales, this reduces the dimension-dependent variance bound.

MSE_ideal = V_c ≤ V_c + ‖τ_sgs‖²

Subgrid feedback corrects curvature mismatch.

In the local-linear limit, exact deterministic correction removes the structural bias relative to the restricted fine-space target, while preserving coarse sampling variance. The paper also analyzes approximate corrections and establishes a stationarity bound under stated assumptions.

04 / THE EVIDENCE

Four models.
Consistent gains.

Best reported average score among the evaluated ZO methods on each model.
Explore the main results below.

GLUE / FP32

Five-seed means. CoLA: MCC × 100. Other tasks: accuracy (%).ENGINE · Bold marks the best displayed score, including ties.

Validation accuracy curves for GPT-2-XL on MNLI, QNLI, and SST-2, comparing ENGINE with six zeroth-order baselines — FIGURE 02Validation accuracy on GPT-2-XL across three GLUE tasks.Click to enlarge ↗

Reach the same target sooner.

Up to 2.61× forward-query speed-up and 2.58× wall-clock speed-up over LOREN on LLaMA-3-8B / CB, at LOREN’s best accuracy.

Forward-query and wall-clock speed-ups over LOREN in four representative settings, alongside validation accuracy on LLaMA-3-8B CB — FIGURE 03Efficiency at LOREN’s best accuracy and a representative convergence curve.Click to enlarge ↗

05 / BUILD ON THIS WORK

A multiscale view
of optimization.

If ENGINE is useful for your research,
please cite our paper.

Official implementation

BIBTEX

@misc{ouyang2026engineendogenousvariationalmultiscale,
      title={ENGINE: Endogenous Variational MultiScale Optimization for Zeroth-Order LLM Fine-Tuning},
      author={Zhuoli Ouyang and Changxi Chi and Siyuan Li and Tailin Wu},
      year={2026},
      eprint={2611.03293},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2611.03293},
}