TL;DR. The noisy quadratic model separates loss into remaining initial error and error from noisy updates. Newton-like updates can remove initial error faster but also amplify noise, so batch size can change which optimizer works best. Under suitable spectral and noise assumptions, NQM gives a power-law form similar to FSL, but its noise stays fixed while FSL's noise depends on the prediction error.
Recently, Dang et al. (2026) uses a quadratic loss model to explain that best optimizers might vary with the choice of batch size (BS). This simple yet powerful model caught my attention, namely noisy quadratic model (NQM), which separates the noise and signal learning while incorporates some second-order information.
Here recall the definition.
<aside>
Definition 1. (Noisy Quadratic Model). Let $\boldsymbol\theta\in\mathbb R^d$ be the parameter vector and $\boldsymbol\theta^\star$ its optimal value. Define the error as $\boldsymbol e=\boldsymbol\theta-\boldsymbol\theta^\star$. The noisy quadratic model consists of a quadratic loss and a noisy estimate of its gradient,
$$ \mathcal L(\boldsymbol e) =\frac12\boldsymbol e^\top\boldsymbol H\boldsymbol e, \qquad \boldsymbol g(\boldsymbol e) =\underbrace{\boldsymbol H\boldsymbol e}{\text{true gradient}}+\underbrace{\frac1{\sqrt B}\boldsymbol C^{1/2}\boldsymbol\xi}{\text{noise}}. $$
Here, $\boldsymbol H\in\mathbb R^{d\times d} \succ 0$ is the Hessian. The noise $\boldsymbol\xi \in \mathbb{R}^d$ samples independent from $\mathcal N(\boldsymbol0,\boldsymbol I_d)$ and $\boldsymbol C\in\mathbb R^{d\times d} \succeq 0$ specifies the noise covariance. $B$ is the BS, so the sampled gradient has covariance $\boldsymbol C/B$.
</aside>
Where does the quadratic loss comes from? During the late phase of training, the iterate remains close to some global minimum $\boldsymbol{\theta}^\star$ and a smooth loss has local approximation.
$$ f(\boldsymbol\theta^\star+\boldsymbol e)-f(\boldsymbol\theta^\star)\approx\frac12\boldsymbol e^\top\boldsymbol H\boldsymbol e \text{, where } \boldsymbol H=\nabla^2f(\boldsymbol\theta^\star) $$
NQM treats this approximation as exact.
How do we simplify the NQM? To make the analysis simpler, Dang et al. (2026) assume that both $\boldsymbol{H}$ and $\boldsymbol{C}$ are diagonal. This lets us describe the curvature and noise one coordinate at a time.
<aside>
Definition 2. (Diagonal Noisy Quadratic Model). Assume
$$ \boldsymbol H=\operatorname{diag}(h_1,\ldots,h_d), \qquad \boldsymbol C=\operatorname{diag}(c_1,\ldots,c_d). $$
Then the loss and sampled gradient are
$$ \mathcal L(\boldsymbol e) =\frac12\sum_{i=1}^{d}h_i e_i^2, \qquad g_i=h_i e_i+\sqrt{\frac{c_i}{B}}\xi_i. $$
</aside>
We can now track each coordinate separately and add their contributions to the loss under SGD.
In this section, we analyze the evolution of loss under the diagonal NQM.
The loss dynamics under SGD. First, let us consider SGD under the simplified NQM. The update rule gives
$$ \begin{equation} \boldsymbol e_{k+1}=\boldsymbol e_k-\eta\boldsymbol g(\boldsymbol{e}_k), \end{equation} $$
where $\eta > 0$ is a constant learning rate.
We want to know how much loss remains after $n$ updates.
<aside>
Theorem 1. (Loss under SGD). Denote $v_{0, i} = \mathbb{E}[{e^{2}_{0, i}}]$ as the initial error, and $L_n = \mathbb{E}[\mathcal{L}(\boldsymbol{e}_n)]$ as the expected loss. For the diagonal NQM, plain SGD gives
\underbrace{\frac12\sum_{i=1}^{d}h_i v_{0,i}(1-\eta h_i)^{2n}}{\text{initial error left}} + \underbrace{\frac{\eta^2}{2B}\sum{i=1}^{d}h_i c_i \sum_{j=0}^{n-1}(1-\eta h_i)^{2j}}_{\text{error from noisy updates}}. \end{equation} $$
The initial error decays in every coordinate when $0<\eta< \frac{2}{\max_i h_i}$.
</aside>
Proof.
Substituting the gradient from Definition 2 into Eq. (1) gives:
$$ \begin{aligned} e_{k+1,i} &=e_{k,i}-\eta\left(h_i e_{k,i}+\sqrt{\frac{c_i}{B}}\,\xi_{k,i}\right)\\ &=(1-\eta h_i)e_{k,i} -\eta\sqrt{\frac{c_i}{B}}\,\xi_{k,i}. \end{aligned} $$
Let $a_i = (1-\eta h_i)$, taking square on both sides and take expectation:
$$ \begin{aligned} e_{k+1,i}^2 &=a_i^2e_{k,i}^2 -2a_i\eta\sqrt{\frac{c_i}{B}}\,e_{k,i}\xi_{k,i} +\frac{\eta^2c_i}{B}\xi_{k,i}^2 \\ \mathbb E[e_{k+1,i}^2] &=a_i^2 \mathbb E [e_{k,i}^2]+\frac{\eta^2c_i}{B}. \end{aligned} $$
Let $v_{k,i}=\mathbb E[e_{k,i}^2]$, we have:
$$ v_{k+1,i}=a_i^2v_{k,i}+\frac{\eta^2c_i}{B} \\ v_{n,i}=a_i^{2n}v_{0,i} +\frac{\eta^2c_i}{B}\sum_{j=0}^{n-1}a_i^{2j}. $$
The initial error passes through all $n$ updates. Noise added at an earlier update passes through more later updates, so it receives a higher power of $a_i^2$.
By Definition 2,
$$ L_n^{\mathrm{SGD}}=\frac12\sum_{i=1}^{d}h_i v_{n,i}. $$
Insert the previous expression directly gives our theorem result.
End of proof.
In Theorem 1, the first term tells us how much of the starting error is still present. The second adds up the errors caused by all noisy updates.