Nadam

Author: Vinamr Jain (vj89) (ChemE 6800 Fall 2024)

Stewards: Nathan Preuss, Wei-Han Chen, Tianqi Xiao, Guoqing Hu

Introduction

Nesterov-accelerated Adaptive Moment Estimation (Nadam) is an optimization algorithm introduced by Timothy Dozat in his 2016 paper, “Incorporating Nesterov Momentum into Adam” . Nadam enhances the Adaptive Moment Estimation (Adam) optimizer by integrating Nesterov momentum, aiming to improve convergence speed and performance in training deep learning models. This algorithm has been applied across various domains, including computer vision and natural language processing, to optimize neural network training. Gradient descent optimization algorithms are fundamental tools in machine learning, often employed as black-box optimizers to minimize loss functions by iteratively adjusting model parameters. Gradient Descent updates parameters by following the negative gradient of the loss function to seek a minimum, but can be slow to converge, especially in regions with flat or elongated curvatures . Several variants address these challenges. Momentum accelerates convergence by accumulating a decaying sum of past gradients, helping navigate ravines and reducing oscillations . Nesterov’s Accelerated Gradient (NAG), also known as Nesterov momentum, improves upon Momentum by calculating the gradient at a projected future position, often resulting in faster convergence. RMSprop adapts learning rates for each parameter individually and normalizes gradients by a decaying average of recent magnitudes, effectively handling non-stationary objectives . Adaptive Moment Estimation (Adam) combines the benefits of Momentum and adaptive learning rates, computing individual adaptive rates based on first and second moments of gradients. However, Adam may sometimes converge to suboptimal solutions due to its adaptive learning rate mechanism and can slow down before reaching global optima . Nadam extends Adam by incorporating Nesterov momentum, evaluating the gradient at a projected future parameter position and improving convergence speed and accuracy.

Background

Gradient Descent is a first-order iterative optimization algorithm used to find a local minimum of a differentiable function. The parameter update rule at iteration $t$ is: $\theta _{t+1}=\theta _{t}-\eta \nabla _{\theta }J\left(\theta _{t}\right)$ Where $\theta _{t}$ denotes the parameters vector at iteration $t$, $\eta$ is the learning rate, and $\nabla _{\theta }J\left(\theta _{t}\right)$ is the gradient of the cost function $J$ with respect to $\theta$ at iteration $t$. Momentum accelerates Gradient Descent by accumulating an exponentially decaying moving average of past gradients. This accelerates gradient descent along stable gradient directions and slows it in oscillating directions .

$m_{t}=\gamma m_{t-1}+\eta \nabla _{\theta }J\left(\theta _{t}\right)$

$\theta _{t+1}=\theta _{t}-m_{t}$

Here in addition to the gradient component, there’s an additional term $m_{t}$, the momentum vector, and $\gamma$ is the momentum coefficient. Nesterov Momentum improves upon standard Momentum by calculating the gradient at the anticipated future position, leading to a more responsive update.

$m_{t}=\gamma m_{t-1}+\eta \nabla _{\theta }J\left(\theta _{t}-\gamma m_{t-1}\right)$

$\theta _{t+1}=\theta _{t}-m_{t}$

Alternatively, one can apply the current look-ahead momentum vector directly to update the current parameters:

$m_{t}=\gamma m_{t-1}+\eta \nabla _{\theta }J\left(\theta _{t}\right)$

$\theta _{t+1}=\theta _{t}-{(\gamma m}t\ +\ \eta {\ \nabla }\theta J\left(\theta _{t}\right)\ )$

AdaGrad (Adaptive Gradient Algorithm) adapts the learning rate for each parameter individually by scaling it inversely proportional to the square root of the sum of all historical squared values of the gradients. This method is particularly useful for dealing with sparse data .

$v_{t}=v_{t-1}+\left[\nabla _{\theta }J\left(\theta _{t}\right)\right]^{2}$

$\theta _{t+1}=\theta _{t}-\eta {\frac {\nabla _{\theta }J\left(\theta _{t}\right)}{{\sqrt {v_{t}}}+\epsilon }}$

$v_{t}$ is the accumulated sum of squared gradients up to time $t$, and $\epsilon$ is a small constant for numerical stability (e.g., $10^{-8}$). AdaGrad effectively scales down the learning rate for parameters associated with frequently occurring features, allowing for larger updates for infrequent features .

RMSprop (Root Mean Square Propagation) is an optimization algorithm that builds upon AdaGrad by modifying the way it accumulates squared gradients. While AdaGrad accumulates all past squared gradients, leading to a continually decreasing learning rate, RMSprop introduces a decay parameter to the accumulated squared gradients $v_{t}$. This parameterization prevents the learning rate from diminishing too quickly.

$v_{t}=\gamma v_{t-1}+\left(1-\gamma \right)\left[\nabla _{\theta }J\left(\theta _{t}\right)\right]^{2}$

$\theta _{t+1}=\theta _{t}-\eta {\frac {\nabla _{\theta }J\left(\theta _{t}\right)}{{\sqrt {v_{t}}}+\epsilon }}$

$\gamma$ is the decay rate (typically $\gamma =0.9$). By introducing the decay parameter in the $v_{t}$ term, RMSprop limits the influence of older gradients, ensuring that the algorithm adapts quickly to the most recent gradients.

Adam (Adaptive Moment Estimation) is an optimization algorithm that combines ideas from both Momentum and RMSprop. It extends RMSprop by not only adapting the learning rates per parameter but also incorporating momentum by using an exponentially decaying average of past gradients (first moment). Adam is designed to work well with sparse gradients and noisy problems, which are common in training deep neural networks.

$m_{t}=\beta _{1}m_{t-1}+\left(1-\beta _{1}\right)\nabla _{\theta }J\left(\theta _{t}\right)$

$v_{t}=\beta _{2}v_{t-1}+\left(1-\beta _{2}\right)\left[\nabla _{\theta }J\left(\theta _{t}\right)\right]^{2}$

$m_{t}$ is the first moment estimate (exponentially decaying average of past gradients), $v_{t}$ is the second moment estimate (exponentially decaying average of past squared gradients), and $\beta _{1}$ and $\beta _{2}$ are the exponential decay rates for the moment estimates (commonly $\beta _{1}=0.9,\beta _{2}=0.999$).

As $m_{t}$ and $v_{t}$ are initialized at zero vectors, they are biased towards zero, especially during the initial time steps and particularly when the decay rates are close to one (i.e., $\beta _{1},\beta _{2}\approx 1$). This bias can slow down the convergence of the algorithm because the moment estimates underestimate the true mean and variance of the gradients. To counteract these biases, the authors of Adam introduced bias-corrected moment estimates:

${\widehat {m_{t}}}={\frac {m_{t}}{1-\beta _{1}^{t}}}$

${\widehat {v_{t}}}={\frac {v_{t}}{1-\beta _{2}^{t}}}$

The denominators $1-\beta _{1}^{t}$ and $1-\beta _{2}^{t}$ approach 1 as $t$ increases, reducing the impact of the bias over time. The bias correction compensates for the initialization at zero, ensuring that the moment estimates are unbiased. The parameters are then updated using the bias-corrected moment estimates:

$\theta _{t+1}=\theta _{t}-\eta {\frac {\widehat {m_{t}}}{{\sqrt {\widehat {v_{t}}}}+\epsilon }}$

Algorithm

Nadam modifies the parameter update step of Adam by integrating the Nesterov accelerated gradient. In Adam, the update relies solely on the bias-corrected first moment estimate ${\widehat {m_{t}}}$. Nadam modifies the parameter update step of Adam by integrating the Nesterov accelerated gradient. In Adam, the update relies solely on the bias-corrected first moment estimate ${\widehat {m_{t}}}$. Nadam, however, adjusts this estimate by adding a term that accounts for the current gradient, effectively looking ahead to where the parameters are heading. This adjustment is similar to the Nesterov momentum in NAG, which computes the gradient at the approximate future position of the parameters. Firstly, consider the bias-corrected first moment ${\widehat {m_{t}}}$ using the definition of $m_{t}$

${\widehat {m_{t}}}={\frac {m_{t}}{1-\beta _{1}^{t}}}={\frac {\beta _{1}m_{t-1}+\left(1-\beta _{1}\right)g_{t}}{1-\beta _{1}^{t}}}$

Where $g_{t}=\nabla _{\theta }J\left(\theta _{t}\right)$. The update rule of Adam can be rewritten as

$\theta _{t+1}=\theta _{t}-{\frac {\eta }{\left({\sqrt {\widehat {v_{t}}}}+\epsilon \right)}}\left[{\frac {\beta _{1}m_{t-1}}{\left(1-\beta _{1}^{t}\right)}}+{\frac {\left(1-\beta _{1}\right)g_{t}}{\left(1-\beta _{1}^{t}\right)}}\right]$

Recognizing that ${\frac {\beta _{1}m_{t-1}}{1-\beta _{1}^{t}}}$ is the bias-corrected estimate of the momentum vector from the previous time step, let’s denote:

${\widehat {m_{t-1}}}={\frac {m_{t-1}}{1-\beta _{1}^{t-1}}}$

However, since $\beta _{1}^{t-1}$ is close to $\beta _{1}^{t}$ for large $t$, it can be approximated as:

${\frac {\beta _{1}m_{t-1}}{1-\beta _{1}^{t}}}\approx \beta _{1}{\widehat {m_{t-1}}}$ Thus, the update rule becomes:

$\theta _{t+1}=\theta _{t}-\eta {\frac {\beta _{1}{\widehat {m_{t-1}}}+{\frac {\left(1-\beta _{1}\right)g_{t}}{1-\beta _{1}^{t}}}}{{\sqrt {\widehat {v_{t}}}}+\epsilon }}$

This form resembles the momentum term in Nesterov Accelerated Gradient (NAG). To incorporate Nesterov momentum into Adam, the previous bias-corrected momentum estimate ${\widehat {m_{t-1}}}$ is replaced with the current estimate ${\widehat {m_{t}}}$. This aligns with the NAG approach of using the look-ahead momentum vector for updating parameters.

The Nadam update rule becomes:

$\theta _{t+1}=\theta _{t}-\eta {\frac {\beta _{1}{\widehat {m_{t}}}+{\frac {\left(1-\beta _{1}\right)g_{t}}{1-\beta _{1}^{t}}}}{{\sqrt {\widehat {v_{t}}}}+\epsilon }}$

By combining the bias-corrected first moment estimate ${\widehat {m_{t}}}$ and the current gradient $g_{t}$, weighted appropriately, Nadam aims to improve the responsiveness of the optimizer to recent gradient information, potentially enhancing convergence speed.

Pseudocode

The pseudocode for Nadam is as follows:

Algorithm Nadam Optimization Input: Initial parameters $\theta _{0}$, learning rate $\eta$, decay rates $\beta _{1}$, $\beta _{2}$, small constant $\epsilon$

Initialize $m_{0}\gets 0,v_{0}\gets 0,t\gets 0$

While not converged do

   $t\ \gets t\ +\ 1$

   Compute gradient: $g_{t}=\nabla _{\theta }J\left(\theta _{t-1}\right)$

   Update biased first moment: $m_{t}\gets \beta _{1}\cdot m_{t-1}+\left(1-\beta _{1}\right)\cdot g_{t}$

   Update biased second moment: $v_{t}\gets \beta _{2}\cdot v_{t-1}+\left(1-\beta _{2}\right)\cdot g_{t}^{2}$

   Bias correction: ${\widehat {m_{t}}}\gets {\frac {m_{t}}{1-\beta _{1}^{t}}},{\widehat {v_{t}}}\gets {\frac {v_{t}}{1-\beta _{2}^{t}}}$

   Nesterov adjustment: $m'_{t}\gets \beta _{1}\cdot {\widehat {m_{t}}}+{\frac {\left(1-\beta _{1}\right)\cdot g_{t}}{1-\beta _{1}^{t}}}$

   Update parameters: $\theta _{t}\gets \theta _{t-1}-\eta \cdot {\frac {m'_{t}}{{\sqrt {\widehat {v_{t}}}}+\epsilon }}$

Output: Optimized parameters $\theta _{t}$

Numerical Examples

This section provides numerical examples to illustrate how the Nadam algorithm operates. We choose a simple quadratic function to make the calculations straightforward and easy to follow, and then we demonstrate the numerical calculations for a linear regression task.

Minimizing a Quadratic Function

Consider the cost function: $J\left(\theta \right)={\frac {1}{2}}\theta ^{2}$which has its global minimum at $\theta =0$. The aim is to minimize this function using the Nadam optimizer, starting from an initial parameter value of $\theta _{0}=5$.

The hyperparameters are set as follows: learning rate $\eta =0.1$, exponential decay rates $\beta _{1},\beta _{2}=0.9$, numerical stability constant $\epsilon ={10}^{-8}$, and initialize the first and second moment estimates and the time step as $m_{0}=0,v_{0}=0,t=0$.

Iteration 1

1. Time Step Update:

$t\ \gets t\ +\ 1\ =\ 1$

2. Compute Gradient at Current Parameter:

$g_{1}=\nabla _{\theta }J\left(\theta _{0}\right)=\theta _{0}=5$

3. Update Biased First Moment Estimate:

$m_{1}=\beta _{1}m_{0}+\left(1-\beta _{1}\right)g_{1}=0.9\times 0+0.1\times 5=0.5$

4. Update Biased Second Moment Estimate:

$v_{1}=\beta _{2}v_{0}+\left(1-\beta _{2}\right)g_{1}^{2}=0.999\times 0+0.001\times 25=0.025$

5. Compute Bias-Corrected Estimates:

${\widehat {m_{1}}}={\frac {m_{1}}{1-\beta _{1}^{t}}}={\frac {0.5}{1-{0.9}^{1}}}={\frac {0.5}{0.1}}=5$

${\widehat {v_{1}}}={\frac {v_{1}}{1-\beta _{2}^{t}}}={\frac {0.025}{1-{0.999}^{1}}}={\frac {0.025}{0.001}}=25$

6. Nesterov Adjustment for First Moment:

$m\prime _{1}=\beta _{1}{\widehat {m_{1}}}+{\frac {\left(1-\beta _{1}\right)g_{1}}{1-\beta _{1}^{t}}}=0.9\times 5+{\frac {0.1\times 5}{0.1}}=4.5+5=9.5$

7. Parameter Update:

$\theta _{1}=\theta _{0}-\eta {\frac {m\prime _{1}}{{\sqrt {\widehat {v_{1}}}}+\epsilon }}=5-0.1\times {\frac {9.5}{{\sqrt {25}}+{10}^{-8}}}=5-0.1\times {\frac {9.5}{5}}=5-0.1\times 1.9=5-0.19=4.81$

Iteration 2

1. Time Step Update:

$t\ \gets t\ +\ 1\ =\ 2$

2. Compute Gradient at Current Parameter:

$g_{2}=\nabla _{\theta }J\left(\theta _{1}\right)=\theta _{1}=4.81$

3. Update Biased First Moment Estimate:

$m_{2}=\beta _{1}m_{1}+\left(1-\beta _{1}\right)g_{2}=0.9\times 0.5+0.1\times 4.81=0.45+0.481=0.931$

4. Update Biased Second Moment Estimate:

$v_{2}=\beta _{2}v_{1}+\left(1-\beta _{2}\right)g_{2}^{2}=0.999\times 0.025+0.001\times \left(4.81\right)^{2}=0.024975+0.001\times 23.1361=0.024975+0.0231361=0.0481111$

5. Compute Bias-Corrected Estimates:

${\widehat {m_{2}}}={\frac {m_{2}}{1-\beta _{1}^{t}}}={\frac {0.931}{1-{0.9}^{2}}}={\frac {0.931}{1-0.81}}={\frac {0.931}{0.19}}\approx 4.9005$

${\widehat {v_{2}}}={\frac {v_{2}}{1-\beta _{2}^{t}}}={\frac {0.0481111}{1-{0.999}^{2}}}={\frac {0.0481111}{1-0.998001}}={\frac {0.0481111}{0.001999}}\approx 24.0696$

6. Nesterov Adjustment for First Moment:

$m\prime _{2}=\beta _{1}{\widehat {m_{2}}}+{\frac {\left(1-\beta _{1}\right)g_{2}}{1-\beta _{1}^{t}}}=0.9\times 4.9005+{\frac {0.1\times 4.81}{0.19}}=4.41045+{\frac {0.481}{0.19}}\approx 4.41045+2.5316=6.9421$

7. Parameter Update:

$\theta _{2}=\theta _{1}-\eta {\frac {m\prime _{2}}{{\sqrt {\widehat {v_{2}}}}+\epsilon }}=4.81-0.1\times {\frac {6.9421}{{\sqrt {24.0696}}+{10}^{-8}}}=4.81-0.1\times {\frac {6.9421}{4.9061}}=4.81-0.1\times 1.4142=4.81-0.14142=4.6686$

Iteration 3

1. Time Step Update:

$t\ \gets t\ +\ 1\ =\ 3$

2. Compute Gradient at Current Parameter:

$g_{3}=\nabla _{\theta }J\left(\theta _{2}\right)=\theta _{2}=4.6686$

3. Update Biased First Moment Estimate:

$m_{3}=\beta _{1}m_{2}+\left(1-\beta _{1}\right)g_{3}=0.9\times 0.931+0.1\times 4.6686=0.8379+0.46686=1.30476$

4. Update Biased Second Moment Estimate:

$v_{3}=\beta _{2}v_{2}+\left(1-\beta _{2}\right)g_{3}^{2}=0.999\times 0.0481111+0.001\times \left(4.6686\right)^{2}=0.048063+0.001\times 21.7899=0.048063+0.0217899=0.0698529$

5. Compute Bias-Corrected Estimates:

${\widehat {m_{3}}}={\frac {m_{3}}{1-\beta _{1}^{t}}}={\frac {1.30476}{1-{0.9}^{3}}}={\frac {1.30476}{1-0.729}}={\frac {1.30476}{0.271}}\approx 4.8134$

${\widehat {v_{3}}}={\frac {v_{3}}{1-\beta _{2}^{t}}}={\frac {0.0698529}{1-{0.999}^{3}}}={\frac {0.0698529}{1-0.997002999}}={\frac {0.0698529}{0.002997001}}\approx 23.3132$

6. Nesterov Adjustment for First Moment:

$m\prime _{3}=\beta _{1}{\widehat {m_{3}}}+{\frac {\left(1-\beta _{1}\right)g_{3}}{1-\beta _{1}^{t}}}=0.9\times 4.8134+{\frac {0.1\times 4.6686}{0.271}}=4.3321+{\frac {0.46686}{0.271}}\approx 4.3321+1.7227=6.0548$

7. Parameter Update:

$\theta _{3}=\theta _{2}-\eta {\frac {m\prime _{3}}{{\sqrt {\widehat {v_{3}}}}+\epsilon }}=4.6686-0.1\times {\frac {6.0548}{{\sqrt {23.3132}}+{10}^{-8}}}=4.6686-0.1\times {\frac {6.0548}{4.8284}}=4.6686-0.1\times 1.2547=4.6686-0.12547=4.5431$

After three iterations, the parameter values have been updated as follows:

$\theta _{1}=4.81$

$\theta _{2}=4.6686$

$\theta _{3}=4.5431$

The parameter $\theta$ is gradually moving towards the global minimum at $\theta =0$, demonstrating how the Nadam optimizer adjusts the parameters efficiently.

Linear Regression Task

A simple linear regression task is demonstrated to predict house prices based on size. The hypothesis function, cost function, and optimization problem are defined, and numerical calculations are provided along with plots to visualize Nadam on the dataset.

Dataset

Consider the following dataset representing the size (in square feet) and price (in thousands of dollars) of several houses:

Size (sq ft)	50	60	70	80	90	100	110	120	130	140	150
Price ($K)	150	180	200	240	300	330	360	400	450	500	550

The aim is to model the relationship between the size of a house and its price using a linear hypothesis function: $f_{\theta }\left(x\right)=\theta _{0}+\theta _{1}x$

The cost function measures the discrepancy between the predicted prices and the actual prices. Using the Mean Squared Error (MSE) as the cost function:

$J\left(\theta \right)={\frac {1}{2}}\sum _{i}^{n}\left(f_{\theta }\left(x_{i}\right)-y_{i}\right)^{2}$

The goal is to find the optimal parameters $\theta =\left(\theta _{0},\theta _{1}\right)$ that minimize the cost function:

$\mathrm {argmin} _{\mathrm {\theta } }{\frac {1}{n}}\sum _{i}^{n}\left(f_{\theta }\left(x_{i}\right)-y_{i}\right)^{2}$

To minimize the cost function using gradient-based optimization, compute the partial derivatives with respect to $\theta _{0}$ and $\theta _{1}$:

${\frac {\partial J\left(\theta \right)}{\partial \theta _{0}}}=\sum _{i=1}^{n}\left(f_{\theta }\left(x_{i}\right)-y_{i}\right)$

${\frac {\partial J\left(\theta \right)}{\partial \theta _{1}}}=\sum _{i=1}^{n}{\left(f_{\theta }\left(x_{i}\right)-y_{i}\right)x_{i}}$

Numerical Example

Considering only the first few data points to simplify the calculations:

Size (sq ft)	50	60	70	80
Price ($K)	150	180	200	240

Initial Conditions

Assume $\theta _{0}=0,\theta _{1}=0,\eta =0.01,\beta _{1}=0.9,\beta _{2}=0.999,\epsilon ={10}^{-8}$. Initialize $m_{0}^{\left(\theta _{0}\right)}=0,m_{0}^{\left(\theta _{1}\right)}=0,v_{0}^{\left(\theta _{0}\right)}=0,v_{0}^{\left(\theta _{1}\right)}=0,t=0$.

Iteration 1

1. Increment time step:

$t\leftarrow t+1=1$

2. Compute predictions and residuals:

$f_{\theta }(x_{i})=\theta _{0}+\theta _{1}x_{i}=0$

Residuals:

$f_{\theta }(x_{i})-y_{i}=(-150),(-180),(-200),(-240)$

3. Compute gradients:

${\frac {\partial J}{\partial \theta _{0}}}=\sum _{i=1}^{4}(f_{\theta }(x_{i})-y_{i})=(-150)+(-180)+(-200)+(-240)=-770$

${\frac {\partial J}{\partial \theta _{1}}}=\sum _{i=1}^{4}(f_{\theta }(x_{i})-y_{i})x_{i}=(-150\cdot 50)+(-180\cdot 60)+(-200\cdot 70)+(-240\cdot 80)$

Compute each product:

$-150\cdot 50=-7500,;-180\cdot 60=-10800,;-200\cdot 70=-14000,;-240\cdot 80=-19200$

Summation: $-7500-10800-14000-19200=-51500$

Thus, ${\frac {\partial J}{\partial \theta _{1}}}=-51500$

4. Update biased first moments:

$m_{1}^{(\theta _{0})}=\beta _{1}m_{0}^{(\theta _{0})}+(1-\beta _{1}){\frac {\partial J}{\partial \theta _{0}}}=0.9\cdot 0+0.1\cdot (-770)=-77$

$m_{1}^{(\theta _{1})}=\beta _{1}m_{0}^{(\theta _{1})}+(1-\beta _{1}){\frac {\partial J}{\partial \theta _{1}}}=0.9\cdot 0+0.1\cdot (-51500)=-5150$

5. Update biased second moments:

$v_{1}^{(\theta _{0})}=\beta _{2}v_{0}^{(\theta _{0})}+(1-\beta _{2})(-770)^{2}=0.999\cdot 0+0.001\cdot 592900=592.9$

$v_{1}^{(\theta _{1})}=\beta _{2}v_{0}^{(\theta _{1})}+(1-\beta _{2})(-51500)^{2}=0.999\cdot 0+0.001\cdot 2652250000=2,652,250$

6. Compute bias-corrected moments:

${\hat {m}}_{1}^{(\theta _{0})}={\frac {m_{1}^{(\theta _{0})}}{1-\beta _{1}^{1}}}={\frac {-77}{1-0.9}}={\frac {-77}{0.1}}=-770$

${\hat {m}}_{1}^{(\theta _{1})}={\frac {m_{1}^{(\theta _{1})}}{1-\beta _{1}^{1}}}={\frac {-5150}{0.1}}=-51500$

${\hat {v}}_{1}^{(\theta _{0})}={\frac {v_{1}^{(\theta _{0})}}{1-\beta _{2}^{1}}}={\frac {592.9}{1-0.999}}=592.9/0.001=592900$

${\hat {v}}_{1}^{(\theta _{1})}={\frac {v_{1}^{(\theta _{1})}}{1-0.999}}=2,652,250/0.001=2,652,250,000$

7. Nesterov adjustment:

For Nadam, incorporate Nesterov momentum by adding ${\frac {(1-\beta _{1})\partial J/\partial \theta }{1-\beta _{1}^{t}}}$ to $\beta _{1}{\hat {m}}_{t}$.

For $\theta _{0}$:

$m{\prime }_{1}(\theta _{0})=\beta _{1}{\hat {m}}_{1}^{(\theta _{0})}+{\frac {(1-\beta _{1}){\frac {\partial J}{\partial \theta _{0}}}}{1-\beta _{1}^{1}}}=0.9\cdot (-770)+{\frac {0.1\cdot (-770)}{0.1}}=(-693)+(-770)=-1463$

For $\theta _{1}$:

$m{\prime }_{1}(\theta _{1})=0.9\cdot (-51500)+{\frac {0.1\cdot (-51500)}{0.1}}=(-46350)+(-51500)=-97850$

8. Parameter update:

$\theta _{0}\leftarrow \theta _{0}-\eta {\frac {m{\prime }_{1}(\theta _{0})}{{\sqrt {{\hat {v}}_{1}^{(\theta _{0})}}}+\epsilon }}=0-0.01{\frac {-1463}{{\sqrt {592900}}+10^{-8}}}$

${\sqrt {592900}}\approx 770$. Thus:

$\theta _{0}\leftarrow 0-0.01{\frac {-1463}{770}}=0-0.01(-1.9)=0+0.019=0.019$

Cost surface trajectory for the linear regression task using NADAM

$\theta _{1}\leftarrow \theta _{1}-0.01{\frac {-97850}{{\sqrt {2,652,250,000}}+10^{-8}}}$

Cost function convergence and the final linear fit for the dataset

${\sqrt {2,652,250,000}}\approx 51500$. Thus:

$\theta _{1}\leftarrow 0-0.01{\frac {-97850}{51500}}=0-0.01(-1.9)=0.019$

After one iteration, the parameters have moved closer to values that reduce the cost.

Applications

Nadam’s integration of adaptive moment estimation with Nesterov accelerated gradients makes it a powerful tool for both convex and non-convex optimization problems, particularly in training complex neural networks. It has been widely applied in diverse domains: for instance, Nadam has been used to optimize convolutional neural networks (CNNs) in tasks like image denoising and human chromosome classification , demonstrating enhanced performance and faster convergence. In the context of time-series prediction, Nadam has been employed to optimize Long Short-Term Memory (LSTM) networks for predicting pressure in coal mining working faces, where they showed the Nadam-LSTM model outperforms LSTM model with basic gradient descent optimization . Furthermore, it has shown effectiveness in text sentiment analysis by training LSTMs, where its combination with L2 regularization achieved higher accuracy in fewer epochs . Nadam’s robustness and adaptability have led to its integration into major deep learning libraries such as TensorFlow [1] and PyTorch [2], making it a readily accessible optimizer for a wide range of machine learning tasks.

Conclusion

Nadam combines the adaptive learning rate adjustment of Adam with the anticipatory gradient updates of Nesterov momentum, making it a powerful extension of Adam. By incorporating the Nesterov approach, Nadam calculates gradients at a projected future position, which allows for more responsive and accurate updates, improving convergence speed and stability. This makes Nadam a robust tool for optimization tasks, particularly for training deep neural networks in both convex and non-convex settings. While Nadam has been effectively applied in areas like image classification, time-series prediction, and natural language processing, its suitability for specific problem types remains a topic for further exploration and research. Overall, Nadam offers several advantages, including faster convergence, improved handling of noisy gradients, and better generalization compared to traditional methods like SGD, Momentum, or Adam. These strengths make Nadam a versatile and reliable optimizer in modern machine learning applications.