Nadam - Revision history

Fall2024 Team17: /* Pseudocode */

2024-12-15T17:31:10Z

Pseudocode

← Older revision		Revision as of 13:31, 15 December 2024
Line 102:		Line 102:
	While ''not converged'' do		While ''not converged'' do

	<math> t\ \gets t\ +\ 1 </math>		<math> t\ \gets t\ +\ 1 </math>

	'''Compute gradient:''' <math> g_t=\nabla_\theta J\left(\theta_{t-1}\right) </math>		'''Compute gradient:''' <math> g_t=\nabla_\theta J\left(\theta_{t-1}\right) </math>

	'''Update biased first moment:''' <math> m_t\gets\beta_1\cdot m_{t-1}+\left(1-\beta_1\right)\cdot g_t </math>		'''Update biased first moment:''' <math> m_t\gets\beta_1\cdot m_{t-1}+\left(1-\beta_1\right)\cdot g_t </math>

	'''Update biased second moment:''' <math> v_t\gets\beta_2\cdot v_{t-1}+\left(1-\beta_2\right)\cdot g_t^2 </math>		'''Update biased second moment:''' <math> v_t\gets\beta_2\cdot v_{t-1}+\left(1-\beta_2\right)\cdot g_t^2 </math>

	'''Bias correction:''' <math> \widehat{m_t}\gets\frac{m_t}{1-\beta_1^t}, \widehat{v_t}\gets\frac{v_t}{1-\beta_2^t} </math>		'''Bias correction:''' <math> \widehat{m_t}\gets\frac{m_t}{1-\beta_1^t}, \widehat{v_t}\gets\frac{v_t}{1-\beta_2^t}
			</math>

	'''Nesterov adjustment:''' <math> m'_t \gets \beta_1 \cdot \widehat{m_t} + \frac{\left(1 - \beta_1\right) \cdot g_t}{1 - \beta_1^t} </math>		'''Nesterov adjustment:''' <math> m'_t \gets \beta_1 \cdot \widehat{m_t} + \frac{\left(1 - \beta_1\right) \cdot g_t}{1
			- \beta_1^t} </math>

	'''Update parameters:''' <math> \theta_t \gets \theta_{t-1} - \eta \cdot \frac{m'_t}{\sqrt{\widehat{v_t}} + \epsilon} </math>		'''Update parameters:''' <math> \theta_t \gets \theta_{t-1} - \eta \cdot \frac{m'_t}{\sqrt{\widehat{v_t}} + \epsilon}
			</math>

	'''Output:''' Optimized parameters <math> \theta_t </math>		'''Output:''' Optimized parameters <math> \theta_t </math>

Fall2024 Team17: added figures

2024-12-15T17:29:14Z

added figures

← Older revision		Revision as of 13:29, 15 December 2024
Line 333:		Line 333:

	<math> \hat{v}_1^{(\theta_1)}=\frac{v_1^{(\theta_1)}}{1-0.999}=2,652,250/0.001=2,652,250,000 </math>		<math> \hat{v}_1^{(\theta_1)}=\frac{v_1^{(\theta_1)}}{1-0.999}=2,652,250/0.001=2,652,250,000 </math>
	7. Nesterov adjustment:
			7. Nesterov adjustment:

	For Nadam, incorporate Nesterov momentum by adding <math> \frac{(1-\beta_1)\partial J/\partial \theta}{1-\beta_1^t} </math> to <math> \beta_1\hat{m}_t </math>.		For Nadam, incorporate Nesterov momentum by adding <math> \frac{(1-\beta_1)\partial J/\partial \theta}{1-\beta_1^t} </math> to <math> \beta_1\hat{m}_t </math>.
Line 352:		Line 353:

	<math> \theta_0 \leftarrow 0 - 0.01\frac{-1463}{770}=0 - 0.01(-1.9)=0 + 0.019=0.019 </math>		<math> \theta_0 \leftarrow 0 - 0.01\frac{-1463}{770}=0 - 0.01(-1.9)=0 + 0.019=0.019 </math>
			[[File:Cost surface.png\|thumb\|436x436px\|Cost surface trajectory for the linear regression task using NADAM]]
	<math> \theta_1 \leftarrow \theta_1 - 0.01\frac{-97850}{\sqrt{2,652,250,000}+10^{-8}} </math>		<math> \theta_1 \leftarrow \theta_1 - 0.01\frac{-97850}{\sqrt{2,652,250,000}+10^{-8}} </math>
			[[File:Loss regression.png\|thumb\|442x442px\|Cost function convergence and the final linear fit for the dataset]]
	<math> \sqrt{2,652,250,000}\approx 51500 </math>. Thus:		<math> \sqrt{2,652,250,000}\approx 51500 </math>. Thus:

Fall2024 Team17 at 17:06, 15 December 2024

2024-12-15T17:06:06Z

← Older revision		Revision as of 13:06, 15 December 2024
Line 362:		Line 362:

	== Applications ==		== Applications ==
	Nadam’s integration of adaptive moment estimation with Nesterov accelerated gradients makes it a powerful tool for both convex and non-convex optimization problems, particularly in training complex neural networks. It has been widely applied in diverse domains: for instance, Nadam has been used to optimize convolutional neural networks (CNNs) in tasks like image denoising ~~[5]~~ and human chromosome classification ~~[6]~~, demonstrating enhanced performance and faster convergence. In the context of time-series prediction, Nadam has been employed to optimize Long Short-Term Memory (LSTM) networks for predicting pressure in coal mining working faces, where they showed the Nadam-LSTM model outperforms LSTM model with basic gradient descent optimization ~~[7]~~. Furthermore, it has shown effectiveness in text sentiment analysis by training LSTMs, where its combination with L2 regularization achieved higher accuracy in fewer epochs ~~[8]~~. Nadam’s robustness and adaptability have led to its integration into major deep learning libraries such as TensorFlow [9] and PyTorch [10], making it a readily accessible optimizer for a wide range of machine learning tasks.		Nadam’s integration of adaptive moment estimation with Nesterov accelerated gradients makes it a powerful tool for both convex and non-convex optimization problems, particularly in training complex neural networks. It has been widely applied in diverse domains: for instance, Nadam has been used to optimize convolutional neural networks (CNNs) in tasks like image denoising <ref> L. Fan and Z. Long, “Optimization of Nadam algorithm for image denoising based on convolutional neural network,” in 2020 7th International Conference on Information Science and Control Engineering (ICISCE), Dec. 2020, pp. 957–961. doi: 10.1109/ICISCE50968.2020.00197.</ref> and human chromosome classification <ref> D. Menaka and S. G. Vaidyanathan, “Chromenet: a CNN architecture with comparison of optimizers for classification of human chromosome images,” Multidimens. Syst. Signal Process., vol. 33, no. 3, pp. 747–768, Sep. 2022, doi: 10.1007/s11045-022-00819-x.</ref>, demonstrating enhanced performance and faster convergence. In the context of time-series prediction, Nadam has been employed to optimize Long Short-Term Memory (LSTM) networks for predicting pressure in coal mining working faces, where they showed the Nadam-LSTM model outperforms LSTM model with basic gradient descent optimization <ref> J. Lu, Z. Liu, W. Zhang, J. Zheng, and C. Han, “Pressure Prediction Study of Coal Mining Working Face Based on Nadam-LSTM,” IEEE Access, vol. 11, pp. 83867–83880, 2023, doi: 10.1109/ACCESS.2023.3302516.</ref>. Furthermore, it has shown effectiveness in text sentiment analysis by training LSTMs, where its combination with L2 regularization achieved higher accuracy in fewer epochs <ref> J. Wang and Z. Cao, “Chinese text sentiment analysis using LSTM network based on L2 and Nadam,” in 2017 IEEE 17th International Conference on Communication Technology (ICCT), Oct. 2017, pp. 1891–1895. doi: 10.1109/ICCT.2017.8359958 </ref>. Nadam’s robustness and adaptability have led to its integration into major deep learning libraries such as TensorFlow [https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Nadam] and PyTorch [https://pytorch.org/docs/stable/generated/torch.optim.NAdam.html?utm_source=chatgpt.com], making it a readily accessible optimizer for a wide range of machine learning tasks.


Line 368:		Line 368:
	Nadam combines the adaptive learning rate adjustment of Adam with the anticipatory gradient updates of Nesterov momentum, making it a powerful extension of Adam. By incorporating the Nesterov approach, Nadam calculates gradients at a projected future position, which allows for more responsive and accurate updates, improving convergence speed and stability. This makes Nadam a robust tool for optimization tasks, particularly for training deep neural networks in both convex and non-convex settings. While Nadam has been effectively applied in areas like image classification, time-series prediction, and natural language processing, its suitability for specific problem types remains a topic for further exploration and research. Overall, Nadam offers several advantages, including faster convergence, improved handling of noisy gradients, and better generalization compared to traditional methods like SGD, Momentum, or Adam. These strengths make Nadam a versatile and reliable optimizer in modern machine learning applications.		Nadam combines the adaptive learning rate adjustment of Adam with the anticipatory gradient updates of Nesterov momentum, making it a powerful extension of Adam. By incorporating the Nesterov approach, Nadam calculates gradients at a projected future position, which allows for more responsive and accurate updates, improving convergence speed and stability. This makes Nadam a robust tool for optimization tasks, particularly for training deep neural networks in both convex and non-convex settings. While Nadam has been effectively applied in areas like image classification, time-series prediction, and natural language processing, its suitability for specific problem types remains a topic for further exploration and research. Overall, Nadam offers several advantages, including faster convergence, improved handling of noisy gradients, and better generalization compared to traditional methods like SGD, Momentum, or Adam. These strengths make Nadam a versatile and reliable optimizer in modern machine learning applications.

			== References ==
	<references/>		<references/>

Fall2024 Team17: /* Pseudocode */

2024-12-15T17:00:55Z

Pseudocode

← Older revision		Revision as of 13:00, 15 December 2024
Line 95:		Line 95:
	The pseudocode for Nadam is as follows:		The pseudocode for Nadam is as follows:

	'''Algorithm Nadam Optimization		'''Algorithm Nadam Optimization Input:'''
	Input:'''		Initial parameters <math> \theta_0 </math>, learning rate <math> \eta </math>, decay rates <math> \beta_1 </math>, <math> \beta_2 </math>, small constant <math> \epsilon </math>
	Initial parameters <math> \theta_0 </math>, learning rate <math> \eta </math>, decay rates <math> \beta_1 </math>, <math> \beta_2 </math>, small constant <math> \epsilon </math>

	'''Initialize''' <math> m_0\gets0,v_0\gets0,t\gets0 </math>		'''Initialize''' <math> m_0\gets0,v_0\gets0,t\gets0 </math>

Fall2024 Team17: /* Pseudocode */

2024-12-15T17:00:29Z

Pseudocode

← Older revision		Revision as of 13:00, 15 December 2024
Line 96:		Line 96:

	'''Algorithm Nadam Optimization		'''Algorithm Nadam Optimization
	Input:''' Initial parameters <math> \theta_0 </math>, learning rate <math> \eta </math>, decay rates <math> \beta_1 </math>, <math> \beta_2 </math>, small constant <math> \epsilon </math>		Input:'''
			Initial parameters <math> \theta_0 </math>, learning rate <math> \eta </math>, decay rates <math> \beta_1 </math>, <math> \beta_2 </math>, small constant <math> \epsilon </math>

	'''Initialize''' <math> m_0\gets0,v_0\gets0,t\gets0 </math>		'''Initialize''' <math> m_0\gets0,v_0\gets0,t\gets0 </math>

Fall2024 Team17: /* Pseudocode */

2024-12-15T16:59:56Z

Pseudocode

Fall2024 Team17: /* Algorithm */

2024-12-15T16:58:10Z

Algorithm

← Older revision		Revision as of 12:58, 15 December 2024
Line 64:		Line 64:
	== Algorithm ==		== Algorithm ==

	Nadam modifies the parameter update step of Adam by integrating the Nesterov accelerated gradient. In Adam, the update relies solely on the bias-corrected first moment estimate <math> \widehat{m_t} </math>. Nadam modifies the parameter update step of Adam by integrating the Nesterov accelerated gradient. In Adam, the update relies solely on the bias-corrected first moment estimate <math> \widehat{m_t} </math>. Nadam, however, adjusts this estimate by adding a term that accounts for the current gradient, effectively looking ahead to where the parameters are heading. This adjustment is similar to the Nesterov momentum in NAG, which computes the gradient at the approximate future position of the parameters. Firstly, consider the bias-corrected first moment <math> \widehat{m_t} </math> using the definition of <math> m_t </math> <ref name="ruder" \>		Nadam modifies the parameter update step of Adam by integrating the Nesterov accelerated gradient. In Adam, the update relies solely on the bias-corrected first moment estimate <math> \widehat{m_t} </math>. Nadam modifies the parameter update step of Adam by integrating the Nesterov accelerated gradient. In Adam, the update relies solely on the bias-corrected first moment estimate <math> \widehat{m_t} </math>. Nadam, however, adjusts this estimate by adding a term that accounts for the current gradient, effectively looking ahead to where the parameters are heading. This adjustment is similar to the Nesterov momentum in NAG, which computes the gradient at the approximate future position of the parameters. Firstly, consider the bias-corrected first moment <math> \widehat{m_t} </math> using the definition of <math> m_t </math> <ref name="ruder" />

	<math> \widehat{m_t}=\frac{m_t}{1-\beta_1^t}=\frac{\beta_1m_{t-1}+\left(1-\beta_1\right)g_t}{1-\beta_1^t} </math>		<math> \widehat{m_t}=\frac{m_t}{1-\beta_1^t}=\frac{\beta_1m_{t-1}+\left(1-\beta_1\right)g_t}{1-\beta_1^t} </math>

Fall2024 Team17: /* Algorithm */

2024-12-15T16:57:54Z

Algorithm

← Older revision		Revision as of 12:57, 15 December 2024
Line 64:		Line 64:
	== Algorithm ==		== Algorithm ==

	Nadam modifies the parameter update step of Adam by integrating the Nesterov accelerated gradient. In Adam, the update relies solely on the bias-corrected first moment estimate <math> \widehat{m_t} </math>. Nadam modifies the parameter update step of Adam by integrating the Nesterov accelerated gradient. In Adam, the update relies solely on the bias-corrected first moment estimate <math> \widehat{m_t} </math>. Nadam, however, adjusts this estimate by adding a term that accounts for the current gradient, effectively looking ahead to where the parameters are heading. This adjustment is similar to the Nesterov momentum in NAG, which computes the gradient at the approximate future position of the parameters. Firstly, consider the bias-corrected first moment <math> \widehat{m_t} </math> using the definition of <math> m_t </math> ~~[3]~~		Nadam modifies the parameter update step of Adam by integrating the Nesterov accelerated gradient. In Adam, the update relies solely on the bias-corrected first moment estimate <math> \widehat{m_t} </math>. Nadam modifies the parameter update step of Adam by integrating the Nesterov accelerated gradient. In Adam, the update relies solely on the bias-corrected first moment estimate <math> \widehat{m_t} </math>. Nadam, however, adjusts this estimate by adding a term that accounts for the current gradient, effectively looking ahead to where the parameters are heading. This adjustment is similar to the Nesterov momentum in NAG, which computes the gradient at the approximate future position of the parameters. Firstly, consider the bias-corrected first moment <math> \widehat{m_t} </math> using the definition of <math> m_t </math> <ref name="ruder" \>

	<math> \widehat{m_t}=\frac{m_t}{1-\beta_1^t}=\frac{\beta_1m_{t-1}+\left(1-\beta_1\right)g_t}{1-\beta_1^t} </math>		<math> \widehat{m_t}=\frac{m_t}{1-\beta_1^t}=\frac{\beta_1m_{t-1}+\left(1-\beta_1\right)g_t}{1-\beta_1^t} </math>
Line 90:		Line 90:

	By combining the bias-corrected first moment estimate <math> \widehat{m_t} </math> and the current gradient <math> g_t </math>, weighted appropriately, Nadam aims to improve the responsiveness of the optimizer to recent gradient information, potentially enhancing convergence speed.		By combining the bias-corrected first moment estimate <math> \widehat{m_t} </math> and the current gradient <math> g_t </math>, weighted appropriately, Nadam aims to improve the responsiveness of the optimizer to recent gradient information, potentially enhancing convergence speed.

			=== Pseudocode ===

	The pseudocode for Nadam is as follows:		The pseudocode for Nadam is as follows:

Fall2024 Team17: /* Background */

2024-12-15T16:56:47Z

Background

← Older revision		Revision as of 12:56, 15 December 2024
Line 28:		Line 28:
	<math> \theta_{t+1}=\theta_t-{(\gamma m}t\ +\ \eta{\ \nabla}\theta J\left(\theta_t\right)\ ) </math>		<math> \theta_{t+1}=\theta_t-{(\gamma m}t\ +\ \eta{\ \nabla}\theta J\left(\theta_t\right)\ ) </math>

	AdaGrad (Adaptive Gradient Algorithm) adapts the learning rate for each parameter individually by scaling it inversely proportional to the square root of the sum of all historical squared values of the gradients. This method is particularly useful for dealing with sparse data <ref name="ruder" \>.		AdaGrad (Adaptive Gradient Algorithm) adapts the learning rate for each parameter individually by scaling it inversely proportional to the square root of the sum of all historical squared values of the gradients. This method is particularly useful for dealing with sparse data <ref name="ruder" />.

	<math> v_t=v_{t-1}+\left[\nabla_\theta J\left(\theta_t\right)\right]^2 </math>		<math> v_t=v_{t-1}+\left[\nabla_\theta J\left(\theta_t\right)\right]^2 </math>

Fall2024 Team17: /* Background */

2024-12-15T16:56:18Z

Background

← Older revision		Revision as of 12:56, 15 December 2024
Line 34:		Line 34:
	<math> \theta_{t+1}=\theta_t-\eta\frac{\nabla_\theta J\left(\theta_t\right)}{\sqrt{v_t}+\epsilon} </math>		<math> \theta_{t+1}=\theta_t-\eta\frac{\nabla_\theta J\left(\theta_t\right)}{\sqrt{v_t}+\epsilon} </math>

	<math> v_t </math> is the accumulated sum of squared gradients up to time <math> t </math>, and <math> \epsilon </math> is a small constant for numerical stability (e.g., <math> 10^{-8} </math>). AdaGrad effectively scales down the learning rate for parameters associated with frequently occurring features, allowing for larger updates for infrequent features <ref name="Dozat" \>.		<math> v_t </math> is the accumulated sum of squared gradients up to time <math> t </math>, and <math> \epsilon </math> is a small constant for numerical stability (e.g., <math> 10^{-8} </math>). AdaGrad effectively scales down the learning rate for parameters associated with frequently occurring features, allowing for larger updates for infrequent features <ref name="Dozat" />.

	RMSprop (Root Mean Square Propagation) is an optimization algorithm that builds upon AdaGrad by modifying the way it accumulates squared gradients. While AdaGrad accumulates all past squared gradients, leading to a continually decreasing learning rate, RMSprop introduces a decay parameter to the accumulated squared gradients <math> v_t </math>. This parameterization prevents the learning rate from diminishing too quickly.		RMSprop (Root Mean Square Propagation) is an optimization algorithm that builds upon AdaGrad by modifying the way it accumulates squared gradients. While AdaGrad accumulates all past squared gradients, leading to a continually decreasing learning rate, RMSprop introduces a decay parameter to the accumulated squared gradients <math> v_t </math>. This parameterization prevents the learning rate from diminishing too quickly.

← Older revision		Revision as of 12:59, 15 December 2024
Line 95:		Line 95:
	The pseudocode for Nadam is as follows:		The pseudocode for Nadam is as follows:

	Algorithm Nadam Optimization		'''Algorithm Nadam Optimization
	Input: Initial parameters <math> \theta_0 </math>, learning rate <math> \eta </math>, decay rates <math> \beta_1 </math>, <math> \beta_2 </math>, small constant <math> \epsilon </math>		Input:''' Initial parameters <math> \theta_0 </math>, learning rate <math> \eta </math>, decay rates <math> \beta_1 </math>, <math> \beta_2 </math>, small constant <math> \epsilon </math>

	Initialize <math> m_0\gets0,v_0\gets0,t\gets0 </math>		'''Initialize''' <math> m_0\gets0,v_0\gets0,t\gets0 </math>

	While not converged do		While ''not converged'' do

	<math> t\ \gets t\ +\ 1 </math>		<math> t\ \gets t\ +\ 1 </math>

	Compute gradient: <math> g_t=\nabla_\theta J\left(\theta_{t-1}\right) </math>		'''Compute gradient:''' <math> g_t=\nabla_\theta J\left(\theta_{t-1}\right) </math>

	Update biased first moment: <math> m_t\gets\beta_1\cdot m_{t-1}+\left(1-\beta_1\right)\cdot g_t </math>		'''Update biased first moment:''' <math> m_t\gets\beta_1\cdot m_{t-1}+\left(1-\beta_1\right)\cdot g_t </math>

	Update biased second moment: <math> v_t\gets\beta_2\cdot v_{t-1}+\left(1-\beta_2\right)\cdot g_t^2 </math>		'''Update biased second moment:''' <math> v_t\gets\beta_2\cdot v_{t-1}+\left(1-\beta_2\right)\cdot g_t^2 </math>

	Bias correction: <math> \widehat{m_t}\gets\frac{m_t}{1-\beta_1^t}, \widehat{v_t}\gets\frac{v_t}{1-\beta_2^t} </math>		'''Bias correction:''' <math> \widehat{m_t}\gets\frac{m_t}{1-\beta_1^t}, \widehat{v_t}\gets\frac{v_t}{1-\beta_2^t} </math>

	Nesterov adjustment: <math> m'_t \gets \beta_1 \cdot \widehat{m_t} + \frac{\left(1 - \beta_1\right) \cdot g_t}{1 - \beta_1^t} </math>		'''Nesterov adjustment:''' <math> m'_t \gets \beta_1 \cdot \widehat{m_t} + \frac{\left(1 - \beta_1\right) \cdot g_t}{1 - \beta_1^t} </math>

	Update parameters: <math> \theta_t \gets \theta_{t-1} - \eta \cdot \frac{m'_t}{\sqrt{\widehat{v_t}} + \epsilon} </math>		'''Update parameters:''' <math> \theta_t \gets \theta_{t-1} - \eta \cdot \frac{m'_t}{\sqrt{\widehat{v_t}} + \epsilon} </math>

	Output: Optimized parameters <math> \theta_t </math>		'''Output:''' Optimized parameters <math> \theta_t </math>

	== Numerical Examples ==		== Numerical Examples ==