Adadelta - Revision history

SYSEN5800 Team19: /* Numerical Example */

2024-12-15T04:12:27Z

Numerical Example

← Older revision		Revision as of 00:12, 15 December 2024
Line 50:		Line 50:
	* The RMS values of <math>\Delta x</math> and <math>g</math> can serve as an approximation to the diagonal Hessian.		* The RMS values of <math>\Delta x</math> and <math>g</math> can serve as an approximation to the diagonal Hessian.

	== Numerical ~~Example~~ ==		== Numerical Examples ==
	=== ~~Problem Definition and Setup~~ ===		=== Example 1 ===
	Consider minimizing the quadratic loss function <math>f(x) = x^2</math>, starting at <math>x = 5</math> using Adadelta with <math>\rho = 0.95</math> and <math>\epsilon = 10^{-6}</math>.		Consider minimizing the quadratic loss function <math>f(x) = x^2</math>, starting at <math>x = 5</math> using Adadelta with <math>\rho = 0.95</math> and <math>\epsilon = 10^{-6}</math>.

Line 63:		Line 63:

	Where <math>y</math> can be either <math>g</math> (gradient) or <math>\Delta x</math> (parameter update).		Where <math>y</math> can be either <math>g</math> (gradient) or <math>\Delta x</math> (parameter update).

	~~=== Numerical Solution ===~~


	'''First Iteration:'''		'''First Iteration:'''
Line 118:		Line 115:
	</li>		</li>
	</ol>		</ol>

	~~=== Solution Graphical Analysis ===~~

	The continuation of this process, displayed in '''Figure 1''', shows that it takes about 1500 iterations to converge.		The continuation of this process, displayed in '''Figure 1''', shows that it takes about 1500 iterations to converge.
Line 125:		Line 120:
	[[File:Figure1.png\|thumb\|center\|500px\|Figure 1. Iterating <math>x</math> and <math>\Delta x</math> during the minimization of <math>f(x) = x^2</math> using the Adadelta algorithm.]]		[[File:Figure1.png\|thumb\|center\|500px\|Figure 1. Iterating <math>x</math> and <math>\Delta x</math> during the minimization of <math>f(x) = x^2</math> using the Adadelta algorithm.]]

	~~'''Next Steps:'''~~		=== Example 2 ===

	Next, Adadelta is applied to optimize a more complex function, <math>u(x) = x^2 + 10 \sin(2x)</math>, such that it equals 10. Mean squared error will be used as the loss function, resulting in the following:		Next, Adadelta is applied to optimize a more complex function, <math>u(x) = x^2 + 10 \sin(2x)</math>, such that it equals 10. Mean squared error will be used as the loss function, resulting in the following:

SYSEN5800 Team19: /* Solution Graphical Analysis */

2024-12-15T04:11:06Z

Solution Graphical Analysis

← Older revision		Revision as of 00:11, 15 December 2024
Line 120:		Line 120:

	=== Solution Graphical Analysis ===		=== Solution Graphical Analysis ===
	[[File:Figure1.png\|center\|~~thumb~~]]
	[[File:Figure2.png\|center\|~~thumb~~]]		The continuation of this process, displayed in '''Figure 1''', shows that it takes about 1500 iterations to converge.
	[[File:Figure3.png\|center\|~~thumb~~]]
			[[File:Figure1.png\|thumb\|center\|500px\|Figure 1. Iterating <math>x</math> and <math>\Delta x</math> during the minimization of <math>f(x) = x^2</math> using the Adadelta algorithm.]]

			'''Next Steps:'''

			Next, Adadelta is applied to optimize a more complex function, <math>u(x) = x^2 + 10 \sin(2x)</math>, such that it equals 10. Mean squared error will be used as the loss function, resulting in the following:

			<math>\text{Loss} = f(x) = (10 - (x^2 + 10 \sin(2x)))^2</math>

			<math>\frac{df}{dx} = -2(10 - x^2 - 10 \sin(2x))(2x + 20 \cos(2x))</math>

			Starting at <math>x_0 = 0.85</math> (shown in '''Figure 2'''), apply the Adadelta algorithm described in the <math>f(x) = x^2</math> case to minimize the new loss function. '''Figure 3''' tracks the algorithm's progress showing both <math>\Delta x</math> (parameter updates) and the loss function value over 50 iterations where it converges at approximately iteration 30 where <math>x = 1.01355</math> and <math>u(x) = 10.00415</math>.

			[[File:Figure2.png\|thumb\|center\|500px\|Figure 2. Plotted <math>u(x)</math> and Loss.]]

			[[File:Figure3.png\|thumb\|center\|500px\|Figure 3. Iterating Loss and <math>\Delta x</math> during the minimization of <math>\text{Loss} = f(x) = (10 - (x^2 + 10 \sin(2x)))^2</math> using the Adadelta algorithm.]]

	== Application ==		== Application ==

SYSEN5800 Team19: /* Solution Graphical Analysis */

2024-12-15T04:06:39Z

Solution Graphical Analysis

← Older revision		Revision as of 00:06, 15 December 2024
Line 120:		Line 120:

	=== Solution Graphical Analysis ===		=== Solution Graphical Analysis ===
			[[File:Figure1.png\|center\|thumb]]
			[[File:Figure2.png\|center\|thumb]]
			[[File:Figure3.png\|center\|thumb]]

	== Application ==		== Application ==

SYSEN5800 Team19: /* Numerical Solution */

2024-12-15T03:51:50Z

Numerical Solution

← Older revision		Revision as of 23:51, 14 December 2024
Line 65:		Line 65:

	=== Numerical Solution ===		=== Numerical Solution ===


			'''First Iteration:'''
			* Initial conditions: <math>x_1 = 5</math>, <math>\mathbb{E}[g^2]_0 = 0</math>, <math>\mathbb{E}[\Delta x^2]_0 = 0</math>
			* Gradient: <math>g_1 = 2(5) = 10.00</math>

			'''Computing step:'''
			<ol>
			<li>Gradient accumulation:
			<ol type="a">
			<li><math>\mathbb{E}[g^2]_1 = 0.95(0) + (1 - 0.95)(100.0) = 5.000</math></li>
			<li><math>\text{RMS}[g]_1 = \sqrt{5.000 + 10^{-6}} = 2.236</math></li>
			</ol>
			</li>
			<li>Parameter update:
			<ol type="a">
			<li><math>\text{RMS}[\Delta x]_0 = \sqrt{0 + 10^{-6}} = 1.000 \times 10^{-3}</math></li>
			<li><math>\Delta x_1 = - \frac{\text{RMS}[\Delta x]_0}{\text{RMS}[g]_1} g_1 = - \frac{1.000 \times 10^{-3}}{2.236} (10.00) = -4.472 \times 10^{-3}</math></li>
			</ol>
			</li>
			<li>Update accumulation:
			<ol type="a">
			<li><math>\mathbb{E}[\Delta x^2]_1 = 0.95(0) + (1 - 0.95)((4.472 \times 10^{-3})^2) = 1.000 \times 10^{-6}</math></li>
			<li><math>x_2 = 5 + (-4.472 \times 10^{-3}) = 4.996</math></li>
			</ol>
			</li>
			</ol>

			'''Second Iteration:'''
			* Current position: <math>x_2 = 4.996</math>
			* Gradient: <math>g_2 = 2(4.996) = 9.992</math>

			'''Computing step:'''
			<ol>
			<li>Gradient accumulation:
			<ol type="a">
			<li><math>\mathbb{E}[g^2]_2 = 0.95(5.000) + (1 - 0.95)(99.84) = 9.742</math></li>
			<li><math>\text{RMS}[g]_2 = \sqrt{9.742 + 10^{-6}} = 3.121</math></li>
			</ol>
			</li>
			<li>Parameter update:
			<ol type="a">
			<li><math>\text{RMS}[\Delta x]_1 = \sqrt{1.000 \times 10^{-6} + 10^{-6}} = 1.414 \times 10^{-3}</math></li>
			<li><math>\Delta x_2 = - \frac{\text{RMS}[\Delta x]_1}{\text{RMS}[g]_2} g_2 = - \frac{1.414 \times 10^{-3}}{3.121} (9.992) = -4.524 \times 10^{-3}</math></li>
			</ol>
			</li>
			<li>Update accumulation:
			<ol type="a">
			<li><math>\mathbb{E}[\Delta x^2]_2 = 0.95(1.000 \times 10^{-6}) + (1 - 0.95)((4.524 \times 10^{-3})^2) = 1.023 \times 10^{-6}</math></li>
			<li><math>x_3 = 4.996 + (-4.524 \times 10^{-3}) = 4.991</math></li>
			</ol>
			</li>
			</ol>

	=== Solution Graphical Analysis ===		=== Solution Graphical Analysis ===

SYSEN5800 Team19: /* Problem Definition and Setup */

2024-12-15T03:31:25Z

Problem Definition and Setup

@@ Line 52: / Line 52: @@
 == Numerical Example ==
 === Problem Definition and Setup ===
 === Numerical Solution ===
 === Solution Graphical Analysis ===

SYSEN5800 Team19: /* Algorithm Discussion */

2024-12-15T03:27:08Z

Algorithm Discussion

← Older revision		Revision as of 23:27, 14 December 2024
Line 36:		Line 36:

	The complete algorithm proceeds as follows:		The complete algorithm proceeds as follows:
	# Initialize accumulation variables <math>\mathbb{E}[g^2]_0 = 0</math> and <math>\mathbb{E}[\Delta x^2]_0 = 0</math>.		# Initialize accumulation variables <math>\mathbb{E}[g^2]_0 = 0</math> and <math>\mathbb{E}[\Delta x^2]_0 = 0</math>
	# For each time step <math>t</math>:		# For each time step <math>t</math>:
	## Compute gradient <math>g_t</math>.		## Compute gradient <math>g_t</math>
	## Accumulate squared gradient: <math>\mathbb{E}[g^2]_t = \rho \mathbb{E}[g^2]_{t-1} + (1 - \rho) g_t^2</math>.		## Accumulate squared gradient: <math>\mathbb{E}[g^2]_t = \rho \mathbb{E}[g^2]_{t-1} + (1 - \rho) g_t^2</math>
	## Compute update: <math>\Delta x_t = - \frac{\sqrt{\mathbb{E}[\Delta x^2]_{t-1} + \epsilon}}{\sqrt{\mathbb{E}[g^2]_t + \epsilon}} g_t</math>.		## Compute update: <math>\Delta x_t = - \frac{\sqrt{\mathbb{E}[\Delta x^2]_{t-1} + \epsilon}}{\sqrt{\mathbb{E}[g^2]_t + \epsilon}} g_t</math>
	## Accumulate squared updates: <math>\mathbb{E}[\Delta x^2]_t = \rho \mathbb{E}[\Delta x^2]_{t-1} + (1 - \rho) (\Delta x_t)^2</math>.		## Accumulate squared updates: <math>\mathbb{E}[\Delta x^2]_t = \rho \mathbb{E}[\Delta x^2]_{t-1} + (1 - \rho) (\Delta x_t)^2</math>
	## Apply update: <math>x_{t+1} = x_t + \Delta x_t</math>.		## Apply update: <math>x_{t+1} = x_t + \Delta x_t</math>

	This algorithm assumes:		This algorithm assumes:

SYSEN5800 Team19: /* Algorithm Discussion */

2024-12-15T03:23:57Z

Algorithm Discussion

SYSEN5800 Team19: /* Algorithm Discussion */

2024-12-15T03:19:00Z

Algorithm Discussion

← Older revision		Revision as of 23:19, 14 December 2024
Line 13:		Line 13:

	== Algorithm Discussion ==		== Algorithm Discussion ==
			Adadelta determines adaptive learning rates by maintaining running averages of squared gradients and parameter updates over a window. By using only first-order gradient information, these averages allow the algorithm to automatically scale the learning rate for each parameter based on its historical behavior.

			The core update rule for Adadelta at time ''t'' is:

			<math>\Delta \theta_t = - \frac{\text{RMS}[g_t]}{\text{RMS}[\Delta \theta_{t-1}]} g_t</math>

			where:
			* <math>g_t</math> is the gradient at time <math>t</math>,
			* <math>\text{RMS}[g_t]</math> is the root mean square of accumulated gradients,
			* <math>\text{RMS}[\Delta \theta_{t-1}]</math> is the root mean square of accumulated parameter updates,
			* <math>\Delta \theta_t</math> is the parameter update at time <math>t</math>.

			The RMS terms are computed using exponentially decaying averages:

			<math>\text{RMS}[g_t] = \sqrt{\mathbb{E}[g_t^2] + \epsilon}</math>

			<math>\text{RMS}[\Delta \theta_t] = \sqrt{\mathbb{E}[\Delta \theta_t^2] + \epsilon}</math>

			where:
			* <math>\mathbb{E}[g_t^2]</math> and <math>\mathbb{E}[\Delta \theta_t^2]</math> are the exponentially decaying averages,
			* <math>\epsilon</math> is a small constant for numerical stability.

			The complete algorithm proceeds as follows:
			# Initialize accumulation variables <math>\mathbb{E}[g^2]_0</math> and <math>\mathbb{E}[\Delta \theta^2]_0</math>.
			# For each time step <math>t</math>:
			## Compute gradient <math>g_t</math>.
			## Accumulate squared gradient: <math>\mathbb{E}[g^2]_t = \rho \mathbb{E}[g^2]_{t-1} + (1 - \rho) g_t^2</math>.
			## Compute update: <math>\Delta \theta_t = - \frac{\sqrt{\mathbb{E}[\Delta \theta^2]_{t-1} + \epsilon}}{\sqrt{\mathbb{E}[g^2]_t + \epsilon}} g_t</math>.
			## Accumulate squared updates: <math>\mathbb{E}[\Delta \theta^2]_t = \rho \mathbb{E}[\Delta \theta^2]_{t-1} + (1 - \rho) (\Delta \theta_t)^2</math>.
			## Apply update: <math>\theta_{t+1} = \theta_t + \Delta \theta_t</math>.

			Assumptions:
			* The objective function is differentiable with respect to all parameters.
			* Parameters can be updated independently.
			* Exponential decay provides a reasonable approximation of recent gradient history.
			* RMS values of <math>g_t</math> and <math>\Delta \theta_t</math> can serve as an approximation to the diagonal Hessian.

	== Numerical Example ==		== Numerical Example ==

SYSEN5800 Team19: /* Application */

2024-12-15T03:12:33Z

Application

← Older revision		Revision as of 23:12, 14 December 2024
Line 21:		Line 21:
	== Application ==		== Application ==
	The Adadelta optimization algorithm is commonly used in deep learning systems with sparse gradients <ref name="Zeiler2012" />. Adadelta particularly excels in training complex neural architectures such as deep convoluted neural networks and sequence models, where gradient magnitudes may vary significantly across different layers. Its adaptive learning rate algorithm makes it best applied in architectures with varying parameter scales. Adadelta has been applied as a core optimizer in deep learning frameworks such as TensorFlow <ref name="TensorFlow" />, PyTorch <ref name="PyTorch" />, and Keras<ref name="Keras" />, each providing specific features and usage.		The Adadelta optimization algorithm is commonly used in deep learning systems with sparse gradients <ref name="Zeiler2012" />. Adadelta particularly excels in training complex neural architectures such as deep convoluted neural networks and sequence models, where gradient magnitudes may vary significantly across different layers. Its adaptive learning rate algorithm makes it best applied in architectures with varying parameter scales. Adadelta has been applied as a core optimizer in deep learning frameworks such as TensorFlow <ref name="TensorFlow" />, PyTorch <ref name="PyTorch" />, and Keras<ref name="Keras" />, each providing specific features and usage.
	TensorFlow: In TensorFlow, Adadelta is available in the tf.keras.optimizers module. This framework is designed with a clearly defined API for setting up Adadelta with popular defaults for learning rate, rho, and epsilon. Users are allowed to change these for better results that best fit model requirements, hence flexible and easy to use.		* '''TensorFlow:''' In TensorFlow, Adadelta is available in the <code>tf.keras.optimizers</code> module. This framework is designed with a clearly defined API for setting up Adadelta with popular defaults for learning rate, rho, and epsilon. Users are allowed to change these for better results that best fit model requirements, hence flexible and easy to use.
	*PyTorch: In PyTorch, Adadelta is available in the torch.optim module. This implementation is highly configurable and naturally interoperates well with the dynamic computational graph in PyTorch, which makes it very suitable for training models where the layer structure is complex and/or where the gradient behavior is irregular.		* '''PyTorch:''' In PyTorch, Adadelta is available in the <code>torch.optim</code> module. This implementation is highly configurable and naturally interoperates well with the dynamic computational graph in PyTorch, which makes it very suitable for training models where the layer structure is complex and/or where the gradient behavior is irregular.
	*Keras: In Keras, Adadelta is available in the keras.optimizers module. The model can be executed directly during the process of model compilation, providing a straightforward yet efficient interface. The architecture emphasizes user-friendliness, enabling developers to focus on the construction and enhancement of their models while taking advantage of Adadelta's adaptive learning features.		* '''Keras:''' In Keras, Adadelta is available in the <code>keras.optimizers</code> module. The model can be executed directly during the process of model compilation, providing a straightforward yet efficient interface. The architecture emphasizes user-friendliness, enabling developers to focus on the construction and enhancement of their models while taking advantage of Adadelta's adaptive learning features.
	*While another similar optimizer, Adam, has recently gained more popularity in application because of its adaptive properties <ref name="Wilson2017" />, Adadelta is still relevantly applicable for specific use cases in favor of per-dimension learning rate adaptation.		While another similar optimizer, Adam, has recently gained more popularity in application because of its adaptive properties <ref name="Wilson2017" />, Adadelta is still relevantly applicable for specific use cases in favor of per-dimension learning rate adaptation.

	For its adaptive nature, Adadelta has the potential to be applied in both engineering and finance <ref name="ScienceDirect2024" />. In engineering, deep learning models, such as recurrent neural networks (RNNs), are widely used to analyze time-series sensor data in predicting when industrial machines need maintenance <ref name="Liu2024" />. Although the benefits of Adadelta have not been clearly explained in this area, its ability to adaptively update gradients makes it a good option for improving these models. In finance, deep learning models have been widely studied for tasks including stock price prediction and credit risk assessment using the optimizer Adam and RMSprop <ref name="Chong2017" />. Not many studies mention Adadelta, but its adaptation capability makes it a good choice for similar tasks in financial modeling. The incorporation of Adadelta in widely used frameworks such as TensorFlow, PyTorch, and Keras convinces its availability for various applications, enabling practitioners to integrate it into predictive analytics and financial modeling activities effectively.		For its adaptive nature, Adadelta has the potential to be applied in both engineering and finance <ref name="ScienceDirect2024" />. In engineering, deep learning models, such as recurrent neural networks (RNNs), are widely used to analyze time-series sensor data in predicting when industrial machines need maintenance <ref name="Liu2024" />. Although the benefits of Adadelta have not been clearly explained in this area, its ability to adaptively update gradients makes it a good option for improving these models. In finance, deep learning models have been widely studied for tasks including stock price prediction and credit risk assessment using the optimizer Adam and RMSprop <ref name="Chong2017" />. Not many studies mention Adadelta, but its adaptation capability makes it a good choice for similar tasks in financial modeling. The incorporation of Adadelta in widely used frameworks such as TensorFlow, PyTorch, and Keras convinces its availability for various applications, enabling practitioners to integrate it into predictive analytics and financial modeling activities effectively.

SYSEN5800 Team19: Deleting Wenqu's name

2024-12-15T03:07:38Z

Deleting Wenqu's name

← Older revision		Revision as of 23:07, 14 December 2024
Line 1:		Line 1:
	Author: Imran Shita-Bey (ias45), Dhruv Misra (dm668), Ifadhila Affia (ia284~~), Wenqu Zhang (wz473~~) (ChemE 6800 Fall 2024)		Author: Imran Shita-Bey (ias45), Dhruv Misra (dm668), Ifadhila Affia (ia284) (ChemE 6800 Fall 2024)

	Stewards: Nathan Preuss, Wei-Han Chen, Tianqi Xiao, Guoqing Hu		Stewards: Nathan Preuss, Wei-Han Chen, Tianqi Xiao, Guoqing Hu

← Older revision		Revision as of 23:23, 14 December 2024
Line 13:		Line 13:

	== Algorithm Discussion ==		== Algorithm Discussion ==
	Adadelta determines adaptive learning rates by maintaining running averages of squared gradients and parameter updates over a window. ~~By using~~ only first-order gradient information, these averages allow the algorithm to automatically scale the learning rate for each parameter based on ~~its~~ historical behavior.		Adadelta determines adaptive learning rates by maintaining running averages of squared gradients and parameter updates over a window. Using only first-order gradient information, these averages allow the algorithm to automatically scale the learning rate for each parameter based on their historical behavior.

	The core update rule for Adadelta at time ''t'' is:		The core update rule for Adadelta at time ''t'' is:

	<math>\Delta ~~\theta_t~~ = - \frac{\text{RMS}[~~g_t~~]}{\text{RMS}[~~\Delta \theta_{t-1}~~]} g_t</math>		<math>\Delta x_t = - \frac{\text{RMS}[\Delta x]_{t-1}}{\text{RMS}[g]_t} g_t</math>

	where:		where:
	* <math>g_t</math> is the gradient at time <math>t</math>,		* <math>g_t</math> is the gradient at time <math>t</math>,
	* <math>\text{RMS}[~~g_t~~]</math> is the root mean square of accumulated gradients,		* <math>\text{RMS}[g]_t</math> is the root mean square of accumulated gradients,
	* <math>\text{RMS}[\Delta ~~\theta_~~{t-1}]</math> is the root mean square of accumulated parameter updates,		* <math>\text{RMS}[\Delta x]_{t-1}</math> is the root mean square of accumulated parameter updates,
	* <math>\Delta ~~\theta_t~~</math> is the parameter update at time <math>t</math>.		* <math>\Delta x_t</math> is the parameter update at time <math>t</math>.

	The RMS terms are computed using exponentially decaying averages:		The RMS terms are computed using exponentially decaying averages:

	<math>\~~text~~{~~RMS~~}[~~g_t~~] = \~~sqrt{~~\mathbb{E}[~~g_t~~^2] + \~~epsilon}~~</math>		<math>\mathbb{E}[g^2]_t = \rho \mathbb{E}[g^2]_{t-1} + (1 - \rho) g_t^2</math>

	<math>\text{RMS}[~~\Delta \theta_t~~] = \sqrt{\mathbb{E}[~~\Delta \theta_t~~^2] + \epsilon}</math>		<math>\text{RMS}[g]_t = \sqrt{\mathbb{E}[g^2]_t + \epsilon}</math>

	where:		where:
	* <math>\~~mathbb{E}[g_t^2]~~</math> ~~and <math>\mathbb{E}[\Delta \theta_t^2]</math> are~~ the ~~exponentially decaying averages~~,		* <math>\rho</math> is the decay constant,
	* <math>\epsilon</math> is a small constant for numerical stability.		* <math>\epsilon</math> is a small constant for numerical stability.

	The complete algorithm proceeds as follows:		The complete algorithm proceeds as follows:
	# Initialize accumulation variables <math>\mathbb{E}[g^2]_0</math> and <math>\mathbb{E}[\Delta ~~\theta~~^2]_0</math>.		# Initialize accumulation variables <math>\mathbb{E}[g^2]_0 = 0</math> and <math>\mathbb{E}[\Delta x^2]_0 = 0</math>.
	# For each time step <math>t</math>:		# For each time step <math>t</math>:
	## Compute gradient <math>g_t</math>.		## Compute gradient <math>g_t</math>.
	## Accumulate squared gradient: <math>\mathbb{E}[g^2]_t = \rho \mathbb{E}[g^2]_{t-1} + (1 - \rho) g_t^2</math>.		## Accumulate squared gradient: <math>\mathbb{E}[g^2]_t = \rho \mathbb{E}[g^2]_{t-1} + (1 - \rho) g_t^2</math>.
	## Compute update: <math>\Delta ~~\theta_t~~ = - \frac{\sqrt{\mathbb{E}[\Delta ~~\theta~~^2]_{t-1} + \epsilon}}{\sqrt{\mathbb{E}[g^2]_t + \epsilon}} g_t</math>.		## Compute update: <math>\Delta x_t = - \frac{\sqrt{\mathbb{E}[\Delta x^2]_{t-1} + \epsilon}}{\sqrt{\mathbb{E}[g^2]_t + \epsilon}} g_t</math>.
	## Accumulate squared updates: <math>\mathbb{E}[\Delta ~~\theta~~^2]_t = \rho \mathbb{E}[\Delta ~~\theta~~^2]_{t-1} + (1 - \rho) (\Delta ~~\theta_t~~)^2</math>.		## Accumulate squared updates: <math>\mathbb{E}[\Delta x^2]_t = \rho \mathbb{E}[\Delta x^2]_{t-1} + (1 - \rho) (\Delta x_t)^2</math>.
	## Apply update: <math>~~\theta_~~{t+1} = ~~\theta_t~~ + \Delta ~~\theta_t~~</math>.		## Apply update: <math>x_{t+1} = x_t + \Delta x_t</math>.

	~~Assumptions~~:		This algorithm assumes:
	* The objective function is differentiable with respect to all parameters.		* The objective function is differentiable with respect to all parameters,
	* Parameters can be updated independently.		* Parameters can be updated independently,
	* ~~Exponential~~ decay provides a reasonable approximation of recent gradient history.		* The exponential decay provides a reasonable approximation of recent gradient history,
	* RMS values of <math>~~g_t~~</math> and <math>~~\Delta \theta_t~~</math> can serve as an approximation to the diagonal Hessian.		* The RMS values of <math>\Delta x</math> and <math>g</math> can serve as an approximation to the diagonal Hessian.

	== Numerical Example ==		== Numerical Example ==