Adafactor - Revision history

Fall2024 Wiki Team6: /* Numerical Examples */

2024-12-16T02:46:28Z

Numerical Examples

← Older revision		Revision as of 22:46, 15 December 2024
Line 313:		Line 313:
	When training ResNet50 and ViT on the ImageNet1k dataset, Adafactor successfully optimized these deep networks with its low memory requirements. Additionally, with new algorithms combining preconditioned diagonalization methods (e.g., AdafacDiag and AdafacDiag++), it outperformed the standard Adam optimizer in both convergence speed and final accuracy.<sup>6</sup>		When training ResNet50 and ViT on the ImageNet1k dataset, Adafactor successfully optimized these deep networks with its low memory requirements. Additionally, with new algorithms combining preconditioned diagonalization methods (e.g., AdafacDiag and AdafacDiag++), it outperformed the standard Adam optimizer in both convergence speed and final accuracy.<sup>6</sup>

	==== ~~'''~~Software Tools and Platforms~~'''~~ ====		==== Software Tools and Platforms ====
	Adafactor has been integrated into the following mainstream deep learning frameworks, making it accessible to developers:		Adafactor has been integrated into the following mainstream deep learning frameworks, making it accessible to developers:

Line 322:		Line 322:
	'''JAX/Flax:''' JAX provides an optimizer library called Optax, which includes the Adafactor optimizer.<sup>9</sup>		'''JAX/Flax:''' JAX provides an optimizer library called Optax, which includes the Adafactor optimizer.<sup>9</sup>

	==== ~~'''~~Future Prospects~~'''~~ ====		==== Future Prospects ====
	As the scale of deep learning models continues to grow, Adafactor’s memory-saving and computational efficiency advantages will become increasingly important. In the training of ultra-large-scale models (e.g., GPT and Vision Transformers), Adafactor is expected to become an indispensable optimization tool. Furthermore, by combining with other optimization strategies, such as mixed precision training, Adafactor may further enhance its applicability in both industrial and research settings.		As the scale of deep learning models continues to grow, Adafactor’s memory-saving and computational efficiency advantages will become increasingly important. In the training of ultra-large-scale models (e.g., GPT and Vision Transformers), Adafactor is expected to become an indispensable optimization tool. Furthermore, by combining with other optimization strategies, such as mixed precision training, Adafactor may further enhance its applicability in both industrial and research settings.

Fall2024 Wiki Team6: /* Step 2: Compute G t 2 {\displaystyle G_{t}^{2}} (Element-wise Square of Gradient) */

2024-12-16T02:43:31Z

Step 2: Compute G t 2 {\displaystyle G_{t}^{2}} (Element-wise Square of Gradient)

← Older revision		Revision as of 22:43, 15 December 2024
Line 173:		Line 173:
	<math>\alpha_1 = max(0.001,0.806)\cdot 0.01=0.00806</math>		<math>\alpha_1 = max(0.001,0.806)\cdot 0.01=0.00806</math>

	=== Step 2: Compute ~~<math>G^{2}_t</math> (Element~~-wise Square of Gradient) ===		=== Step 2: Compute the element-wise (Square of Gradient) ===
	'''Step 2.1: Compute the gradient of the loss function'''		'''Step 2.1: Compute the gradient of the loss function'''

Line 183:		Line 183:

	<math>G_1 = \begin{bmatrix} 0.7 &-0.5& 0.9\\ -1.1 & 0.8& -1.6\\1.2&-0.7& 0.4 \end{bmatrix} - \begin{bmatrix} 0.4 & -0.3 &0.5 \\ -0.6 & 0.2&-1.5\\1.0&-0.3&0.1 \end{bmatrix}</math>		<math>G_1 = \begin{bmatrix} 0.7 &-0.5& 0.9\\ -1.1 & 0.8& -1.6\\1.2&-0.7& 0.4 \end{bmatrix} - \begin{bmatrix} 0.4 & -0.3 &0.5 \\ -0.6 & 0.2&-1.5\\1.0&-0.3&0.1 \end{bmatrix}</math>



	<math>G_1 = \begin{bmatrix} 0.3&-0.2&0.4\\ -0.5&0.6&-0.1\\0.2&-0.4 &0.3 \end{bmatrix}</math>		<math>G_1 = \begin{bmatrix} 0.3&-0.2&0.4\\ -0.5&0.6&-0.1\\0.2&-0.4 &0.3 \end{bmatrix}</math>



Line 235:		Line 237:
	<math>\hat{V}_1 = \begin{bmatrix} 0.0122&0.0180&0.0084\\ 0.0262&0.0386&0.0179\\ 0.0122&0.0180&0.0084\end{bmatrix} </math>		<math>\hat{V}_1 = \begin{bmatrix} 0.0122&0.0180&0.0084\\ 0.0262&0.0386&0.0179\\ 0.0122&0.0180&0.0084\end{bmatrix} </math>

	=== Step 4: Update the vector ~~(<math>U_t </math>)~~ ===		=== Step 4: Update the vector ===
	Computed by scaling the gradient matrix '''<math>G_t</math>''' element-wise with the inverse square root of the second moment estimate (<math>\hat{V_t}</math>)		Computed by scaling the gradient matrix '''<math>G_t</math>''' element-wise with the inverse square root of the second moment estimate (<math>\hat{V_t}</math>)

Line 272:		Line 274:


	=== Step 5: Weight Update ~~(<math>X_1 </math>)~~ ===		=== Step 5: Weight Update ===
	Adjust the weights (<math>X_t </math>) by subtracting the product of the learning rate (<math>\alpha_t </math>) and the clipped update vector (<math>\hat{U_t} </math> ).		Adjust the weights (<math>X_t </math>) by subtracting the product of the learning rate (<math>\alpha_t </math>) and the clipped update vector (<math>\hat{U_t} </math> ).

Fall2024 Wiki Team6: /* Problem setup */

2024-12-16T02:02:27Z

Problem setup

← Older revision		Revision as of 22:02, 15 December 2024
Line 121:		Line 121:

	=== Problem setup ===		=== Problem setup ===
			'''Minimize the loss function:'''

			<math>f(X) = \frac{1}{2}\sum_{i,j}(X_{ij}-C_{ij})^2</math>

	'''Initial weights ('''<math>X_0</math>'''):'''		'''Initial weights ('''<math>X_0</math>'''):'''

	<math>X_0 = \begin{bmatrix} 0.7 &-0.5& 0.9\\ -1.1 & 0.8& -1.6\\1.2&-0.7& 0.4 \end{bmatrix}</math>		<math>X_0 = \begin{bmatrix} 0.7 &-0.5& 0.9\\ -1.1 & 0.8& -1.6\\1.2&-0.7& 0.4 \end{bmatrix}</math>

	'''~~Gradient for first iteration~~ (<math>~~G_1~~</math>):'''		'''Target matrix (<math>C</math>)：'''

	~~Gradient of the loss function with respect to X~~

	<math>~~G_1~~ = \begin{bmatrix} 0.3&-0.2&0.4\\ -0.5&0.6&-0.1\\0.2&-0.4 &0.3 \end{bmatrix}</math>		<math>C = \begin{bmatrix} 0.4 & -0.3 &0.5 \\ -0.6 & 0.2&-1.5\\1.0&-0.3&0.1 \end{bmatrix}</math>

	=== Hyperparameters setup ===		=== Hyperparameters setup ===
Line 172:		Line 174:

	=== Step 2: Compute <math>G^{2}_t</math> (Element-wise Square of Gradient) ===		=== Step 2: Compute <math>G^{2}_t</math> (Element-wise Square of Gradient) ===
	Compute the ~~squared value~~ of ~~each element in~~ the ~~gradient matrix~~ '''<math>G_t</math>~~'''~~.		'''Step 2.1: Compute the gradient of the loss function'''

			Gradient formula

			<math>G_t = {\partial f(X)\over\partial X} = X_{t-1} - C</math>

			Subtract C from <math>X_0</math>

			<math>G_1 = \begin{bmatrix} 0.7 &-0.5& 0.9\\ -1.1 & 0.8& -1.6\\1.2&-0.7& 0.4 \end{bmatrix} - \begin{bmatrix} 0.4 & -0.3 &0.5 \\ -0.6 & 0.2&-1.5\\1.0&-0.3&0.1 \end{bmatrix}</math>


			<math>G_1 = \begin{bmatrix} 0.3&-0.2&0.4\\ -0.5&0.6&-0.1\\0.2&-0.4 &0.3 \end{bmatrix}</math>

	~~<math>G^{2}_1 = \begin{bmatrix} 0.3^2&(-0.2)^2&0.4^2\\ (-0.5)^2&0.6^2&(-0.1)^2\\0.2^2&(-0.4)^2 &0.3^2 \end{bmatrix}</math>~~

			'''Step 2.2: Compute the squared value of each element in the gradient matrix <math>G_t</math>'''

			<math>G^{2}_1 = \begin{bmatrix} 0.3^2&(-0.2)^2&0.4^2\\ (-0.5)^2&0.6^2&(-0.1)^2\\0.2^2&(-0.4)^2 &0.3^2 \end{bmatrix}</math>

	<math>G^{2}_1 = \begin{bmatrix} 0.09& 0.04&0.16\\ 0.25&0.36&0.01\\0.04&0.16&0.09\end{bmatrix}</math>		<math>G^{2}_1 = \begin{bmatrix} 0.09& 0.04&0.16\\ 0.25&0.36&0.01\\0.04&0.16&0.09\end{bmatrix}</math>

Fall2024 Wiki Team6: /* Numerical Examples */

2024-12-15T03:12:42Z

Numerical Examples

← Older revision		Revision as of 23:12, 14 December 2024
Line 157:		Line 157:
	Substitute the initial weights		Substitute the initial weights

	<math>RMS(X_0) = \sqrt{\tfrac{1}{9}(0.72^2+(-0.5)^2+0.9^2+(-1.1)^2+0.8^2+(-0.6)^2+1.2^2+(-0.7)^2+0.4^2)}</math>		<math>RMS(X_0) = \sqrt{\tfrac{1}{9}(0.7^2+(-0.5)^2+0.9^2+(-1.1)^2+0.8^2+(-0.6)^2+1.2^2+(-0.7)^2+0.4^2)}</math>

	<math>RMS(X_0) = \sqrt{\frac{6.85}{9}}\approx 0.806</math>		<math>RMS(X_0) = \sqrt{\frac{6.85}{9}}\approx 0.806</math>
Line 175:		Line 175:

	<math>G^{2}_1 = \begin{bmatrix} 0.3^2&(-0.2)^2&0.4^2\\ (-0.5)^2&0.6^2&(-0.1)^2\\0.2^2&(-0.4)^2 &0.3^2 \end{bmatrix}</math>		<math>G^{2}_1 = \begin{bmatrix} 0.3^2&(-0.2)^2&0.4^2\\ (-0.5)^2&0.6^2&(-0.1)^2\\0.2^2&(-0.4)^2 &0.3^2 \end{bmatrix}</math>

Fall2024 Wiki Team6 at 03:06, 15 December 2024

2024-12-15T03:06:59Z

← Older revision		Revision as of 23:06, 14 December 2024
Line 260:		Line 260:


	=== ~~'''<big>~~Step 5: Weight Update (~~</big>'''~~<math>X_1 </math~~>'''<big~~>)~~</big>'''~~ ===		=== Step 5: Weight Update (<math>X_1 </math>) ===
	Adjust the weights (<math>X_t </math>) by subtracting the product of the learning rate (<math>\alpha_t </math>) and the clipped update vector (<math>\hat{U_t} </math> ).		Adjust the weights (<math>X_t </math>) by subtracting the product of the learning rate (<math>\alpha_t </math>) and the clipped update vector (<math>\hat{U_t} </math> ).

Fall2024 Wiki Team6 at 03:04, 15 December 2024

2024-12-15T03:04:30Z

← Older revision		Revision as of 23:04, 14 December 2024
Line 120:		Line 120:
	Step-by-step instructions for determining the result of the first iteration.		Step-by-step instructions for determining the result of the first iteration.

	~~'''<big>~~Problem setup~~</big>'''~~		=== Problem setup ===

	'''Initial weights ('''<math>X_0</math>'''):'''		'''Initial weights ('''<math>X_0</math>'''):'''

Line 132:		Line 131:
	<math>G_1 = \begin{bmatrix} 0.3&-0.2&0.4\\ -0.5&0.6&-0.1\\0.2&-0.4 &0.3 \end{bmatrix}</math>		<math>G_1 = \begin{bmatrix} 0.3&-0.2&0.4\\ -0.5&0.6&-0.1\\0.2&-0.4 &0.3 \end{bmatrix}</math>

	~~'''<big>~~Hyperparameters setup~~</big>'''~~		=== Hyperparameters setup ===

	<math>\epsilon_1 = 10^{-30}</math> (Minimum learning rate scaling factor))		<math>\epsilon_1 = 10^{-30}</math> (Minimum learning rate scaling factor))

Line 144:		Line 142:
	<math>\hat{\beta}_{2t} = 1 - t^{-0.8}</math> (Second moment decay)		<math>\hat{\beta}_{2t} = 1 - t^{-0.8}</math> (Second moment decay)

	~~'''<big>~~Step 1: Learning Rate Scaling~~</big>'''~~		=== Step 1: Learning Rate Scaling ===

	Define the relative step size		Define the relative step size

Line 174:		Line 171:
	<math>\alpha_1 = max(0.001,0.806)\cdot 0.01=0.00806</math>		<math>\alpha_1 = max(0.001,0.806)\cdot 0.01=0.00806</math>

			=== Step 2: Compute <math>G^{2}_t</math> (Element-wise Square of Gradient) ===

	~~'''<big>~~Step 2: Compute <math>G^{2}_t</math> (Element-wise Square of Gradient)~~</big>'''~~

	Compute the squared value of each element in the gradient matrix '''<math>G_t</math>'''.		Compute the squared value of each element in the gradient matrix '''<math>G_t</math>'''.

	<math>G^{2}_1 = \begin{bmatrix} 0.3^2&(-0.2)^2&0.4^2\\ (-0.5)^2&0.6^2&(-0.1)^2\\0.2^2&(-0.4)^2 &0.3^2 \end{bmatrix}</math>		<math>G^{2}_1 = \begin{bmatrix} 0.3^2&(-0.2)^2&0.4^2\\ (-0.5)^2&0.6^2&(-0.1)^2\\0.2^2&(-0.4)^2 &0.3^2 \end{bmatrix}</math>



Line 187:		Line 182:
	<math>G^{2}_1 = \begin{bmatrix} 0.09& 0.04&0.16\\ 0.25&0.36&0.01\\0.04&0.16&0.09\end{bmatrix}</math>		<math>G^{2}_1 = \begin{bmatrix} 0.09& 0.04&0.16\\ 0.25&0.36&0.01\\0.04&0.16&0.09\end{bmatrix}</math>

	~~'''<big>~~Step 3: Find the moment estimate~~</big>'''~~		=== Step 3: Find the moment estimate ===

	Compute the exponential moving average of squared gradients to capture the variance or scale of gradients.		Compute the exponential moving average of squared gradients to capture the variance or scale of gradients.

Line 229:		Line 223:
	<math>\hat{V}_1 = \begin{bmatrix} 0.0122&0.0180&0.0084\\ 0.0262&0.0386&0.0179\\ 0.0122&0.0180&0.0084\end{bmatrix} </math>		<math>\hat{V}_1 = \begin{bmatrix} 0.0122&0.0180&0.0084\\ 0.0262&0.0386&0.0179\\ 0.0122&0.0180&0.0084\end{bmatrix} </math>

			=== Step 4: Update the vector (<math>U_t </math>) ===
	~~'''<big>~~Step 4: Update the vector (<math>U_t </math>)~~</big>'''~~

	Computed by scaling the gradient matrix '''<math>G_t</math>''' element-wise with the inverse square root of the second moment estimate (<math>\hat{V_t}</math>)		Computed by scaling the gradient matrix '''<math>G_t</math>''' element-wise with the inverse square root of the second moment estimate (<math>\hat{V_t}</math>)

Line 243:		Line 235:

	<math>U_1 = \frac{\begin{bmatrix}0.3&-0.2&0.4 \\ -0.5&0.6&-0.1\\0.2&-0.4&0.3 \end{bmatrix}}{\sqrt{\begin{bmatrix} 0.0122&0.0180&0.0084\\ 0.0262&0.0386&0.0179\\0.0122&0.0180&0.0084 \end{bmatrix}}} </math>		<math>U_1 = \frac{\begin{bmatrix}0.3&-0.2&0.4 \\ -0.5&0.6&-0.1\\0.2&-0.4&0.3 \end{bmatrix}}{\sqrt{\begin{bmatrix} 0.0122&0.0180&0.0084\\ 0.0262&0.0386&0.0179\\0.0122&0.0180&0.0084 \end{bmatrix}}} </math>



Line 267:		Line 260:


	'''<big>Step 5: Weight Update (</big>'''<math>X_1 </math>'''<big>)</big>'''		=== '''<big>Step 5: Weight Update (</big>'''<math>X_1 </math>'''<big>)</big>''' ===

	Adjust the weights (<math>X_t </math>) by subtracting the product of the learning rate (<math>\alpha_t </math>) and the clipped update vector (<math>\hat{U_t} </math> ).		Adjust the weights (<math>X_t </math>) by subtracting the product of the learning rate (<math>\alpha_t </math>) and the clipped update vector (<math>\hat{U_t} </math> ).

Fall2024 Wiki Team6: /* Numerical Examples */

2024-12-15T02:43:25Z

Numerical Examples

Fall2024 Wiki Team6: /* Software Tools and Platforms */

2024-12-13T21:54:00Z

Software Tools and Platforms

← Older revision		Revision as of 17:54, 13 December 2024
Line 303:		Line 303:
	When training ResNet50 and ViT on the ImageNet1k dataset, Adafactor successfully optimized these deep networks with its low memory requirements. Additionally, with new algorithms combining preconditioned diagonalization methods (e.g., AdafacDiag and AdafacDiag++), it outperformed the standard Adam optimizer in both convergence speed and final accuracy.<sup>6</sup>		When training ResNet50 and ViT on the ImageNet1k dataset, Adafactor successfully optimized these deep networks with its low memory requirements. Additionally, with new algorithms combining preconditioned diagonalization methods (e.g., AdafacDiag and AdafacDiag++), it outperformed the standard Adam optimizer in both convergence speed and final accuracy.<sup>6</sup>

	=== '''Software Tools and Platforms''' ===		==== '''Software Tools and Platforms''' ====
	Adafactor has been integrated into the following mainstream deep learning frameworks, making it accessible to developers:		Adafactor has been integrated into the following mainstream deep learning frameworks, making it accessible to developers:

Line 312:		Line 312:
	'''JAX/Flax:''' JAX provides an optimizer library called Optax, which includes the Adafactor optimizer.<sup>9</sup>		'''JAX/Flax:''' JAX provides an optimizer library called Optax, which includes the Adafactor optimizer.<sup>9</sup>

	=== '''Future Prospects''' ===		==== '''Future Prospects''' ====
	As the scale of deep learning models continues to grow, Adafactor’s memory-saving and computational efficiency advantages will become increasingly important. In the training of ultra-large-scale models (e.g., GPT and Vision Transformers), Adafactor is expected to become an indispensable optimization tool. Furthermore, by combining with other optimization strategies, such as mixed precision training, Adafactor may further enhance its applicability in both industrial and research settings.		As the scale of deep learning models continues to grow, Adafactor’s memory-saving and computational efficiency advantages will become increasingly important. In the training of ultra-large-scale models (e.g., GPT and Vision Transformers), Adafactor is expected to become an indispensable optimization tool. Furthermore, by combining with other optimization strategies, such as mixed precision training, Adafactor may further enhance its applicability in both industrial and research settings.

Fall2024 Wiki Team6: /* Conclusion */

2024-12-13T21:51:26Z

Conclusion

← Older revision		Revision as of 17:51, 13 December 2024
Line 316:		Line 316:

	== Conclusion ==		== Conclusion ==
			Adafactor addresses the memory consumption challenge of training large-scale deep learning models. By factorizing the second-order moment matrix and dynamically adjusting the learning rate, Adafactor minimizes resource usage without compromising performance. Adafactor can be applied to the training tasks of large language models such as Transformers, T5 models, and Vision Transformers.

	== Reference ==		== Reference ==

			# Shazeer, Noam, and Mitchell Stern. "Adafactor: Adaptive learning rates with sublinear memory cost." ''International Conference on Machine Learning''. PMLR, 2018.
			# Raffel, Colin, et al. "Exploring the limits of transfer learning with a unified text-to-text transformer." ''Journal of machine learning research'' 21.140 (2020): 1-67.
			# DIAGONALIZATION, VIA PRECONDITIONER. "Improving Adaptive Moment Optimization via Preconditioner Diagonalization."
			# Chauhan, Tavishee, and Hemant Palivela. "The Fine tuning of Language models for automation of Humor Detection." ''INFOCOMP Journal of Computer Science'' 20.2 (2021).
			# Lepikhin, Dmitry, et al. "Gshard: Scaling giant models with conditional computation and automatic sharding." ''arXiv preprint arXiv:2006.16668'' (2020).
			# DIAGONALIZATION, VIA PRECONDITIONER. "Improving Adaptive Moment Optimization via Preconditioner Diagonalization."
			# <nowiki>https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adafactor</nowiki>
			# <nowiki>https://pytorch.org/docs/stable/generated/torch.optim.Adafactor.html</nowiki>
			# <nowiki>https://flax.readthedocs.io/en/v0.5.3/_autosummary/flax.optim.Adafactor.html</nowiki>

Fall2024 Wiki Team6: /* Applications */ change tensorflow

2024-12-13T21:48:11Z

Applications: change tensorflow

← Older revision		Revision as of 17:48, 13 December 2024
Line 306:		Line 306:
	Adafactor has been integrated into the following mainstream deep learning frameworks, making it accessible to developers:		Adafactor has been integrated into the following mainstream deep learning frameworks, making it accessible to developers:

	'''TensorFlow''': Provides a built-in implementation of Adafactor~~, supporting T5 model optimization~~.<sup>7</sup>		'''TensorFlow''': Provides a built-in implementation of Adafactor.<sup>7</sup>

	'''PyTorch:''' PyTorch provides the Adafactor optimizer through the torch.optim.AdaFactor class.<sup>8</sup>		'''PyTorch:''' PyTorch provides the Adafactor optimizer through the torch.optim.AdaFactor class.<sup>8</sup>

← Older revision		Revision as of 22:43, 14 December 2024
Line 173:		Line 173:

	<math>\alpha_1 = max(0.001,0.806)\cdot 0.01=0.00806</math>		<math>\alpha_1 = max(0.001,0.806)\cdot 0.01=0.00806</math>



Line 227:		Line 228:

	<math>\hat{V}_1 = \begin{bmatrix} 0.0122&0.0180&0.0084\\ 0.0262&0.0386&0.0179\\ 0.0122&0.0180&0.0084\end{bmatrix} </math>		<math>\hat{V}_1 = \begin{bmatrix} 0.0122&0.0180&0.0084\\ 0.0262&0.0386&0.0179\\ 0.0122&0.0180&0.0084\end{bmatrix} </math>


	'''<big>Step 4: Update the vector (<math>U_t </math>)</big>'''		'''<big>Step 4: Update the vector (<math>U_t </math>)</big>'''
Line 232:		Line 234:
	Computed by scaling the gradient matrix '''<math>G_t</math>''' element-wise with the inverse square root of the second moment estimate (<math>\hat{V_t}</math>)		Computed by scaling the gradient matrix '''<math>G_t</math>''' element-wise with the inverse square root of the second moment estimate (<math>\hat{V_t}</math>)

	'''~~step~~ 4.1: Find the vector value of <math>U_t </math>'''		'''Step 4.1: Find the vector value of <math>U_t </math>'''

	Formula of '''<small><math>U_t </math></small>'''		Formula of '''<small><math>U_t </math></small>'''
Line 241:		Line 243:

	<math>U_1 = \frac{\begin{bmatrix}0.3&-0.2&0.4 \\ -0.5&0.6&-0.1\\0.2&-0.4&0.3 \end{bmatrix}}{\sqrt{\begin{bmatrix} 0.0122&0.0180&0.0084\\ 0.0262&0.0386&0.0179\\0.0122&0.0180&0.0084 \end{bmatrix}}} </math>		<math>U_1 = \frac{\begin{bmatrix}0.3&-0.2&0.4 \\ -0.5&0.6&-0.1\\0.2&-0.4&0.3 \end{bmatrix}}{\sqrt{\begin{bmatrix} 0.0122&0.0180&0.0084\\ 0.0262&0.0386&0.0179\\0.0122&0.0180&0.0084 \end{bmatrix}}} </math>



Line 246:		Line 249:
	<math>U_1 = \begin{bmatrix} 2.711&-1.489&4.370\\-3.090&3.055&-0.747\\1.807&-2.978&3.278 \end{bmatrix} </math>		<math>U_1 = \begin{bmatrix} 2.711&-1.489&4.370\\-3.090&3.055&-0.747\\1.807&-2.978&3.278 \end{bmatrix} </math>

	'''~~step~~ 4.2: Clipped Update Vector <math>\hat{U_t} </math>'''		'''Step 4.2: Clipped Update Vector <math>\hat{U_t} </math>'''

	Scale the update vector ( '''<math>U_t </math>''' ) to ensure its RMS value does not exceed a predefined clipping threshold (<math>d </math>), maintaining stability in updates.		Scale the update vector ( '''<math>U_t </math>''' ) to ensure its RMS value does not exceed a predefined clipping threshold (<math>d </math>), maintaining stability in updates.
Line 261:		Line 264:

	'''<math>\hat{U_1} = \begin{bmatrix} 0.965&-0.53&1.556 \\-1.1&1.088&-0.266\\0.664&-1.06&1.167 \end{bmatrix} </math>'''		'''<math>\hat{U_1} = \begin{bmatrix} 0.965&-0.53&1.556 \\-1.1&1.088&-0.266\\0.664&-1.06&1.167 \end{bmatrix} </math>'''

Adafactor - Revision history

Fall2024 Wiki Team6: /* Numerical Examples */

Fall2024 Wiki Team6: /* Step 2: Compute G t 2 {\displaystyle G_{t}^{2}} ​ (Element-wise Square of Gradient) */

Fall2024 Wiki Team6: /* Problem setup */

Fall2024 Wiki Team6: /* Numerical Examples */

Fall2024 Wiki Team6 at 03:06, 15 December 2024

Fall2024 Wiki Team6 at 03:04, 15 December 2024

Fall2024 Wiki Team6: /* Numerical Examples */

Fall2024 Wiki Team6: /* Software Tools and Platforms */

Fall2024 Wiki Team6: /* Conclusion */

Fall2024 Wiki Team6: /* Applications */ change tensorflow

Fall2024 Wiki Team6: /* Step 2: Compute G t 2 {\displaystyle G_{t}^{2}} (Element-wise Square of Gradient) */