Fall2024 Team4 Wiki: Minor changes to punctuation around reference numbers and removing a couple of unwanted symbols

2024-12-16T00:52:09Z

Minor changes to punctuation around reference numbers and removing a couple of unwanted symbols

← Older revision		Revision as of 20:52, 15 December 2024
Line 4:		Line 4:

	== Introduction ==		== Introduction ==
	Bayesian Optimization (BO) is a probabilistic model-based method for the optimization of objective functions that are computationally expensive and prevent thorough evaluation. It has found its place in optimization by handling objectives that are costly to compute and function as “black boxes” without an efficient mechanism to estimate their gradient. [1]		Bayesian Optimization (BO) is a probabilistic model-based method for the optimization of objective functions that are computationally expensive and prevent thorough evaluation. It has found its place in optimization by handling objectives that are costly to compute and function as “black boxes” without an efficient mechanism to estimate their gradient [1].

	The origins of BO trace back to the 1960s, with foundational work from H.J. Kushner and his introduction of a GP Model for noisy, one-dimensional optimization and leveraging Bayesian decision theory to derive optimization policies. V.R. Šaltenis expanded these ideas in the 1970s by introducing the Expected Improvement (EI) acquisition function and demonstrating its effectiveness through empirical studies on high-dimensional problems. An J. Mockus further developed the field by endorsing the one-step lookahead approach with the knowledge gradient, emphasizing its practical use in resource-constrained settings where it aligned with EI. [1]		The origins of BO trace back to the 1960s, with foundational work from H.J. Kushner and his introduction of a GP Model for noisy, one-dimensional optimization and leveraging Bayesian decision theory to derive optimization policies. V.R. Šaltenis expanded these ideas in the 1970s by introducing the Expected Improvement (EI) acquisition function and demonstrating its effectiveness through empirical studies on high-dimensional problems. An J. Mockus further developed the field by endorsing the one-step lookahead approach with the knowledge gradient, emphasizing its practical use in resource-constrained settings where it aligned with EI [1].

	BO has experienced a modern resurgence driven by advancements in machine learning, computational power, and the increasing demand to optimize complex systems across fields like artificial intelligence, robotics, and materials discovery, particularly excelling in scenarios with costly function evaluations, such as hyperparameter tuning, engineering design, and experimental sciences.		BO has experienced a modern resurgence driven by advancements in machine learning, computational power, and the increasing demand to optimize complex systems across fields like artificial intelligence, robotics, and materials discovery, particularly excelling in scenarios with costly function evaluations, such as hyperparameter tuning, engineering design, and experimental sciences.

	The significance of Bayesian Optimization lies in its ability to solve problems where traditional optimization techniques fail due to limited resources or the black-box nature of the objective function. For example, tuning the hyperparameters of a neural network involves computationally intensive evaluations, often requiring substantial time and resources. BO provides a systematic and efficient framework to minimize the number of evaluations while still identifying near-optimal solutions. [1]		The significance of Bayesian Optimization lies in its ability to solve problems where traditional optimization techniques fail due to limited resources or the black-box nature of the objective function. For example, tuning the hyperparameters of a neural network involves computationally intensive evaluations, often requiring substantial time and resources. BO provides a systematic and efficient framework to minimize the number of evaluations while still identifying near-optimal solutions [1].

	== Algorithm Discussion ==		== Algorithm Discussion ==
Line 62:		Line 62:
	At a new point ''x*'', given the observations, a posterior probability distribution can be derived. By Bayes’ rule, it can be shown that [4]		At a new point ''x*'', given the observations, a posterior probability distribution can be derived. By Bayes’ rule, it can be shown that [4]

	<math display="block">f(x^) \mid f(x_1,\ldots,x_k) \sim N(\mu_k(x^),\sigma_2^k), where </math>		<math display="block">f(x^) \mid f(x_1,\ldots,x_k) \sim N(\mu_k(x^),\sigma_2^k), where </math><math display="block">\mu_k(x^) = \Sigma_0(x^{},\,x_1,\ldots,x_k) \Sigma_0(x_1,\ldots,x_k, x_1,\,\ldots,x_k)^{-1}(f(x_1,\ldots,x_k) - \mu_0(x_1,\ldots,x_k)) + \mu_0(x^{}) </math><math display="block">\sigma_k^2(x^) = \Sigma_0(x^{},\,x^{})-\Sigma_0(x^, x_1,\,\ldots,x_k)\Sigma_0(x_1,\ldots,x_k,\,x_1,\ldots,x_k)^{-1} \Sigma_0(x_1,\,\ldots,x_k,x^) </math>

	<math display="block">\mu_k(x^) = \Sigma_0(x^{},\,x_1,\ldots,x_k) \Sigma_0(x_1,\ldots,x_k, x_1,\,\ldots,x_k)^{-1}(f(x_1,\ldots,x_k) - \mu_0(x_1,\ldots,x_k)) + \mu_0(x^{*}) </math>

	<math display="block">\sigma_k^2(x^) = \Sigma_0(x^{},\,x^{})-\Sigma_0(x^, x_1,\,\ldots,x_k)\Sigma_0(x_1,\ldots,x_k,\,x_1,\ldots,x_k)^{-1} \Sigma_0(x_1,\,\ldots,x_k,x^*) </math>

	For the mean and the covariance function, <math>\mu_0		For the mean and the covariance function, <math>\mu_0
Line 137:		Line 133:

	== Application ==		== Application ==
	BO has become vital to optimization of functions in a variety of fields. It is the most useful in scenarios where each iteration of the evaluation is costly in terms of time, resources, or computation. The most prominent application of BO comes in tuning hyperparameters of machine learning models. The problem lies in the high-dimensional and non-convex nature of hyperparameters such as learning rates and regularization coefficients, most commonly. Their values can drastically change the model, so it is important to optimize them appropriately. Snoek et al. (2012) demonstrated the effectiveness of BO in tuning neural network hyperparameters that resulted in state-of-the-art performance on benchmark datasets. Several hyperparameters were targeted including learning rate, momentum, weight decay, number of layers, number of units per layer, and other parameters specific to the architecture of the models. GPs were used to model the validation error as a function of the hyperparameters selected, and the EI criterion was used to select the next set of hyperparameters. BO found hyperparameter combinations that gave lower validation errors compared to traditional tuning techniques and had fewer function evaluations, reducing time complexity. [8]		BO has become vital to optimization of functions in a variety of fields. It is the most useful in scenarios where each iteration of the evaluation is costly in terms of time, resources, or computation. The most prominent application of BO comes in tuning hyperparameters of machine learning models. The problem lies in the high-dimensional and non-convex nature of hyperparameters such as learning rates and regularization coefficients, most commonly. Their values can drastically change the model, so it is important to optimize them appropriately. Snoek et al. (2012) demonstrated the effectiveness of BO in tuning neural network hyperparameters that resulted in state-of-the-art performance on benchmark datasets. Several hyperparameters were targeted including learning rate, momentum, weight decay, number of layers, number of units per layer, and other parameters specific to the architecture of the models. GPs were used to model the validation error as a function of the hyperparameters selected, and the EI criterion was used to select the next set of hyperparameters. BO found hyperparameter combinations that gave lower validation errors compared to traditional tuning techniques and had fewer function evaluations, reducing time complexity [8].

	Another major area of application is the optimization of engineering design where BO aids in optimizing design parameters that would otherwise require costly simulations or physical experiments. This is applicable to all forms of engineering where design is crucial such as aerospace, automotive design, and structural engineering. According to Edgar et al. (2001), optimization plays a crucial role in enhancing the efficiency and safety of chemical processes by systematically adjusting design parameters. Edgar et al. (2001) present a case study on the optimization of distillation column design. The basic design parameters are the number of stages, reflux ratio, feed stage location, and column diameter. The objective function is the total annual cost, and while the complex and nonlinear relationship between the parameters and the function poses a challenge, BO is well-suited to handle such a problem. By fitting a GP to initial design points, and selecting an appropriate acquisition function, optimal hyperparameters can be determined through iteration. [9]		Another major area of application is the optimization of engineering design where BO aids in optimizing design parameters that would otherwise require costly simulations or physical experiments. This is applicable to all forms of engineering where design is crucial such as aerospace, automotive design, and structural engineering. According to Edgar et al. (2001), optimization plays a crucial role in enhancing the efficiency and safety of chemical processes by systematically adjusting design parameters. Edgar et al. (2001) present a case study on the optimization of distillation column design. The basic design parameters are the number of stages, reflux ratio, feed stage location, and column diameter. The objective function is the total annual cost, and while the complex and nonlinear relationship between the parameters and the function poses a challenge, BO is well-suited to handle such a problem. By fitting a GP to initial design points, and selecting an appropriate acquisition function, optimal hyperparameters can be determined through iteration [9].

	Similarly, in fields like chemistry and material science, experiments can be costly and time consuming. In this case, Bayesian Optimization can be utilized by selecting apt experimental conditions to conduct experiments under which are analogous to selecting hyperparameters in machine learning models. For instance, Gomez-Bombarelli et al (2018) proposed using BO to design novel organic molecules with specific electronic properties by altering and optimizing molecular structures. In this case, the molecular structure, the functional groups, and the bond angles and lengths were chosen as the parameters. Since electronic properties are nonlinearly dependent on molecular structures, and quantum mechanical calculations are computationally intensive, BO is appropriate. By defining a particular electronic property as the objective function, initializing a surrogate model, and selecting an acquisition function, the parameters can be found through iterative optimization. [10]		Similarly, in fields like chemistry and material science, experiments can be costly and time consuming. In this case, Bayesian Optimization can be utilized by selecting apt experimental conditions to conduct experiments under which are analogous to selecting hyperparameters in machine learning models. For instance, Gomez-Bombarelli et al (2018) proposed using BO to design novel organic molecules with specific electronic properties by altering and optimizing molecular structures. In this case, the molecular structure, the functional groups, and the bond angles and lengths were chosen as the parameters. Since electronic properties are nonlinearly dependent on molecular structures, and quantum mechanical calculations are computationally intensive, BO is appropriate. By defining a particular electronic property as the objective function, initializing a surrogate model, and selecting an acquisition function, the parameters can be found through iterative optimization [10].

	== Conclusion ==		== Conclusion ==

Fall2024 Team4 Wiki: Made some grammar changes and renumbered the references

2024-12-15T23:12:51Z

Made some grammar changes and renumbered the references

Show changes

Fall2024 Team4 Wiki: Inserted text from the initial deadline into the Bayesian Optimization Wiki. This text includes improvements suggested by the TAs.

2024-12-15T16:40:17Z

Inserted text from the initial deadline into the Bayesian Optimization Wiki. This text includes improvements suggested by the TAs.

Show changes

SYSEN5800TAs at 18:35, 10 December 2024

2024-12-10T18:35:00Z

← Older revision		Revision as of 14:35, 10 December 2024
Line 1:		Line 1:
	Author: Rithvik Duggireddy, Brad Edgerton, Sochetra Sor, Jerry Feng, James Chen (ChemE 6800 Fall 2024)		Author: Rithvik Duggireddy (rd532), Brad Edgerton (bme44), Sochetra Sor (ss3437), Jerry Feng (zf245), James Chen (jzc28) (ChemE 6800 Fall 2024)

	Stewards: Nathan Preuss, Wei-Han Chen, Tianqi Xiao, Guoqing Hu		Stewards: Nathan Preuss, Wei-Han Chen, Tianqi Xiao, Guoqing Hu

SYSEN5800TAs at 18:24, 10 December 2024

2024-12-10T18:24:01Z

← Older revision		Revision as of 14:24, 10 December 2024
Line 1:		Line 1:
	Fall 2024 ~~new page creation~~		Author: Rithvik Duggireddy, Brad Edgerton, Sochetra Sor, Jerry Feng, James Chen (ChemE 6800 Fall 2024)

			Stewards: Nathan Preuss, Wei-Han Chen, Tianqi Xiao, Guoqing Hu

Youfengqi: Created page with "Fall 2024 new page creation"

2024-12-04T20:46:34Z

Created page with "Fall 2024 new page creation"

New page