Cornell University Computational Optimization Open Textbook - Optimization Wiki - User contributions [en]

Differential evolution

2024-12-16T03:15:59Z

Fall2024 Wiki Team26: /* Visualization */

Author: Rohit Kumar (rk787) (ChemE 6800 Fall 2024)

Stewards: Nathan Preuss, Wei-Han Chen, Tianqi Xiao, Guoqing Hu

== Introduction ==
Differential Evolution (DE) is a robust and efficient optimization algorithm widely used for solving non-linear, non-differentiable, and multimodal optimization problems. It was first introduced by R. Storn and K. Price in their seminal work on global optimization over continuous spaces <ref>R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.</ref>, where they described DE as:-

''"A new heuristic approach for minimizing possibly nonlinear and non-differentiable continuous-space functions."''

A heuristic is any approximation method for a solution to a problem capable of producing satisfactory solutions within tolerable amounts of time when classic methods cannot find an exact answer or work quite slowly. Unlike the Gradient Descent algorithm, the optimization problem does not need to be differentiable.

The key idea is to generate a population of candidate solutions through successive generations using vector differences for deviations. This approach allows DE to adapt its search methodology to the objective function, making it particularly effective for non-linear, non-differentiable, and multimodal optimization problems.

The algorithm is renowned for its simplicity and efficiency. Its primary advantages include:

* Requires minimal parameter tuning.
* High robustness.
* Ability to avoid local optima and converge toward global solutions.

In DE, the currently evolving population of possible solutions is represented as a vector of real-valued numbers. The algorithm attempts to enhance these solutions through a series of iterations built upon three main steps:

* Mutation
* Crossover
* Selection

DE has become highly popular for applications in such fields as engineering design, machine learning, signal processing, and financial modeling. Despite its simplicity, Differential Evolution is competitive against many other optimization techniques, often performing better on challenging problems.

== Algorithm Discussion ==
Differential Evolution operates through three main steps: initialization, mutation, crossover, and selection, as outlined in optimization literature <ref>J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>.

=== Initialization ===
The initial population is generated randomly within the bounds of the search space: <math>x_{i,j} \sim U(x_j^{\text{min}}, x_j^{\text{max}})</math> where:

* <math>x_{i,j}</math> represents the <math>j^{\text{th}}</math> parameter of the <math>i^{\text{th}}</math> individual.
* <math>U(a, b)</math> is a uniform random distribution.

=== Mutation ===
A mutant vector <math>v_i</math> is generated by combining three randomly chosen vectors: <math>v_i = x_{r1} + F \cdot (x_{r2} - x_{r3})</math> where:

* <math>F</math> is the mutation factor.
* <math>x_{r1}, x_{r2}, x_{r3}</math> are distinct vectors from the population.

=== Crossover ===
A trial vector <math>u_i</math> is created by mixing the mutant vector <math>v_i</math> and the target vector <math>x_i</math>: <math>
u_{i,j} =
\begin{cases}
v_{i,j}, & \text{if } \text{rand}(0, 1) \leq CR \text{ or } j = j_{\text{rand}}, \\
x_{i,j}, & \text{otherwise}.
\end{cases}
</math> where:

* <math>CR</math> is the crossover probability.
* <math>j_{\text{rand}}</math> is a randomly chosen index.

=== Selection ===
The trial vector <math>u_i</math> replaces the target vector <math>x_i</math> in the population if it yields a better objective function value: <math>
x_i =
\begin{cases}
u_i, & \text{if } f(u_i) \leq f(x_i), \\
x_i, & \text{otherwise}.
\end{cases}
</math>

=== Pseudocode ===
The pseudocode of Differential Evolution is shown below:<pre>
Initialize population P randomly
While termination criterion not met:
For each individual x_i in P:
Generate mutant vector v_i
Perform crossover to create trial vector u_i
If f(u_i) < f(x_i): Replace x_i with u_i
Return the best solution
</pre>The pseudocode presented in this article is adapted from S. Das and P.N. Suganthan's survey on the state-of-the-art in DE <ref>S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.</ref>.

== Numerical Example ==
We aim to minimize the function <math>f(x) = x^2</math>, a common benchmark for optimization algorithms <ref>J. Birge, F. Louveaux, ''Introduction to Stochastic Programming''. Springer, 2011.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>. The example uses key principles of DE, including mutation and crossover, as described in foundational texts like R. Storn and K. Price’s work <ref>R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.</ref>.

=== Initial Setup ===
The initial population and fitness values are: <math>P = \{-7, 3, -2, 6, 8\}, \quad f(P) = \{49, 9, 4, 36, 64\}</math>

=== Iteration 1 ===
'''Individual 1''' (<math>x = -7</math>):

* Mutant vector: <math>v_1 = 3 + 0.8 \cdot (-2 - 8) = -5</math>
* Trial vector: <math display="inline">u_1 = -5</math>
* Fitness: <math>f(u_1) = 25</math>
* Since <math>f(u_1) < f(x_1)</math> (<math>25 < 49</math>), replace <math>x_1</math> with <math>u_1</math>
* Updated value: <math>x_1 = -5</math>

'''Individual 2''' (<math>x = 3</math>):

* Mutant vector: <math>v_2 = -2 + 0.8 \cdot (6 - 8) = -3.6</math>
* Trial vector: <math>u_2 = 3</math> (crossover fails)
* Fitness: <math>f(u_2) = 9</math>
* No replacement as <math>f(u_2) = f(x_2)</math>
* Updated value: <math>x_2 = 3</math>

'''Individual 3''' (<math>x = -2</math>):

* Mutant vector: <math>v_3 = 6 + 0.8 \cdot (8 - (-7)) = 18</math>
* Trial vector: <math>u_3 = -2</math> (crossover fails)
* Fitness: <math>f(u_3) = 4</math>
* No replacement as <math>f(u_3) = f(x_3)</math>
* Updated value: <math>x_3 = -2</math>

'''Individual 4''' (<math>x = 6</math>):

* Mutant vector: <math display="inline">v_4 = -7 + 0.8 \cdot (3 - (-2)) = -3</math>
* Trial vector: <math>u_4 = -3</math>
* Fitness: <math>f(u_4) = 9</math>
* Since <math>f(u_4) < f(x_4)</math> (<math>9 < 36</math>), replace <math>x_4</math> with <math>u_4</math>
* Updated value: <math>x_4 = -3</math>

'''Individual 5''' (<math>x = 8</math>):

* Mutant vector: <math>v_5 = 3 + 0.8 \cdot (-7 - (-2)) = -1</math>
* Trial vector: <math>u_5 = -1</math>
* Fitness: <math>f(u_5) = 1</math>
* Since <math>f(u_5) < f(x_5)</math> (<math>1 < 64</math>), replace <math>x_5</math> with <math>u_5</math>
* Updated value: <math>x_5 = -1</math>

After the first iteration, the updated population is: <math>P = \{-5, 3, -2, -3, -1\}, \quad f(P) = \{25, 9, 4, 9, 1\}</math>

=== Iterations 2–5 ===
The process is repeated for four more iterations. For each individual, the mutant vector, trial vector, and fitness are calculated. Individuals in the population are replaced whenever <math>f(u_i) < f(x_i)</math>. Over successive iterations, the population evolves toward the minimum.

=== Final Population ===
After five iterations, the population converges to: <math>P = \{0, 0, 0, 0, 0\}, \quad f(P) = \{0, 0, 0, 0, 0\}</math>

== Visualization ==
The progression of the population during the Differential Evolution process is illustrated in the figures below. Each figure shows the distribution of the population at a specific iteration, demonstrating how the algorithm converges toward the global minimum of the function <math>f(x) = x^2</math>.

'''Figure 1''': Initial population distribution before any iterations. [https://drive.google.com/file/d/12bfDfZCRMy0a85zDrEXwsifuWDf0Wcf2/view?usp=share_link View Figure 1: Initial population distribution before any iterations.]

'''Figure 2''': Population after the first iteration, showing improvement toward the global minimum. [https://drive.google.com/file/d/1A-miB_wZ9I13dNli09vaL81s_DkkYFAE/view?usp=share_link View Figure 2: Population after the first iteration, showing improvement toward the global minimum.]

'''Figure 3''': Population after the second iteration, with further refinement of solutions. [https://drive.google.com/file/d/1hMAQ_Cxbe3JFImiCzgnkKh2lnk2D1R7k/view?usp=share_link View Figure 3: Population after the second iteration, with further refinement of solutions.]

'''Figure 4''': Population after the fourth iteration, nearing convergence. [https://drive.google.com/file/d/132i2nEUx1PP6gjCv17TPMMjtCvCxi1VT/view?usp=sharing View Figure 4: Population after the fourth iteration, nearing convergence.]

'''Figure 5''': Final population converging to the global minimum at <math>x = 0</math>. [https://drive.google.com/file/d/1itJ6gvHnp_WdOeHAWXBHM_F2IEYm9WkZ/view?usp=sharing View Figure 5: Final population converging to the global minimum at <math>x = 0</math>.]

== Applications ==
Differential Evolution has been applied across various domains, including:

* '''Engineering Design''': Applications include minimizing the weight of tension/compression springs, following the approaches outlined by R. Storn and K. Price <ref>R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.</ref><ref>S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.</ref>.
* '''Water Resource Management''': Optimization strategies for reservoirs, as seen in recent research <ref>J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>.
* '''Machine Learning''': Parameter optimization for neural networks <ref>T. Hastie, R. Tibshirani, J. Friedman, ''The Elements of Statistical Learning''. Springer, 2009.</ref>.
* '''Medical Applications''': DE’s effectiveness in radiotherapy optimization is discussed in Abbass (2002) <ref>A. Abbass, "An Evolutionary Artificial Neural Networks Approach for Breast Cancer Diagnosis," ''Artificial Intelligence in Medicine'', vol. 25, no. 3, pp. 265-281, 2002.</ref>.

=== Software Tools ===
Several software libraries and frameworks have integrated Differential Evolution into their core optimization processes:

* '''ALGLIB''': A powerful library in C++, C#, and Java, capable of solving complex constrained and unconstrained optimization problems.
* '''DEAP (Distributed Evolutionary Algorithms in Python)''': A Python framework for rapid prototyping and testing of evolutionary algorithms, including DE.
* '''ECJ''': An open-source evolutionary computation system implemented in Java, supporting a variety of algorithms, including DE.
* '''MOEA Framework''': A Java library dedicated to multi-objective optimization, offering robust DE implementations.

Differential Evolution’s adaptability and effectiveness have made it an indispensable tool for tackling challenging optimization problems across these diverse domains.

=== Conclusion ===
Differential Evolution’s ability to handle a wide range of optimization problems has made it a preferred tool in various domains <ref>J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>. Despite its strengths, opportunities for improvement exist, including faster convergence and hybrid models <ref>S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.</ref>.

Throughout its application in engineering design, water resource management, machine learning, and medical fields, DE has demonstrated its effectiveness in solving complex and constrained problems. Its capacity for global convergence and minimal parameter tuning requirements further enhance its appeal for real-world problems.

Despite its strengths, there are areas for future research and improvement:

* Enhancing the convergence speed of the algorithm.
* Developing hybrid approaches by combining DE with other optimization techniques.
* Extending its application to emerging domains such as quantum computing and autonomous systems.
* Improving parameter tuning methods and software implementations to make DE more accessible and efficient for practitioners.

In summary, Differential Evolution continues to be a reliable and powerful tool for solving challenging optimization problems and holds promise for further advancements in both methodology and applications.

=== References ===
Below is a list of references used to support the content in this article:

# R.J. Vanderbei, ''Linear Programming: Foundations and Extensions''. Springer, 2008. Available at: [http://link.springer.com/book/10.1007/978-0-387-74388-2 SpringerLink].
# S. Boyd, L. Vandenberghe, ''Convex Optimization''. Cambridge University Press, 2009. Available at: [http://www.stanford.edu/~boyd/cvxbook/ Stanford University].
# J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999. Available at: [http://link.springer.com/book/10.1007/b98874 SpringerLink].
# J. Birge, F. Louveaux, ''Introduction to Stochastic Programming''. Springer, 2011. Available at: [http://link.springer.com/book/10.1007/978-1-4614-0237-4 SpringerLink].
# A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009. Available at: [http://www2.isye.gatech.edu/~nemirovs/FullBookDec11.pdf Georgia Institute of Technology].
# L.A. Wolsey, ''Integer Programming''. Wiley, 1998.
# L.T. Biegler, I.E. Grossmann, A.W. Westerberg, ''Systematic Methods of Chemical Process Design''. Prentice Hall Press, 1997.
# L.T. Biegler, ''Nonlinear Programming: Concepts, Algorithms and Applications to Chemical Engineering''. SIAM, 2010.
# T.F. Edgar, D.M. Himmelblau, L.S. Lasdon, ''Optimization of Chemical Processes''. McGraw-Hill, 2001.
# T. Hastie, R. Tibshirani, J. Friedman, ''The Elements of Statistical Learning''. Springer, 2009. Available at: [http://statweb.stanford.edu/~tibs/ElemStatLearn/ Stanford University].
# K. Murphy, ''Machine Learning: A Probabilistic Perspective''. MIT Press, 2012.

Additional references for application examples:

* S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.
* R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.
* A. Abbass, "An Evolutionary Artificial Neural Networks Approach for Breast Cancer Diagnosis," ''Artificial Intelligence in Medicine'', vol. 25, no. 3, pp. 265-281, 2002.

Differential evolution

2024-12-16T03:10:12Z

Fall2024 Wiki Team26: /* Visualization */

Author: Rohit Kumar (rk787) (ChemE 6800 Fall 2024)

Stewards: Nathan Preuss, Wei-Han Chen, Tianqi Xiao, Guoqing Hu

== Introduction ==
Differential Evolution (DE) is a robust and efficient optimization algorithm widely used for solving non-linear, non-differentiable, and multimodal optimization problems. It was first introduced by R. Storn and K. Price in their seminal work on global optimization over continuous spaces <ref>R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.</ref>, where they described DE as:-

''"A new heuristic approach for minimizing possibly nonlinear and non-differentiable continuous-space functions."''

A heuristic is any approximation method for a solution to a problem capable of producing satisfactory solutions within tolerable amounts of time when classic methods cannot find an exact answer or work quite slowly. Unlike the Gradient Descent algorithm, the optimization problem does not need to be differentiable.

The key idea is to generate a population of candidate solutions through successive generations using vector differences for deviations. This approach allows DE to adapt its search methodology to the objective function, making it particularly effective for non-linear, non-differentiable, and multimodal optimization problems.

The algorithm is renowned for its simplicity and efficiency. Its primary advantages include:

* Requires minimal parameter tuning.
* High robustness.
* Ability to avoid local optima and converge toward global solutions.

In DE, the currently evolving population of possible solutions is represented as a vector of real-valued numbers. The algorithm attempts to enhance these solutions through a series of iterations built upon three main steps:

* Mutation
* Crossover
* Selection

DE has become highly popular for applications in such fields as engineering design, machine learning, signal processing, and financial modeling. Despite its simplicity, Differential Evolution is competitive against many other optimization techniques, often performing better on challenging problems.

== Algorithm Discussion ==
Differential Evolution operates through three main steps: initialization, mutation, crossover, and selection, as outlined in optimization literature <ref>J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>.

=== Initialization ===
The initial population is generated randomly within the bounds of the search space: <math>x_{i,j} \sim U(x_j^{\text{min}}, x_j^{\text{max}})</math> where:

* <math>x_{i,j}</math> represents the <math>j^{\text{th}}</math> parameter of the <math>i^{\text{th}}</math> individual.
* <math>U(a, b)</math> is a uniform random distribution.

=== Mutation ===
A mutant vector <math>v_i</math> is generated by combining three randomly chosen vectors: <math>v_i = x_{r1} + F \cdot (x_{r2} - x_{r3})</math> where:

* <math>F</math> is the mutation factor.
* <math>x_{r1}, x_{r2}, x_{r3}</math> are distinct vectors from the population.

=== Crossover ===
A trial vector <math>u_i</math> is created by mixing the mutant vector <math>v_i</math> and the target vector <math>x_i</math>: <math>
u_{i,j} =
\begin{cases}
v_{i,j}, & \text{if } \text{rand}(0, 1) \leq CR \text{ or } j = j_{\text{rand}}, \\
x_{i,j}, & \text{otherwise}.
\end{cases}
</math> where:

* <math>CR</math> is the crossover probability.
* <math>j_{\text{rand}}</math> is a randomly chosen index.

=== Selection ===
The trial vector <math>u_i</math> replaces the target vector <math>x_i</math> in the population if it yields a better objective function value: <math>
x_i =
\begin{cases}
u_i, & \text{if } f(u_i) \leq f(x_i), \\
x_i, & \text{otherwise}.
\end{cases}
</math>

=== Pseudocode ===
The pseudocode of Differential Evolution is shown below:<pre>
Initialize population P randomly
While termination criterion not met:
For each individual x_i in P:
Generate mutant vector v_i
Perform crossover to create trial vector u_i
If f(u_i) < f(x_i): Replace x_i with u_i
Return the best solution
</pre>The pseudocode presented in this article is adapted from S. Das and P.N. Suganthan's survey on the state-of-the-art in DE <ref>S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.</ref>.

== Numerical Example ==
We aim to minimize the function <math>f(x) = x^2</math>, a common benchmark for optimization algorithms <ref>J. Birge, F. Louveaux, ''Introduction to Stochastic Programming''. Springer, 2011.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>. The example uses key principles of DE, including mutation and crossover, as described in foundational texts like R. Storn and K. Price’s work <ref>R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.</ref>.

=== Initial Setup ===
The initial population and fitness values are: <math>P = \{-7, 3, -2, 6, 8\}, \quad f(P) = \{49, 9, 4, 36, 64\}</math>

=== Iteration 1 ===
'''Individual 1''' (<math>x = -7</math>):

* Mutant vector: <math>v_1 = 3 + 0.8 \cdot (-2 - 8) = -5</math>
* Trial vector: <math display="inline">u_1 = -5</math>
* Fitness: <math>f(u_1) = 25</math>
* Since <math>f(u_1) < f(x_1)</math> (<math>25 < 49</math>), replace <math>x_1</math> with <math>u_1</math>
* Updated value: <math>x_1 = -5</math>

'''Individual 2''' (<math>x = 3</math>):

* Mutant vector: <math>v_2 = -2 + 0.8 \cdot (6 - 8) = -3.6</math>
* Trial vector: <math>u_2 = 3</math> (crossover fails)
* Fitness: <math>f(u_2) = 9</math>
* No replacement as <math>f(u_2) = f(x_2)</math>
* Updated value: <math>x_2 = 3</math>

'''Individual 3''' (<math>x = -2</math>):

* Mutant vector: <math>v_3 = 6 + 0.8 \cdot (8 - (-7)) = 18</math>
* Trial vector: <math>u_3 = -2</math> (crossover fails)
* Fitness: <math>f(u_3) = 4</math>
* No replacement as <math>f(u_3) = f(x_3)</math>
* Updated value: <math>x_3 = -2</math>

'''Individual 4''' (<math>x = 6</math>):

* Mutant vector: <math display="inline">v_4 = -7 + 0.8 \cdot (3 - (-2)) = -3</math>
* Trial vector: <math>u_4 = -3</math>
* Fitness: <math>f(u_4) = 9</math>
* Since <math>f(u_4) < f(x_4)</math> (<math>9 < 36</math>), replace <math>x_4</math> with <math>u_4</math>
* Updated value: <math>x_4 = -3</math>

'''Individual 5''' (<math>x = 8</math>):

* Mutant vector: <math>v_5 = 3 + 0.8 \cdot (-7 - (-2)) = -1</math>
* Trial vector: <math>u_5 = -1</math>
* Fitness: <math>f(u_5) = 1</math>
* Since <math>f(u_5) < f(x_5)</math> (<math>1 < 64</math>), replace <math>x_5</math> with <math>u_5</math>
* Updated value: <math>x_5 = -1</math>

After the first iteration, the updated population is: <math>P = \{-5, 3, -2, -3, -1\}, \quad f(P) = \{25, 9, 4, 9, 1\}</math>

=== Iterations 2–5 ===
The process is repeated for four more iterations. For each individual, the mutant vector, trial vector, and fitness are calculated. Individuals in the population are replaced whenever <math>f(u_i) < f(x_i)</math>. Over successive iterations, the population evolves toward the minimum.

=== Final Population ===
After five iterations, the population converges to: <math>P = \{0, 0, 0, 0, 0\}, \quad f(P) = \{0, 0, 0, 0, 0\}</math>

== Visualization ==
The progression of the population during the Differential Evolution process is illustrated in the figures below. Each figure shows the distribution of the population at a specific iteration, demonstrating how the algorithm converges toward the global minimum of the function <math>f(x) = x^2</math>.

'''Figure 1''': Initial population distribution before any iterations.
[[index.php?title=File:Ezgif-frame-001.png|link=File:Ezgif-frame-001.png|center|thumb|600x600px|Initial population distribution before any iterations.]]
'''Figure 2''': Population after the first iteration, showing improvement toward the global minimum.
[[index.php?title=File:Ezgif-frame-006.png|link=File:Ezgif-frame-006.png|center|thumb|600x600px|Population after the first iteration, showing improvement toward the global minimum.]]
'''Figure 3''': Population after the second iteration, with further refinement of solutions.
[[index.php?title=File:Ezgif-frame-011.png|link=File:Ezgif-frame-011.png|center|thumb|600x600px|Population after the second iteration, with further refinement of solutions.]]
'''Figure 4''': Population after the fourth iteration, nearing convergence.
[[index.php?title=File:Ezgif-frame-018.png|link=File:Ezgif-frame-018.png|center|thumb|600x600px|Population after the fourth iteration, nearing convergence.]]
'''Figure 5''': Final population converging to the global minimum at <math>x = 0</math>.
[[index.php?title=File:Ezgif-frame-025.png|link=File:Ezgif-frame-025.png|center|thumb|600x600px|Final population converging to the global minimum at <math>x = 0</math>.]]

== Applications ==
Differential Evolution has been applied across various domains, including:

* '''Engineering Design''': Applications include minimizing the weight of tension/compression springs, following the approaches outlined by R. Storn and K. Price <ref>R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.</ref><ref>S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.</ref>.
* '''Water Resource Management''': Optimization strategies for reservoirs, as seen in recent research <ref>J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>.
* '''Machine Learning''': Parameter optimization for neural networks <ref>T. Hastie, R. Tibshirani, J. Friedman, ''The Elements of Statistical Learning''. Springer, 2009.</ref>.
* '''Medical Applications''': DE’s effectiveness in radiotherapy optimization is discussed in Abbass (2002) <ref>A. Abbass, "An Evolutionary Artificial Neural Networks Approach for Breast Cancer Diagnosis," ''Artificial Intelligence in Medicine'', vol. 25, no. 3, pp. 265-281, 2002.</ref>.

=== Software Tools ===
Several software libraries and frameworks have integrated Differential Evolution into their core optimization processes:

* '''ALGLIB''': A powerful library in C++, C#, and Java, capable of solving complex constrained and unconstrained optimization problems.
* '''DEAP (Distributed Evolutionary Algorithms in Python)''': A Python framework for rapid prototyping and testing of evolutionary algorithms, including DE.
* '''ECJ''': An open-source evolutionary computation system implemented in Java, supporting a variety of algorithms, including DE.
* '''MOEA Framework''': A Java library dedicated to multi-objective optimization, offering robust DE implementations.

Differential Evolution’s adaptability and effectiveness have made it an indispensable tool for tackling challenging optimization problems across these diverse domains.

=== Conclusion ===
Differential Evolution’s ability to handle a wide range of optimization problems has made it a preferred tool in various domains <ref>J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>. Despite its strengths, opportunities for improvement exist, including faster convergence and hybrid models <ref>S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.</ref>.

Throughout its application in engineering design, water resource management, machine learning, and medical fields, DE has demonstrated its effectiveness in solving complex and constrained problems. Its capacity for global convergence and minimal parameter tuning requirements further enhance its appeal for real-world problems.

Despite its strengths, there are areas for future research and improvement:

* Enhancing the convergence speed of the algorithm.
* Developing hybrid approaches by combining DE with other optimization techniques.
* Extending its application to emerging domains such as quantum computing and autonomous systems.
* Improving parameter tuning methods and software implementations to make DE more accessible and efficient for practitioners.

In summary, Differential Evolution continues to be a reliable and powerful tool for solving challenging optimization problems and holds promise for further advancements in both methodology and applications.

=== References ===
Below is a list of references used to support the content in this article:

# R.J. Vanderbei, ''Linear Programming: Foundations and Extensions''. Springer, 2008. Available at: [http://link.springer.com/book/10.1007/978-0-387-74388-2 SpringerLink].
# S. Boyd, L. Vandenberghe, ''Convex Optimization''. Cambridge University Press, 2009. Available at: [http://www.stanford.edu/~boyd/cvxbook/ Stanford University].
# J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999. Available at: [http://link.springer.com/book/10.1007/b98874 SpringerLink].
# J. Birge, F. Louveaux, ''Introduction to Stochastic Programming''. Springer, 2011. Available at: [http://link.springer.com/book/10.1007/978-1-4614-0237-4 SpringerLink].
# A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009. Available at: [http://www2.isye.gatech.edu/~nemirovs/FullBookDec11.pdf Georgia Institute of Technology].
# L.A. Wolsey, ''Integer Programming''. Wiley, 1998.
# L.T. Biegler, I.E. Grossmann, A.W. Westerberg, ''Systematic Methods of Chemical Process Design''. Prentice Hall Press, 1997.
# L.T. Biegler, ''Nonlinear Programming: Concepts, Algorithms and Applications to Chemical Engineering''. SIAM, 2010.
# T.F. Edgar, D.M. Himmelblau, L.S. Lasdon, ''Optimization of Chemical Processes''. McGraw-Hill, 2001.
# T. Hastie, R. Tibshirani, J. Friedman, ''The Elements of Statistical Learning''. Springer, 2009. Available at: [http://statweb.stanford.edu/~tibs/ElemStatLearn/ Stanford University].
# K. Murphy, ''Machine Learning: A Probabilistic Perspective''. MIT Press, 2012.

Additional references for application examples:

* S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.
* R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.
* A. Abbass, "An Evolutionary Artificial Neural Networks Approach for Breast Cancer Diagnosis," ''Artificial Intelligence in Medicine'', vol. 25, no. 3, pp. 265-281, 2002.

Differential evolution

2024-12-16T03:06:03Z

Fall2024 Wiki Team26: /* Visualization */

Author: Rohit Kumar (rk787) (ChemE 6800 Fall 2024)

Stewards: Nathan Preuss, Wei-Han Chen, Tianqi Xiao, Guoqing Hu

== Introduction ==
Differential Evolution (DE) is a robust and efficient optimization algorithm widely used for solving non-linear, non-differentiable, and multimodal optimization problems. It was first introduced by R. Storn and K. Price in their seminal work on global optimization over continuous spaces <ref>R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.</ref>, where they described DE as:-

''"A new heuristic approach for minimizing possibly nonlinear and non-differentiable continuous-space functions."''

A heuristic is any approximation method for a solution to a problem capable of producing satisfactory solutions within tolerable amounts of time when classic methods cannot find an exact answer or work quite slowly. Unlike the Gradient Descent algorithm, the optimization problem does not need to be differentiable.

The key idea is to generate a population of candidate solutions through successive generations using vector differences for deviations. This approach allows DE to adapt its search methodology to the objective function, making it particularly effective for non-linear, non-differentiable, and multimodal optimization problems.

The algorithm is renowned for its simplicity and efficiency. Its primary advantages include:

* Requires minimal parameter tuning.
* High robustness.
* Ability to avoid local optima and converge toward global solutions.

In DE, the currently evolving population of possible solutions is represented as a vector of real-valued numbers. The algorithm attempts to enhance these solutions through a series of iterations built upon three main steps:

* Mutation
* Crossover
* Selection

DE has become highly popular for applications in such fields as engineering design, machine learning, signal processing, and financial modeling. Despite its simplicity, Differential Evolution is competitive against many other optimization techniques, often performing better on challenging problems.

== Algorithm Discussion ==
Differential Evolution operates through three main steps: initialization, mutation, crossover, and selection, as outlined in optimization literature <ref>J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>.

=== Initialization ===
The initial population is generated randomly within the bounds of the search space: <math>x_{i,j} \sim U(x_j^{\text{min}}, x_j^{\text{max}})</math> where:

* <math>x_{i,j}</math> represents the <math>j^{\text{th}}</math> parameter of the <math>i^{\text{th}}</math> individual.
* <math>U(a, b)</math> is a uniform random distribution.

=== Mutation ===
A mutant vector <math>v_i</math> is generated by combining three randomly chosen vectors: <math>v_i = x_{r1} + F \cdot (x_{r2} - x_{r3})</math> where:

* <math>F</math> is the mutation factor.
* <math>x_{r1}, x_{r2}, x_{r3}</math> are distinct vectors from the population.

=== Crossover ===
A trial vector <math>u_i</math> is created by mixing the mutant vector <math>v_i</math> and the target vector <math>x_i</math>: <math>
u_{i,j} =
\begin{cases}
v_{i,j}, & \text{if } \text{rand}(0, 1) \leq CR \text{ or } j = j_{\text{rand}}, \\
x_{i,j}, & \text{otherwise}.
\end{cases}
</math> where:

* <math>CR</math> is the crossover probability.
* <math>j_{\text{rand}}</math> is a randomly chosen index.

=== Selection ===
The trial vector <math>u_i</math> replaces the target vector <math>x_i</math> in the population if it yields a better objective function value: <math>
x_i =
\begin{cases}
u_i, & \text{if } f(u_i) \leq f(x_i), \\
x_i, & \text{otherwise}.
\end{cases}
</math>

=== Pseudocode ===
The pseudocode of Differential Evolution is shown below:<pre>
Initialize population P randomly
While termination criterion not met:
For each individual x_i in P:
Generate mutant vector v_i
Perform crossover to create trial vector u_i
If f(u_i) < f(x_i): Replace x_i with u_i
Return the best solution
</pre>The pseudocode presented in this article is adapted from S. Das and P.N. Suganthan's survey on the state-of-the-art in DE <ref>S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.</ref>.

== Numerical Example ==
We aim to minimize the function <math>f(x) = x^2</math>, a common benchmark for optimization algorithms <ref>J. Birge, F. Louveaux, ''Introduction to Stochastic Programming''. Springer, 2011.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>. The example uses key principles of DE, including mutation and crossover, as described in foundational texts like R. Storn and K. Price’s work <ref>R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.</ref>.

=== Initial Setup ===
The initial population and fitness values are: <math>P = \{-7, 3, -2, 6, 8\}, \quad f(P) = \{49, 9, 4, 36, 64\}</math>

=== Iteration 1 ===
'''Individual 1''' (<math>x = -7</math>):

* Mutant vector: <math>v_1 = 3 + 0.8 \cdot (-2 - 8) = -5</math>
* Trial vector: <math display="inline">u_1 = -5</math>
* Fitness: <math>f(u_1) = 25</math>
* Since <math>f(u_1) < f(x_1)</math> (<math>25 < 49</math>), replace <math>x_1</math> with <math>u_1</math>
* Updated value: <math>x_1 = -5</math>

'''Individual 2''' (<math>x = 3</math>):

* Mutant vector: <math>v_2 = -2 + 0.8 \cdot (6 - 8) = -3.6</math>
* Trial vector: <math>u_2 = 3</math> (crossover fails)
* Fitness: <math>f(u_2) = 9</math>
* No replacement as <math>f(u_2) = f(x_2)</math>
* Updated value: <math>x_2 = 3</math>

'''Individual 3''' (<math>x = -2</math>):

* Mutant vector: <math>v_3 = 6 + 0.8 \cdot (8 - (-7)) = 18</math>
* Trial vector: <math>u_3 = -2</math> (crossover fails)
* Fitness: <math>f(u_3) = 4</math>
* No replacement as <math>f(u_3) = f(x_3)</math>
* Updated value: <math>x_3 = -2</math>

'''Individual 4''' (<math>x = 6</math>):

* Mutant vector: <math display="inline">v_4 = -7 + 0.8 \cdot (3 - (-2)) = -3</math>
* Trial vector: <math>u_4 = -3</math>
* Fitness: <math>f(u_4) = 9</math>
* Since <math>f(u_4) < f(x_4)</math> (<math>9 < 36</math>), replace <math>x_4</math> with <math>u_4</math>
* Updated value: <math>x_4 = -3</math>

'''Individual 5''' (<math>x = 8</math>):

* Mutant vector: <math>v_5 = 3 + 0.8 \cdot (-7 - (-2)) = -1</math>
* Trial vector: <math>u_5 = -1</math>
* Fitness: <math>f(u_5) = 1</math>
* Since <math>f(u_5) < f(x_5)</math> (<math>1 < 64</math>), replace <math>x_5</math> with <math>u_5</math>
* Updated value: <math>x_5 = -1</math>

After the first iteration, the updated population is: <math>P = \{-5, 3, -2, -3, -1\}, \quad f(P) = \{25, 9, 4, 9, 1\}</math>

=== Iterations 2–5 ===
The process is repeated for four more iterations. For each individual, the mutant vector, trial vector, and fitness are calculated. Individuals in the population are replaced whenever <math>f(u_i) < f(x_i)</math>. Over successive iterations, the population evolves toward the minimum.

=== Final Population ===
After five iterations, the population converges to: <math>P = \{0, 0, 0, 0, 0\}, \quad f(P) = \{0, 0, 0, 0, 0\}</math>

== Visualization ==
The progression of the population during the Differential Evolution process is illustrated in the figures below. Each figure shows the distribution of the population at a specific iteration, demonstrating how the algorithm converges toward the global minimum of the function <math>f(x) = x^2</math>.
[[index.php?title=File:Ezgif-frame-001.png|link=File:Ezgif-frame-001.png|center|thumb|600x600px|Figure 1: Initial population distribution before any iterations.]]
[[index.php?title=File:Ezgif-frame-006.png|link=File:Ezgif-frame-006.png|center|thumb|600x600px|Figure 2: Population after the first iteration, showing improvement toward the global minimum.]]
[[index.php?title=File:Ezgif-frame-011.png|link=File:Ezgif-frame-011.png|center|thumb|600x600px|Figure 3: Population after the second iteration, with further refinement of solutions.]]
[[index.php?title=File:Ezgif-frame-018.png|link=File:Ezgif-frame-018.png|center|thumb|600x600px|Figure 4: Population after the fourth iteration, nearing convergence.]]
[[index.php?title=File:Ezgif-frame-025.png|link=File:Ezgif-frame-025.png|center|thumb|600x600px|Figure 5: Final population converging to the global minimum at <math>x = 0</math>.]]

== Applications ==
Differential Evolution has been applied across various domains, including:

* '''Engineering Design''': Applications include minimizing the weight of tension/compression springs, following the approaches outlined by R. Storn and K. Price <ref>R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.</ref><ref>S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.</ref>.
* '''Water Resource Management''': Optimization strategies for reservoirs, as seen in recent research <ref>J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>.
* '''Machine Learning''': Parameter optimization for neural networks <ref>T. Hastie, R. Tibshirani, J. Friedman, ''The Elements of Statistical Learning''. Springer, 2009.</ref>.
* '''Medical Applications''': DE’s effectiveness in radiotherapy optimization is discussed in Abbass (2002) <ref>A. Abbass, "An Evolutionary Artificial Neural Networks Approach for Breast Cancer Diagnosis," ''Artificial Intelligence in Medicine'', vol. 25, no. 3, pp. 265-281, 2002.</ref>.

=== Software Tools ===
Several software libraries and frameworks have integrated Differential Evolution into their core optimization processes:

* '''ALGLIB''': A powerful library in C++, C#, and Java, capable of solving complex constrained and unconstrained optimization problems.
* '''DEAP (Distributed Evolutionary Algorithms in Python)''': A Python framework for rapid prototyping and testing of evolutionary algorithms, including DE.
* '''ECJ''': An open-source evolutionary computation system implemented in Java, supporting a variety of algorithms, including DE.
* '''MOEA Framework''': A Java library dedicated to multi-objective optimization, offering robust DE implementations.

Differential Evolution’s adaptability and effectiveness have made it an indispensable tool for tackling challenging optimization problems across these diverse domains.

=== Conclusion ===
Differential Evolution’s ability to handle a wide range of optimization problems has made it a preferred tool in various domains <ref>J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>. Despite its strengths, opportunities for improvement exist, including faster convergence and hybrid models <ref>S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.</ref>.

Throughout its application in engineering design, water resource management, machine learning, and medical fields, DE has demonstrated its effectiveness in solving complex and constrained problems. Its capacity for global convergence and minimal parameter tuning requirements further enhance its appeal for real-world problems.

Despite its strengths, there are areas for future research and improvement:

* Enhancing the convergence speed of the algorithm.
* Developing hybrid approaches by combining DE with other optimization techniques.
* Extending its application to emerging domains such as quantum computing and autonomous systems.
* Improving parameter tuning methods and software implementations to make DE more accessible and efficient for practitioners.

In summary, Differential Evolution continues to be a reliable and powerful tool for solving challenging optimization problems and holds promise for further advancements in both methodology and applications.

=== References ===
Below is a list of references used to support the content in this article:

# R.J. Vanderbei, ''Linear Programming: Foundations and Extensions''. Springer, 2008. Available at: [http://link.springer.com/book/10.1007/978-0-387-74388-2 SpringerLink].
# S. Boyd, L. Vandenberghe, ''Convex Optimization''. Cambridge University Press, 2009. Available at: [http://www.stanford.edu/~boyd/cvxbook/ Stanford University].
# J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999. Available at: [http://link.springer.com/book/10.1007/b98874 SpringerLink].
# J. Birge, F. Louveaux, ''Introduction to Stochastic Programming''. Springer, 2011. Available at: [http://link.springer.com/book/10.1007/978-1-4614-0237-4 SpringerLink].
# A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009. Available at: [http://www2.isye.gatech.edu/~nemirovs/FullBookDec11.pdf Georgia Institute of Technology].
# L.A. Wolsey, ''Integer Programming''. Wiley, 1998.
# L.T. Biegler, I.E. Grossmann, A.W. Westerberg, ''Systematic Methods of Chemical Process Design''. Prentice Hall Press, 1997.
# L.T. Biegler, ''Nonlinear Programming: Concepts, Algorithms and Applications to Chemical Engineering''. SIAM, 2010.
# T.F. Edgar, D.M. Himmelblau, L.S. Lasdon, ''Optimization of Chemical Processes''. McGraw-Hill, 2001.
# T. Hastie, R. Tibshirani, J. Friedman, ''The Elements of Statistical Learning''. Springer, 2009. Available at: [http://statweb.stanford.edu/~tibs/ElemStatLearn/ Stanford University].
# K. Murphy, ''Machine Learning: A Probabilistic Perspective''. MIT Press, 2012.

Additional references for application examples:

* S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.
* R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.
* A. Abbass, "An Evolutionary Artificial Neural Networks Approach for Breast Cancer Diagnosis," ''Artificial Intelligence in Medicine'', vol. 25, no. 3, pp. 265-281, 2002.

Differential evolution

2024-12-16T03:03:08Z

Fall2024 Wiki Team26: Solved Image error

Author: Rohit Kumar (rk787) (ChemE 6800 Fall 2024)

Stewards: Nathan Preuss, Wei-Han Chen, Tianqi Xiao, Guoqing Hu

== Introduction ==
Differential Evolution (DE) is a robust and efficient optimization algorithm widely used for solving non-linear, non-differentiable, and multimodal optimization problems. It was first introduced by R. Storn and K. Price in their seminal work on global optimization over continuous spaces <ref>R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.</ref>, where they described DE as:-

''"A new heuristic approach for minimizing possibly nonlinear and non-differentiable continuous-space functions."''

A heuristic is any approximation method for a solution to a problem capable of producing satisfactory solutions within tolerable amounts of time when classic methods cannot find an exact answer or work quite slowly. Unlike the Gradient Descent algorithm, the optimization problem does not need to be differentiable.

The key idea is to generate a population of candidate solutions through successive generations using vector differences for deviations. This approach allows DE to adapt its search methodology to the objective function, making it particularly effective for non-linear, non-differentiable, and multimodal optimization problems.

The algorithm is renowned for its simplicity and efficiency. Its primary advantages include:

* Requires minimal parameter tuning.
* High robustness.
* Ability to avoid local optima and converge toward global solutions.

In DE, the currently evolving population of possible solutions is represented as a vector of real-valued numbers. The algorithm attempts to enhance these solutions through a series of iterations built upon three main steps:

* Mutation
* Crossover
* Selection

DE has become highly popular for applications in such fields as engineering design, machine learning, signal processing, and financial modeling. Despite its simplicity, Differential Evolution is competitive against many other optimization techniques, often performing better on challenging problems.

== Algorithm Discussion ==
Differential Evolution operates through three main steps: initialization, mutation, crossover, and selection, as outlined in optimization literature <ref>J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>.

=== Initialization ===
The initial population is generated randomly within the bounds of the search space: <math>x_{i,j} \sim U(x_j^{\text{min}}, x_j^{\text{max}})</math> where:

* <math>x_{i,j}</math> represents the <math>j^{\text{th}}</math> parameter of the <math>i^{\text{th}}</math> individual.
* <math>U(a, b)</math> is a uniform random distribution.

=== Mutation ===
A mutant vector <math>v_i</math> is generated by combining three randomly chosen vectors: <math>v_i = x_{r1} + F \cdot (x_{r2} - x_{r3})</math> where:

* <math>F</math> is the mutation factor.
* <math>x_{r1}, x_{r2}, x_{r3}</math> are distinct vectors from the population.

=== Crossover ===
A trial vector <math>u_i</math> is created by mixing the mutant vector <math>v_i</math> and the target vector <math>x_i</math>: <math>
u_{i,j} =
\begin{cases}
v_{i,j}, & \text{if } \text{rand}(0, 1) \leq CR \text{ or } j = j_{\text{rand}}, \\
x_{i,j}, & \text{otherwise}.
\end{cases}
</math> where:

* <math>CR</math> is the crossover probability.
* <math>j_{\text{rand}}</math> is a randomly chosen index.

=== Selection ===
The trial vector <math>u_i</math> replaces the target vector <math>x_i</math> in the population if it yields a better objective function value: <math>
x_i =
\begin{cases}
u_i, & \text{if } f(u_i) \leq f(x_i), \\
x_i, & \text{otherwise}.
\end{cases}
</math>

=== Pseudocode ===
The pseudocode of Differential Evolution is shown below:<pre>
Initialize population P randomly
While termination criterion not met:
For each individual x_i in P:
Generate mutant vector v_i
Perform crossover to create trial vector u_i
If f(u_i) < f(x_i): Replace x_i with u_i
Return the best solution
</pre>The pseudocode presented in this article is adapted from S. Das and P.N. Suganthan's survey on the state-of-the-art in DE <ref>S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.</ref>.

== Numerical Example ==
We aim to minimize the function <math>f(x) = x^2</math>, a common benchmark for optimization algorithms <ref>J. Birge, F. Louveaux, ''Introduction to Stochastic Programming''. Springer, 2011.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>. The example uses key principles of DE, including mutation and crossover, as described in foundational texts like R. Storn and K. Price’s work <ref>R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.</ref>.

=== Initial Setup ===
The initial population and fitness values are: <math>P = \{-7, 3, -2, 6, 8\}, \quad f(P) = \{49, 9, 4, 36, 64\}</math>

=== Iteration 1 ===
'''Individual 1''' (<math>x = -7</math>):

* Mutant vector: <math>v_1 = 3 + 0.8 \cdot (-2 - 8) = -5</math>
* Trial vector: <math display="inline">u_1 = -5</math>
* Fitness: <math>f(u_1) = 25</math>
* Since <math>f(u_1) < f(x_1)</math> (<math>25 < 49</math>), replace <math>x_1</math> with <math>u_1</math>
* Updated value: <math>x_1 = -5</math>

'''Individual 2''' (<math>x = 3</math>):

* Mutant vector: <math>v_2 = -2 + 0.8 \cdot (6 - 8) = -3.6</math>
* Trial vector: <math>u_2 = 3</math> (crossover fails)
* Fitness: <math>f(u_2) = 9</math>
* No replacement as <math>f(u_2) = f(x_2)</math>
* Updated value: <math>x_2 = 3</math>

'''Individual 3''' (<math>x = -2</math>):

* Mutant vector: <math>v_3 = 6 + 0.8 \cdot (8 - (-7)) = 18</math>
* Trial vector: <math>u_3 = -2</math> (crossover fails)
* Fitness: <math>f(u_3) = 4</math>
* No replacement as <math>f(u_3) = f(x_3)</math>
* Updated value: <math>x_3 = -2</math>

'''Individual 4''' (<math>x = 6</math>):

* Mutant vector: <math display="inline">v_4 = -7 + 0.8 \cdot (3 - (-2)) = -3</math>
* Trial vector: <math>u_4 = -3</math>
* Fitness: <math>f(u_4) = 9</math>
* Since <math>f(u_4) < f(x_4)</math> (<math>9 < 36</math>), replace <math>x_4</math> with <math>u_4</math>
* Updated value: <math>x_4 = -3</math>

'''Individual 5''' (<math>x = 8</math>):

* Mutant vector: <math>v_5 = 3 + 0.8 \cdot (-7 - (-2)) = -1</math>
* Trial vector: <math>u_5 = -1</math>
* Fitness: <math>f(u_5) = 1</math>
* Since <math>f(u_5) < f(x_5)</math> (<math>1 < 64</math>), replace <math>x_5</math> with <math>u_5</math>
* Updated value: <math>x_5 = -1</math>

After the first iteration, the updated population is: <math>P = \{-5, 3, -2, -3, -1\}, \quad f(P) = \{25, 9, 4, 9, 1\}</math>

=== Iterations 2–5 ===
The process is repeated for four more iterations. For each individual, the mutant vector, trial vector, and fitness are calculated. Individuals in the population are replaced whenever <math>f(u_i) < f(x_i)</math>. Over successive iterations, the population evolves toward the minimum.

=== Final Population ===
After five iterations, the population converges to: <math>P = \{0, 0, 0, 0, 0\}, \quad f(P) = \{0, 0, 0, 0, 0\}</math>

== Visualization ==
The progression of the population during the Differential Evolution process is illustrated in the figures below. Each figure shows the distribution of the population at a specific iteration, demonstrating how the algorithm converges toward the global minimum of the function <math>f(x) = x^2</math>.

'''Figure 1''': Initial population distribution.
[[index.php?title=File:Ezgif-frame-001.png|link=File:Ezgif-frame-001.png|center|600x600px|Initial population distribution before any iterations.]]
'''Figure 2''': Population after Iteration 1.
[[index.php?title=File:Ezgif-frame-006.png|link=File:Ezgif-frame-006.png|center|600x600px|Population after the first iteration, showing improvement toward the global minimum.]]
'''Figure 3''': Population after Iteration 2.
[[index.php?title=File:Ezgif-frame-011.png|link=File:Ezgif-frame-011.png|center|600x600px|Population after the second iteration, with further refinement of solutions.]]
'''Figure 4''': Population after Iteration 4.
[[index.php?title=File:Ezgif-frame-018.png|link=File:Ezgif-frame-018.png|center|600x600px|Population after the fourth iteration, nearing convergence.]]
'''Figure 5''': Final population after Iteration 5.
[[index.php?title=File:Ezgif-frame-025.png|link=File:Ezgif-frame-025.png|alt=Final population converging to the global minimum at

x
<nowiki/>=
0

{\displaystyle x=0}

.|center|600x600px|Final population converging to the global minimum at <math>x = 0</math>.]]

== Applications ==
Differential Evolution has been applied across various domains, including:

* '''Engineering Design''': Applications include minimizing the weight of tension/compression springs, following the approaches outlined by R. Storn and K. Price <ref>R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.</ref><ref>S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.</ref>.
* '''Water Resource Management''': Optimization strategies for reservoirs, as seen in recent research <ref>J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>.
* '''Machine Learning''': Parameter optimization for neural networks <ref>T. Hastie, R. Tibshirani, J. Friedman, ''The Elements of Statistical Learning''. Springer, 2009.</ref>.
* '''Medical Applications''': DE’s effectiveness in radiotherapy optimization is discussed in Abbass (2002) <ref>A. Abbass, "An Evolutionary Artificial Neural Networks Approach for Breast Cancer Diagnosis," ''Artificial Intelligence in Medicine'', vol. 25, no. 3, pp. 265-281, 2002.</ref>.

=== Software Tools ===
Several software libraries and frameworks have integrated Differential Evolution into their core optimization processes:

* '''ALGLIB''': A powerful library in C++, C#, and Java, capable of solving complex constrained and unconstrained optimization problems.
* '''DEAP (Distributed Evolutionary Algorithms in Python)''': A Python framework for rapid prototyping and testing of evolutionary algorithms, including DE.
* '''ECJ''': An open-source evolutionary computation system implemented in Java, supporting a variety of algorithms, including DE.
* '''MOEA Framework''': A Java library dedicated to multi-objective optimization, offering robust DE implementations.

Differential Evolution’s adaptability and effectiveness have made it an indispensable tool for tackling challenging optimization problems across these diverse domains.

=== Conclusion ===
Differential Evolution’s ability to handle a wide range of optimization problems has made it a preferred tool in various domains <ref>J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>. Despite its strengths, opportunities for improvement exist, including faster convergence and hybrid models <ref>S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.</ref>.

Throughout its application in engineering design, water resource management, machine learning, and medical fields, DE has demonstrated its effectiveness in solving complex and constrained problems. Its capacity for global convergence and minimal parameter tuning requirements further enhance its appeal for real-world problems.

Despite its strengths, there are areas for future research and improvement:

* Enhancing the convergence speed of the algorithm.
* Developing hybrid approaches by combining DE with other optimization techniques.
* Extending its application to emerging domains such as quantum computing and autonomous systems.
* Improving parameter tuning methods and software implementations to make DE more accessible and efficient for practitioners.

In summary, Differential Evolution continues to be a reliable and powerful tool for solving challenging optimization problems and holds promise for further advancements in both methodology and applications.

=== References ===
Below is a list of references used to support the content in this article:

# R.J. Vanderbei, ''Linear Programming: Foundations and Extensions''. Springer, 2008. Available at: [http://link.springer.com/book/10.1007/978-0-387-74388-2 SpringerLink].
# S. Boyd, L. Vandenberghe, ''Convex Optimization''. Cambridge University Press, 2009. Available at: [http://www.stanford.edu/~boyd/cvxbook/ Stanford University].
# J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999. Available at: [http://link.springer.com/book/10.1007/b98874 SpringerLink].
# J. Birge, F. Louveaux, ''Introduction to Stochastic Programming''. Springer, 2011. Available at: [http://link.springer.com/book/10.1007/978-1-4614-0237-4 SpringerLink].
# A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009. Available at: [http://www2.isye.gatech.edu/~nemirovs/FullBookDec11.pdf Georgia Institute of Technology].
# L.A. Wolsey, ''Integer Programming''. Wiley, 1998.
# L.T. Biegler, I.E. Grossmann, A.W. Westerberg, ''Systematic Methods of Chemical Process Design''. Prentice Hall Press, 1997.
# L.T. Biegler, ''Nonlinear Programming: Concepts, Algorithms and Applications to Chemical Engineering''. SIAM, 2010.
# T.F. Edgar, D.M. Himmelblau, L.S. Lasdon, ''Optimization of Chemical Processes''. McGraw-Hill, 2001.
# T. Hastie, R. Tibshirani, J. Friedman, ''The Elements of Statistical Learning''. Springer, 2009. Available at: [http://statweb.stanford.edu/~tibs/ElemStatLearn/ Stanford University].
# K. Murphy, ''Machine Learning: A Probabilistic Perspective''. MIT Press, 2012.

Additional references for application examples:

* S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.
* R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.
* A. Abbass, "An Evolutionary Artificial Neural Networks Approach for Breast Cancer Diagnosis," ''Artificial Intelligence in Medicine'', vol. 25, no. 3, pp. 265-281, 2002.

Differential evolution

2024-12-16T02:59:24Z

Fall2024 Wiki Team26: Updated Differential Evolution

Author: Rohit Kumar (rk787) (ChemE 6800 Fall 2024)

Stewards: Nathan Preuss, Wei-Han Chen, Tianqi Xiao, Guoqing Hu

== Introduction ==
Differential Evolution (DE) is a robust and efficient optimization algorithm widely used for solving non-linear, non-differentiable, and multimodal optimization problems. It was first introduced by R. Storn and K. Price in their seminal work on global optimization over continuous spaces <ref>R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.</ref>, where they described DE as:-

''"A new heuristic approach for minimizing possibly nonlinear and non-differentiable continuous-space functions."''

A heuristic is any approximation method for a solution to a problem capable of producing satisfactory solutions within tolerable amounts of time when classic methods cannot find an exact answer or work quite slowly. Unlike the Gradient Descent algorithm, the optimization problem does not need to be differentiable.

The key idea is to generate a population of candidate solutions through successive generations using vector differences for deviations. This approach allows DE to adapt its search methodology to the objective function, making it particularly effective for non-linear, non-differentiable, and multimodal optimization problems.

The algorithm is renowned for its simplicity and efficiency. Its primary advantages include:

* Requires minimal parameter tuning.
* High robustness.
* Ability to avoid local optima and converge toward global solutions.

In DE, the currently evolving population of possible solutions is represented as a vector of real-valued numbers. The algorithm attempts to enhance these solutions through a series of iterations built upon three main steps:

* Mutation
* Crossover
* Selection

DE has become highly popular for applications in such fields as engineering design, machine learning, signal processing, and financial modeling. Despite its simplicity, Differential Evolution is competitive against many other optimization techniques, often performing better on challenging problems.

== Algorithm Discussion ==
Differential Evolution operates through three main steps: initialization, mutation, crossover, and selection, as outlined in optimization literature <ref>J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>.

=== Initialization ===
The initial population is generated randomly within the bounds of the search space: <math>x_{i,j} \sim U(x_j^{\text{min}}, x_j^{\text{max}})</math> where:

* <math>x_{i,j}</math> represents the <math>j^{\text{th}}</math> parameter of the <math>i^{\text{th}}</math> individual.
* <math>U(a, b)</math> is a uniform random distribution.

=== Mutation ===
A mutant vector <math>v_i</math> is generated by combining three randomly chosen vectors: <math>v_i = x_{r1} + F \cdot (x_{r2} - x_{r3})</math> where:

* <math>F</math> is the mutation factor.
* <math>x_{r1}, x_{r2}, x_{r3}</math> are distinct vectors from the population.

=== Crossover ===
A trial vector <math>u_i</math> is created by mixing the mutant vector <math>v_i</math> and the target vector <math>x_i</math>: <math>
u_{i,j} =
\begin{cases}
v_{i,j}, & \text{if } \text{rand}(0, 1) \leq CR \text{ or } j = j_{\text{rand}}, \\
x_{i,j}, & \text{otherwise}.
\end{cases}
</math> where:

* <math>CR</math> is the crossover probability.
* <math>j_{\text{rand}}</math> is a randomly chosen index.

=== Selection ===
The trial vector <math>u_i</math> replaces the target vector <math>x_i</math> in the population if it yields a better objective function value: <math>
x_i =
\begin{cases}
u_i, & \text{if } f(u_i) \leq f(x_i), \\
x_i, & \text{otherwise}.
\end{cases}
</math>

=== Pseudocode ===
The pseudocode of Differential Evolution is shown below:<pre>
Initialize population P randomly
While termination criterion not met:
For each individual x_i in P:
Generate mutant vector v_i
Perform crossover to create trial vector u_i
If f(u_i) < f(x_i): Replace x_i with u_i
Return the best solution
</pre>The pseudocode presented in this article is adapted from S. Das and P.N. Suganthan's survey on the state-of-the-art in DE <ref>S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.</ref>.

== Numerical Example ==
We aim to minimize the function <math>f(x) = x^2</math>, a common benchmark for optimization algorithms <ref>J. Birge, F. Louveaux, ''Introduction to Stochastic Programming''. Springer, 2011.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>. The example uses key principles of DE, including mutation and crossover, as described in foundational texts like R. Storn and K. Price’s work <ref>R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.</ref>.

=== Initial Setup ===
The initial population and fitness values are: <math>P = \{-7, 3, -2, 6, 8\}, \quad f(P) = \{49, 9, 4, 36, 64\}</math>

=== Iteration 1 ===
'''Individual 1''' (<math>x = -7</math>):

* Mutant vector: <math>v_1 = 3 + 0.8 \cdot (-2 - 8) = -5</math>
* Trial vector: <math display="inline">u_1 = -5</math>
* Fitness: <math>f(u_1) = 25</math>
* Since <math>f(u_1) < f(x_1)</math> (<math>25 < 49</math>), replace <math>x_1</math> with <math>u_1</math>
* Updated value: <math>x_1 = -5</math>

'''Individual 2''' (<math>x = 3</math>):

* Mutant vector: <math>v_2 = -2 + 0.8 \cdot (6 - 8) = -3.6</math>
* Trial vector: <math>u_2 = 3</math> (crossover fails)
* Fitness: <math>f(u_2) = 9</math>
* No replacement as <math>f(u_2) = f(x_2)</math>
* Updated value: <math>x_2 = 3</math>

'''Individual 3''' (<math>x = -2</math>):

* Mutant vector: <math>v_3 = 6 + 0.8 \cdot (8 - (-7)) = 18</math>
* Trial vector: <math>u_3 = -2</math> (crossover fails)
* Fitness: <math>f(u_3) = 4</math>
* No replacement as <math>f(u_3) = f(x_3)</math>
* Updated value: <math>x_3 = -2</math>

'''Individual 4''' (<math>x = 6</math>):

* Mutant vector: <math display="inline">v_4 = -7 + 0.8 \cdot (3 - (-2)) = -3</math>
* Trial vector: <math>u_4 = -3</math>
* Fitness: <math>f(u_4) = 9</math>
* Since <math>f(u_4) < f(x_4)</math> (<math>9 < 36</math>), replace <math>x_4</math> with <math>u_4</math>
* Updated value: <math>x_4 = -3</math>

'''Individual 5''' (<math>x = 8</math>):

* Mutant vector: <math>v_5 = 3 + 0.8 \cdot (-7 - (-2)) = -1</math>
* Trial vector: <math>u_5 = -1</math>
* Fitness: <math>f(u_5) = 1</math>
* Since <math>f(u_5) < f(x_5)</math> (<math>1 < 64</math>), replace <math>x_5</math> with <math>u_5</math>
* Updated value: <math>x_5 = -1</math>

After the first iteration, the updated population is: <math>P = \{-5, 3, -2, -3, -1\}, \quad f(P) = \{25, 9, 4, 9, 1\}</math>

=== Iterations 2–5 ===
The process is repeated for four more iterations. For each individual, the mutant vector, trial vector, and fitness are calculated. Individuals in the population are replaced whenever <math>f(u_i) < f(x_i)</math>. Over successive iterations, the population evolves toward the minimum.

=== Final Population ===
After five iterations, the population converges to: <math>P = \{0, 0, 0, 0, 0\}, \quad f(P) = \{0, 0, 0, 0, 0\}</math>

=== Visualization ===
The progression of the population during the Differential Evolution process is illustrated in the figures below. Each figure shows the distribution of the population at a specific iteration, demonstrating how the algorithm converges toward the global minimum of the function <math>f(x) = x^2</math>.

'''Figure 1''': Initial population distribution.
[[index.php?title=File:Ezgif-frame-001.png|link=File:Ezgif-frame-001.png|center|600x600px|Initial population distribution before any iterations.]]
'''Figure 2''': Population after Iteration 1.
[[index.php?title=File:Ezgif-frame-006.png|link=File:Ezgif-frame-006.png|center|600x600px|Population after the first iteration, showing improvement toward the global minimum.]]
'''Figure 3''': Population after Iteration 2.
[[index.php?title=File:Ezgif-frame-011.png|link=File:Ezgif-frame-011.png|center|600x600px|Population after the second iteration, with further refinement of solutions.]]
'''Figure 4''': Population after Iteration 4.
[[index.php?title=File:Ezgif-frame-018.png|link=File:Ezgif-frame-018.png|center|600x600px|Population after the fourth iteration, nearing convergence.]]
'''Figure 5''': Final population after Iteration 5.
[[index.php?title=File:Ezgif-frame-025.png|link=File:Ezgif-frame-025.png|alt=Final population converging to the global minimum at

x
<nowiki/>=
0

{\displaystyle x=0}

.|center|600x600px|Final population converging to the global minimum at <math>x = 0</math>.]]

== Applications ==
Differential Evolution has been applied across various domains, including:

* '''Engineering Design''': Applications include minimizing the weight of tension/compression springs, following the approaches outlined by R. Storn and K. Price <ref>R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.</ref><ref>S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.</ref>.
* '''Water Resource Management''': Optimization strategies for reservoirs, as seen in recent research <ref>J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>.
* '''Machine Learning''': Parameter optimization for neural networks <ref>T. Hastie, R. Tibshirani, J. Friedman, ''The Elements of Statistical Learning''. Springer, 2009.</ref>.
* '''Medical Applications''': DE’s effectiveness in radiotherapy optimization is discussed in Abbass (2002) <ref>A. Abbass, "An Evolutionary Artificial Neural Networks Approach for Breast Cancer Diagnosis," ''Artificial Intelligence in Medicine'', vol. 25, no. 3, pp. 265-281, 2002.</ref>.

=== Software Tools ===
Several software libraries and frameworks have integrated Differential Evolution into their core optimization processes:

* '''ALGLIB''': A powerful library in C++, C#, and Java, capable of solving complex constrained and unconstrained optimization problems.
* '''DEAP (Distributed Evolutionary Algorithms in Python)''': A Python framework for rapid prototyping and testing of evolutionary algorithms, including DE.
* '''ECJ''': An open-source evolutionary computation system implemented in Java, supporting a variety of algorithms, including DE.
* '''MOEA Framework''': A Java library dedicated to multi-objective optimization, offering robust DE implementations.

Differential Evolution’s adaptability and effectiveness have made it an indispensable tool for tackling challenging optimization problems across these diverse domains.

=== Conclusion ===
Differential Evolution’s ability to handle a wide range of optimization problems has made it a preferred tool in various domains <ref>J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999.</ref><ref>A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009.</ref>. Despite its strengths, opportunities for improvement exist, including faster convergence and hybrid models <ref>S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.</ref>.

Throughout its application in engineering design, water resource management, machine learning, and medical fields, DE has demonstrated its effectiveness in solving complex and constrained problems. Its capacity for global convergence and minimal parameter tuning requirements further enhance its appeal for real-world problems.

Despite its strengths, there are areas for future research and improvement:

* Enhancing the convergence speed of the algorithm.
* Developing hybrid approaches by combining DE with other optimization techniques.
* Extending its application to emerging domains such as quantum computing and autonomous systems.
* Improving parameter tuning methods and software implementations to make DE more accessible and efficient for practitioners.

In summary, Differential Evolution continues to be a reliable and powerful tool for solving challenging optimization problems and holds promise for further advancements in both methodology and applications.

=== References ===
Below is a list of references used to support the content in this article:

# R.J. Vanderbei, ''Linear Programming: Foundations and Extensions''. Springer, 2008. Available at: [http://link.springer.com/book/10.1007/978-0-387-74388-2 SpringerLink].
# S. Boyd, L. Vandenberghe, ''Convex Optimization''. Cambridge University Press, 2009. Available at: [http://www.stanford.edu/~boyd/cvxbook/ Stanford University].
# J. Nocedal, S.J. Wright, ''Numerical Optimization''. Springer, 1999. Available at: [http://link.springer.com/book/10.1007/b98874 SpringerLink].
# J. Birge, F. Louveaux, ''Introduction to Stochastic Programming''. Springer, 2011. Available at: [http://link.springer.com/book/10.1007/978-1-4614-0237-4 SpringerLink].
# A. Ben-Tal, L.E. Ghaoui, A. Nemirovski, ''Robust Optimization''. Princeton University Press, 2009. Available at: [http://www2.isye.gatech.edu/~nemirovs/FullBookDec11.pdf Georgia Institute of Technology].
# L.A. Wolsey, ''Integer Programming''. Wiley, 1998.
# L.T. Biegler, I.E. Grossmann, A.W. Westerberg, ''Systematic Methods of Chemical Process Design''. Prentice Hall Press, 1997.
# L.T. Biegler, ''Nonlinear Programming: Concepts, Algorithms and Applications to Chemical Engineering''. SIAM, 2010.
# T.F. Edgar, D.M. Himmelblau, L.S. Lasdon, ''Optimization of Chemical Processes''. McGraw-Hill, 2001.
# T. Hastie, R. Tibshirani, J. Friedman, ''The Elements of Statistical Learning''. Springer, 2009. Available at: [http://statweb.stanford.edu/~tibs/ElemStatLearn/ Stanford University].
# K. Murphy, ''Machine Learning: A Probabilistic Perspective''. MIT Press, 2012.

Additional references for application examples:

* S. Das, P.N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," ''IEEE Transactions on Evolutionary Computation'', vol. 15, no. 1, pp. 4-31, 2011.
* R. Storn, K. Price, "Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," ''Journal of Global Optimization'', vol. 11, pp. 341-359, 1997.
* A. Abbass, "An Evolutionary Artificial Neural Networks Approach for Breast Cancer Diagnosis," ''Artificial Intelligence in Medicine'', vol. 25, no. 3, pp. 265-281, 2002.

File:Ezgif-frame-025.png

2024-12-16T02:32:07Z

Fall2024 Wiki Team26:

Figure 5: Iteration 5: Population converged at the global minimum.

File:Ezgif-frame-018.png

2024-12-16T02:31:43Z

Fall2024 Wiki Team26:

Figure 4: Iteration 4: Approaching the global minimum.

File:Ezgif-frame-011.png

2024-12-16T02:31:15Z

Fall2024 Wiki Team26:

Figure 3: Iteration 3: Further refinement of population.

File:Ezgif-frame-006.png

2024-12-16T02:30:35Z

Fall2024 Wiki Team26:

Figure 2: Iteration 2: Population evolving toward the minimum.

File:Ezgif-frame-001.png

2024-12-16T02:16:25Z

Fall2024 Wiki Team26:

Iteration 1