McCormick envelopes

Author: Susan Urban (smu29) (SYSEN 5800 Fall 2021)

Introduction

Optimization of a non-convex function f(x) is challenging since it may have multiple locally optimal solutions or no solution and it can take a significant amount of time, computing resources, and effort to determine if the solution is global or the problem has no feasible solution. Different techniques are used to address this challenge depending on the characteristics of the problem.

One technique used for a given non-convex function is the identification of a concave envelope and a convex envelope. The concave envelope, and respectively the convex envelope, is the concave over-estimator and convex under-estimator for the given function providing the tighest fit to the given function. The envelope surrounds the given function, like an envelope encloses a letter, and limits the feasible solution space the most in comparison to other concave over-estimators and convex under-estimators.

McCormick Envelopes: Theory, Methodology and Algorithmic Discussions

The McCormick Envelope² is a type of convex relaxation used for optimization of bilinear (e.g., x*y, x+y) non-linear programming (NLP) problems.

Each bilinear term is replaced with a new variable and four sets of constraints are added. The non-linear programming is converted to a relaxed convex linear programming which can be more easily solved.

The LP solution gives a lower bound and any feasible solution gives an upper bound.

As noted by Scott et al⁵, McCormick envelopes are effective since they are recursive, can be applied to a variety of applications, and are typically stronger than those resulting from convexification or linearization procedures.

The following is a derivation of the McCormick Envelopes:

$Let\ w=xy$

$x^{L}\leq x\leq x^{U}$

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle y^{L}\leq y\leq y^{U}}

where Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle x^{L}, x^{U}, y^{L}, y^{U} } are upper and lower bound values for Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle x} and Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle y} , respectively.

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle a=\left (x-x^{L} \right )}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle b=\left (y-y^{L} \right )}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle a * b\geq 0}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle a * b=\left ( x-x^{L} \right )\left ( y-y^{L} \right )=xy-x^{L}y-xy^{L}+x^{L}y^{L}\geq 0 }

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w\geq x^{L}y+xy^{L}-x^{L}y^{L}}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle a=\left ( x^{U}-x \right )}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle b = \left ( y^{U} -y\right )}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w\geq x^{U}y+xy^{U}-x^{U}y^{U}}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle a=\left ( x^{U}-x\right )}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle b=\left ( y-y^{L} \right )}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w\leq x^{U}y+xy^{L}-x^{U}y^{L}}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle a=\left ( x-x^{L} \right )}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle b=\left ( y^{U}-y \right )}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w\leq xy^{U}+x^{L}y-x^{L}y^{U}}

The under-estimators of the function are represented by:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w\geq x^{L}y+xy^{L}-x^{L}y^{L}}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w\geq x^{U}y+xy^{U}-x^{U}y^{U}}

The over-estimators of the function are represented by:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w\leq x^{U}y+xy^{L}-x^{U}y^{L}}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w\leq xy^{U}+x^{L}y-x^{L}y^{U}}

The following shows the relaxation of a non-convex problem:

Original non-convex problem:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \min Z= \textstyle \sum_{i=1} \sum_{j=1} c_{i,j}x_i x_j +g_0(x) }

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle s.t. \sum_{i=1}\sum_{j=1} c_{i,j}^l x_i x_j + g_l (x) \leq 0,\forall l \in L }

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle x^ L \leq x \leq x^U }

Replacing Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle u_{i,j} = x_i x_j}

we obtain a relaxed, convex problem:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \min Z= \textstyle \sum_{i=1} \sum_{j=1} c_{i,j} u_{i,j} +g_0(x)}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle s.t. \sum_{i=1}\sum_{j=1} c_{i,j}^l u_{i,j}+ g_l (x) \leq 0,\forall l \in L}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle u_{i,j} \geq x_i^L x_j + x_i x_j^L - {x_i}^L {x_j}^L, \forall i,j \ }

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle u_{i,j} \geq x_i^U x_j + x_i x_j^U - {x_i}^U {x_j}^U, \forall i,j \ }

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle u_{i,j} \geq x_i^L x_j + x_i x_j^U - {x_i}^L {x_j}^U, \forall i,j \ }

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle u_{i,j} \geq x_i^U x_j + x_i x_j^L - {x_i}^U {x_j}^L, \forall i,j \ }

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle x^L \leq x \leq x^U, \ \ u^L \leq u \leq u^U}

Good lower and upper bounds focus and minimize the feasible solution space; they reduce the number of iterations to find the optimal solution.

As discussed by Hazaji⁶, global optimization solvers focus initially on optimizing the lower and upper bounds, and when necessary, focus on domain partitioning. By dividing the domain of a given variable into partitions or smaller regions, the solver is able to tailor and further tighten the convex relaxations of each partition.

Example: Numerical

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle min \ Z = xy + 6x + y}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle s.t. \ xy \leq 18}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle 0 \leq x \leq 10 }

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle 0 \leq y \leq 2}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle Let\ w = xy }

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle min \ Z = -w + 6x + y }

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle s.t. \ w \leq 18}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w \geq 0 }

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w \geq 10y + 2x - 20 }

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w \leq 10y }

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w \leq 2x }

Using GAMS, the solution is z= -24, x=6, y=2.

GAMS code sample:

variable z;

positive variable x, y, w;

equations obj, c1, c2, c3, c4, c5 ;

obj.. z =e= -w -2*x ;

c1.. w =l= 12 ;

c2.. w =g= 0;

c3.. w =g= 6*y +3*x -18;

c4.. w =l= 6*y;

c5.. w =l= 3*x;

x.up = 10;

x.lo =0;

y.up = 2;

y.lo = 0;

model course5800 /all/;

option mip = baron;

option optcr = 0;

solve course5800 minimizing z using mip ;

Application

Bilinear functions occur in numerous engineering and natural science applications, including the following :

Computer vision 9

Super resolution imaging ¹⁰

Conclusion

Since non-convex NLPs are challenging to solve and may require a significant amount of time, computing resources, and effort to determine if the solution is global or the problem has no feasible solution. McCormick Envelopes provide a relaxation technique for bilinear non-convex nonlinear programming problems. McCormick Envelopes provide a straightforward technique that may be applied to bilinear expressions from multiple application areas.

References

Dombrowski, J. (2015, June 7). Northwestern University Open Text Book on Process Optimization, McCormick Envelopes Retrieved from https://optimization.mccormick.northwestern.edu/index.php/McCormick_envelopes
McCormick, Garth P. Computability of Global Solutions To Factorable Nonconvex Solutions: Part I: Convex Underestimating Problems
Scott, J. K. Stuber, M. D. & Barton, P. I. (2011). Generalized McCormick Relaxations. Journal of Global Optimization, Vol. 51, Issue 4, 569-606 doi: 10.1007/s10898-011-9664-7
Hijazi, H., Perspective Envelopes for Bilinear Functions, unpublished manuscript, retrieved from: Perspective Envelopes for Bilinear Functions
Androulakis, I., Maranas, C., Floudas, C.: alphabb: A global optimization method for general constrained nonconvex problems. Journal of Global Optimization 7(4), 337{363 (1995)
Smith, E., Pantelides, C.: A symbolic reformulation/spatial branch-and-bound algorithm for the global optimisation of nonconvex fMINLPsg. Computers & Chemical Engineering 23(4), 457 { 478 (1999)
Geunes, J., Pardalos, P.: Supply chain optimization, vol. 98. Springer Science & BusinessMedia (2006)
Nahapetyan, A.: Bilinear programming: applications in the supply chain management bilinear programming: Applications in the supply chain management. In: C.A. Floudas, P.M. Pardalos (eds.) Encyclopedia of Optimization, pp. 282{288. Springer US (2009)
Chandraker, M. & Kriegman, D. (n.d.): Globally Optimal Bilinear Programming for Computer Vision Applications. University of San Diego, CA. Retrieved from: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwis69Say7H0AhXWp3IEHVGLBW4QFnoECC4QAQ&url=http%3A%2F%2Fvision.ucsd.edu%2F~manu%2Fpdf%2Fcvpr08_bilinear.pdf&usg=AOvVaw1h-cpWO81s41howVxYKq7F
Gronski, J. (2019). Non-Convex Optimization and Applications to Bilinear Programming and Super-Resolution Imaging. University of Colorado. Retrieved from: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwif3qO017H0AhWNknIEHebkAhMQFnoECA0QAQ&url=https%3A%2F%2Fscholar.colorado.edu%2Fdownloads%2Fwd375w61z&usg=AOvVaw09lnJJoZX3i_wwBGHin9LK