https://optimization.cbe.cornell.edu/api.php?action=feedcontributions&user=Bramado&feedformat=atom Cornell University Computational Optimization Open Textbook - Optimization Wiki - User contributions [en] 2021-06-24T03:06:44Z User contributions MediaWiki 1.35.0 https://optimization.cbe.cornell.edu/index.php?title=Optimization_with_absolute_values&diff=2473 Optimization with absolute values 2020-12-13T15:21:40Z <p>Bramado: /* Absolute Values in Nonlinear Optimization Problems */</p> <hr /> <div>Authors: Matthew Chan (mdc297), Yilian Yin (yy896), Brian Amado (ba392), Peter (pmw99), Dewei Xiao (dx58) - SYSEN 5800 Fall 2020<br /> <br /> Steward: Fengqi You<br /> <br /> == Introduction ==<br /> Absolute values can make it relatively difficult to determine the optimal solution when handled without first converting to standard form. This conversion of the objective function is a good first step in solving optimization problems with absolute values. As a result, one can go on to solve the problem using linear programing techniques. With the addition of a new variable (ex: &lt;math&gt;\textstyle X^a &lt;/math&gt;) in the objective function the problem is considered nonlinear. Additional constraints must be added to find the optimal solution.<br /> <br /> == Method ==<br /> <br /> === Defining Absolute Values ===<br /> An absolute value of a real number can be described as its distance away from zero, or the non-negative magnitude of the number. &lt;ref&gt; Mendelson, Elliott, Schaum's Outline of Beginning Calculus, McGraw-Hill Professional, 2008. https://books.google.com/books?id=A8hAm38zsCMC&amp;pg=PA2#v=onepage&amp;q&amp;f=false &lt;/ref&gt; Thus,<br /> <br /> &lt;math&gt;\displaystyle |x|={\begin{cases}-x,&amp;{\text{if }}x&lt;0\\x,&amp;{\text{if }}x\geq 0\end{cases}}&lt;/math&gt;<br /> <br /> Absolute values can exist in linear optimization problems in two primary instances: in constraints and in the objective function. &lt;ref&gt; &quot;Absolute Values.&quot; ''lp_solve'', http://lpsolve.sourceforge.net/. Accessed 20 Nov. 2020. &lt;/ref&gt;<br /> <br /> === Absolute Values in Constraints ===<br /> Within constraints, absolute value relations can be transformed into one of the following forms:<br /> <br /> &lt;math&gt; \begin{align}<br /> |X| &amp;= 0 \\<br /> |X| &amp;\le C \\<br /> |X| &amp;\ge C<br /> \end{align} &lt;/math&gt;<br /> <br /> Where &lt;math&gt;\textstyle X&lt;/math&gt; is a linear combination (&lt;math&gt;\textstyle ax_1 + bx_2 + ...&lt;/math&gt; where &lt;math&gt;\textstyle a, b&lt;/math&gt; are constants) and &lt;math&gt;\textstyle C&lt;/math&gt; is a constant &lt;math&gt;\textstyle &gt; 0&lt;/math&gt;.<br /> <br /> ==== Form when &lt;math&gt;\displaystyle |X| = 0&lt;/math&gt; ====<br /> In this form, the only possible solution is if &lt;math&gt;\displaystyle X = 0&lt;/math&gt; simplifying the constraint. Note that this solution also occurs if the constraint is in the form &lt;math&gt;\displaystyle |X| \le 0&lt;/math&gt; due to the same conclusion that the only possible solution is &lt;math&gt;\textstyle X = 0&lt;/math&gt;.<br /> <br /> ==== Form when &lt;math&gt;\displaystyle |X| \le C&lt;/math&gt; ====<br /> The second form a linear constraint can exist in is &lt;math&gt;\displaystyle |X|\leq C&lt;/math&gt;. In this case, an equivalent feasible solution can be described by splitting the constraint into two:<br /> <br /> &lt;math&gt; \begin{align}<br /> X &amp;\leq C \\<br /> -X &amp;\leq C<br /> \end{align} &lt;/math&gt;<br /> <br /> The solution can be understood visually since &lt;math&gt;\textstyle X&lt;/math&gt; must lie between &lt;math&gt;\textstyle -C&lt;/math&gt; and &lt;math&gt;\textstyle C&lt;/math&gt;, as shown below:<br /> <br /> [[File:Number Line X Less Than C.png|none|thumb]]<br /> <br /> ==== Form when &lt;math&gt;\displaystyle |X| \ge C&lt;/math&gt; ====<br /> Visually, the solution space for the last form is the complement of the second solution above, resulting in the following representation:[[File:Number Line for X Greater Than C.png|none|thumb]]In expression form, the solutions can be written as:<br /> <br /> &lt;math&gt; \begin{align}<br /> X &amp;\geq C \\<br /> -X &amp;\geq C<br /> \end{align} &lt;/math&gt;<br /> <br /> As seen visually, the feasible region has a gap and thus non-convex. The expressions also make it impossible for both to simultaneously hold true. This means that it is not possible to transform constraints in this form to linear equations. &lt;ref&gt; ''Optimization Methods in Management Science / Operations Research.'' Massachusetts Institute of Technology, Spring 2013, https://ocw.mit.edu/courses/sloan-school-of-management/15-053-optimization-methods-in-management-science-spring-2013/tutorials/MIT15_053S13_tut04.pdf. Accessed 20 Nov. 2020. &lt;/ref&gt; <br /> <br /> An approach to reach a solution for this particular case exists in the form of Mixed-Integer Linear Programming, where only one of the equations above is “active”.<br /> <br /> The inequality can be reformulated into the following:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X + N*Y \ge C \\<br /> -&amp;X + N*(1-Y) \ge C \\<br /> &amp;Y = 0, 1 <br /> \end{align} &lt;/math&gt;<br /> <br /> With this new set of constraints, a large constant &lt;math&gt;\textstyle N&lt;/math&gt; is introduced, along with a binary variable &lt;math&gt;\textstyle Y&lt;/math&gt;. So long as &lt;math&gt;\textstyle N&lt;/math&gt; is sufficiently larger than the upper bound of &lt;math&gt;\textstyle X + C&lt;/math&gt;, the large constant multiplied with the binary variable ensures that one of the constraints must be satisfied. For instance, if &lt;math&gt;\textstyle Y = 0&lt;/math&gt;, the new constraints will resolve to:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X \ge C \\<br /> -&amp;X + N \ge C<br /> \end{align} &lt;/math&gt;<br /> <br /> Since &lt;math&gt;\textstyle N&lt;/math&gt; is sufficiently large, the latter constraint will always be satisfied, leaving only one relation active: &lt;math&gt;\textstyle X \ge C&lt;/math&gt;. Functionally, this allows for the XOR logical operation of &lt;math&gt;\textstyle X \geq C&lt;/math&gt; and &lt;math&gt;\textstyle -X \geq C&lt;/math&gt;.<br /> <br /> === Absolute Values in Objective Functions ===<br /> In objective functions, to leverage transformations of absolute functions, all constraints must be linear.<br /> <br /> Similar to the case of absolute values in constraints, there are different approaches to the reformation of the objective function, depending on the satisfaction of sign constraints. The satisfaction of sign constraints is when the coefficient signs of the absolute terms must all be either:<br /> <br /> * Positive for a minimization problem<br /> * Negative for a maximization problem<br /> <br /> ==== Sign Constraints are Satisfied ====<br /> At a high level, the transformation works similarly to the second case of absolute value in constraints – aiming to bound the solution space for the absolute value term with a new variable, &lt;math&gt;\textstyle Z&lt;/math&gt;.<br /> <br /> If &lt;math&gt;\textstyle |X|&lt;/math&gt; is the absolute value term in our objective function, two additional constraints are added to the linear program:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X\leq Z \\<br /> -&amp;X\leq Z <br /> \end{align} &lt;/math&gt;<br /> <br /> The &lt;math&gt;\textstyle |X|&lt;/math&gt; term in the objective function is then replaced by &lt;math&gt;\textstyle Z&lt;/math&gt;, relaxing the original function into a collection of linear constraints.<br /> <br /> ==== Sign Constraints are Not Satisfied ====<br /> In order to transform problems where the coefficient signs of the absolute terms do not fulfill the conditions above, a similar conclusion is reached to that of the last case for absolute values in constraints – the use of integer variables is needed to reach an LP format. <br /> <br /> The following constraints need to be added to the problem:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X + N*Y \ge Z \\<br /> -&amp;X + N*(1-Y) \ge Z \\<br /> &amp;X \le Z \\<br /> -&amp;X \le Z \\<br /> &amp;Y = 0, 1<br /> \end{align} &lt;/math&gt;<br /> <br /> Again, &lt;math&gt;\textstyle N&lt;/math&gt; is a large constant, &lt;math&gt;\textstyle Z&lt;/math&gt; is a replacement variable for &lt;math&gt;\textstyle |X|&lt;/math&gt; in the objective function, and &lt;math&gt;\textstyle Y&lt;/math&gt; is a binary variable. The first two constraints ensure that one and only one constraint is active while the other will be automatically satisfied, following the same logic as above. The third and fourth constraints ensure that &lt;math&gt;\textstyle Z&lt;/math&gt; must be equal to &lt;math&gt;\textstyle |X|&lt;/math&gt; and has either a positive or negative value. For instance, for the case of &lt;math&gt;\textstyle Y = 0&lt;/math&gt;, the new constraints will resolve to:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X \ge Z \\<br /> -&amp;X + N \ge Z \\<br /> &amp;X \le Z \\<br /> -&amp;X \le Z<br /> \end{align} &lt;/math&gt;<br /> <br /> As &lt;math&gt;\textstyle N&lt;/math&gt; is sufficiently large (&lt;math&gt;\textstyle N&lt;/math&gt; must be at least &lt;math&gt;\textstyle 2|X|&lt;/math&gt; for this approach), the second constraint must be satisfied. Since &lt;math&gt;\textstyle Z&lt;/math&gt; is non-negative, the fourth constraint must also be satisfied. The remaining constraints, &lt;math&gt;\textstyle X \ge Z&lt;/math&gt; and &lt;math&gt;\textstyle X \le Z&lt;/math&gt; can only be satisfied when &lt;math&gt;\textstyle Z = X&lt;/math&gt; and is of non-negative signage. Together, these constraints will allow for the selection of the largest &lt;math&gt;\textstyle |X|&lt;/math&gt; for maximization problems (or smallest for minimization problems).<br /> <br /> === Absolute Values in Nonlinear Optimization Problems ===<br /> The addition of a new variable &lt;math&gt; (X_a) &lt;/math&gt; to an objective function with absolute value quantities forms a nonlinear optimization problem. The absolute value quantities would require that the problem be reformatted before proceeding. Additional constraints must be added to account for the added variable.<br /> <br /> ==Numerical Example==<br /> '''Example when All Sign Constraints are Satisfied'''<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{2|x_1| + 3|x_2| + |x_3|} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14<br /> \end{align}&lt;/math&gt;<br /> <br /> The absolute value quantities will be replaced with single variables:<br /> <br /> &lt;math&gt;|x_1| = U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt;|x_2| = U_2&lt;/math&gt;<br /> <br /> &lt;math&gt;|x_3| = U_3&lt;/math&gt;<br /> <br /> We must introduce additional constraints to ensure we do not lose any information by doing this substitution:<br /> <br /> &lt;math&gt; -U_1 \le x_1 \le U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_2 \le x_2 \le U_2 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_3 \le x_3 \le U_3 &lt;/math&gt;<br /> <br /> The problem has now been reformulated as a linear programming problem that can be solved normally:<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{ 2U_1 + 3U_2 + U_3} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14 \\<br /> -&amp;U_1 \le x_1 \le U_1 \\<br /> -&amp;U_2 \le x_2 \le U_2 \\<br /> -&amp;U_3 \le x_3 \le U_3 <br /> \end{align}&lt;/math&gt;<br /> <br /> The optimum value for the objective function is &lt;math&gt;3.5&lt;/math&gt;, which occurs when &lt;math&gt;x_1 = 0 &lt;/math&gt; and &lt;math&gt;x_2 = 0 &lt;/math&gt; and &lt;math&gt;x_3 = 3.5 &lt;/math&gt;.<br /> <br /> '''Example when Sign Constraints are not Satisfied'''<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{2|x_1| + 3|x_2| - |x_3|} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14<br /> \end{align}&lt;/math&gt;<br /> <br /> The absolute value quantities will be replaced with single variables:<br /> <br /> &lt;math&gt;|x_1| = U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt;|x_2| = U_2&lt;/math&gt;<br /> <br /> &lt;math&gt;|x_3| = U_3&lt;/math&gt;<br /> <br /> We must introduce additional constraints to ensure we do not lose any information by doing this substitution:<br /> <br /> &lt;math&gt; \begin{align}<br /> -&amp;U_1 \le x_1 \le U_1 \\<br /> -&amp;U_2 \le x_2 \le U_2 \\<br /> &amp;x_3 + M*Y \ge U_3 \\<br /> -&amp;x_3 + M*(1-Y) \ge U_3 \\<br /> &amp;x_3 \le U_3 \\<br /> -&amp;x_3 \le U_3 \\<br /> &amp;Y = 0,1<br /> \end{align}&lt;/math&gt;<br /> <br /> The problem has now been reformulated as a linear programming problem that can be solved normally:<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{ 2U_1 + 3U_2 - U_3} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14 \\<br /> -&amp;U_1 \le x_1 \le U_1 \\<br /> -&amp;U_2 \le x_2 \le U_2 \\<br /> &amp;x_3 + M*Y \ge U_3 \\<br /> -&amp;x_3 + M*(1-Y) \ge U_3 \\<br /> &amp;x_3 \le U_3 \\<br /> -&amp;x_3 \le U_3 \\<br /> &amp;Y = 0,1 <br /> \end{align}&lt;/math&gt;<br /> <br /> The optimum value for the objective function is &lt;math&gt;-3.5&lt;/math&gt;, which occur when &lt;math&gt;x_1 = 0 &lt;/math&gt; and &lt;math&gt;x_2 = 0 &lt;/math&gt; and &lt;math&gt;x_3 = 3.5 &lt;/math&gt;.<br /> <br /> == Applications ==<br /> <br /> <br /> Consider the problem &lt;math&gt;Ax=b; \quad max \quad z= x c,jx,i&lt;/math&gt;. This problem cannot, in general, be solved with the simplex method. The problem has a simplex method solution (with unrestricted basis entry) only if c, are nonpositive (non-negative for minimizing problems).<br /> <br /> The primary application of absolute-value functionals in linear programming has been for absolute-value or L(i)-metric regression analysis. Such application is always a minimization problem with all C(j) equal to 1 so that the required conditions for valid use of the simplex method are met. <br /> <br /> By reformulating the original problem into a Mixed-Integer Linear Program (MILP), we can utilize known programs to solve for the optimal solution(s). <br /> <br /> === Application in Financial: Portfolio Selection===<br /> Under this topic, the same tricks played in the Numerical Example section to perform '''Reduction to a Linear Programming Problem''' will be applied here again, to reform the problem into a MILP in order to solve the problem. An example is given as below. <br /> <br /> <br /> <br /> A portfolio is determined by what fraction of one's assets to put into each investment. It can be denoted as a collection of nonnegative numbers &lt;math&gt;\textstyle x_j&lt;/math&gt;, where &lt;math&gt; j = 1, 2,...,n &lt;/math&gt;. Because each &lt;math&gt; \textstyle x_j &lt;/math&gt;stands for a portion of the assets, it sums to one. In order to get a highest reward through finding a right mix of assets, let &lt;math&gt;\mu&lt;/math&gt;, the positive parameter, denote the importance of risk relative to the return, and &lt;math&gt;/textstyle Rj&lt;/math&gt; denote the return in the next time period on investment &lt;math&gt;j, j = 1, 2,..., n&lt;/math&gt;. The total return one would obtain from the investment is &lt;math&gt;R = \sum_{j}\!x_j\!R_j &lt;/math&gt;. The expected return is &lt;math&gt;\mathbb{E}\!R = \sum_{j}\!x_j\mathbb{E}\!R_j &lt;/math&gt;. And the Mean Absolute Deviation from the Mean (MAD) is &lt;math&gt;\mathbb{E}\left\vert \!R - \mathbb{E}\!R \right\vert = \mathbb{E}\left\vert \sum_{j}\!x_j\tilde{R}_j \right\vert &lt;/math&gt;. <br /> <br /> maximize &lt;math display=&quot;inline&quot;&gt;\mu\sum_j\!x_j\mathbb{E}\!R_j - \mathbb{E}\left\vert \sum_j \!x_j\tilde{R}_j \right\vert &lt;/math&gt;<br /> <br /> subject to &lt;math&gt;\sum_j\!x_j = 1&lt;/math&gt;<br /> <br /> &lt;math&gt;x_j \geq 0&lt;/math&gt; &lt;math&gt; j = 1,2,..n.&lt;/math&gt;<br /> <br /> where &lt;math&gt;\tilde{R}_j = \!R_j - \mathbb{E}\!R_j &lt;/math&gt;<br /> <br /> <br /> <br /> Very obviously, this problem is not a linear programming problem yet. Similar to the numerical example showed above, the right thing to do is to replace each absolute value with a new variable and impose inequality constraints to ensure that the new variable is the appropriate absolute value once an optimal value is obtained. To simplify the program, an average of the historical returns can be taken in order to get the mean expected return: &lt;math&gt;r_j = \mathbb{E}\!R_j = \left ( \frac{1}{T} \right ) \sum_{t=1}^T \!R_j(t)<br /> &lt;/math&gt;. Thus the objective function is turned into: &lt;math&gt;\mu\sum_{j}\!x_j\!r_j - \left ( \frac{1}{T} \right ) \sum_{t=1}^T\left\vert \sum_{j} \!x_j \bigl(R_j (t) - \!r_j\bigr) \right\vert <br /> &lt;/math&gt;<br /> <br /> Now, replace &lt;math&gt;\left\vert \sum_{j} \!x_j \bigl(R_j (t) - \!r_j\bigr) \right\vert <br /> &lt;/math&gt; with a new variable &lt;math&gt;y_t<br /> &lt;/math&gt;and thus the problem can be rewrote as:<br /> <br /> <br /> maximize &lt;math&gt;\mu \sum_j \!x_j\!r_j - \left ( \frac{1}{T} \right ) \sum_{t=1}^T \!y_t<br /> <br /> &lt;/math&gt;<br /> <br /> subject to &lt;math&gt;-\!y_t \leq \sum_{j} \!x_j \bigl(R_j (t) - \!r_j\bigr) \leq y_t <br /> &lt;/math&gt;. t = 1, 2,...,T<br /> <br /> where &lt;math&gt;\sum_j \!x_j = 1<br /> <br /> &lt;/math&gt;<br /> <br /> &lt;math&gt;x_j\geq 0<br /> <br /> &lt;/math&gt;. j = 1, 2,...,n<br /> <br /> &lt;math&gt;y_t \geq 0<br /> <br /> &lt;/math&gt;. t = 1, 2,...,T<br /> <br /> <br /> So finally, after some simplifications methods and some tricks applied, the original problem is converted into a linear programming which is easier to be solved further.<br /> <br /> <br /> ===Data Transfer Rate===<br /> Faster-than-nyquist, or FTNS, is a framework to transmit signals beyond the Nyquist rate. The refence to this section proposed a 24.7% faster symbol rate by utilizing Sum-of-Absolute-Values optimization.<br /> <br /> The initial model is defined as follows:<br /> &lt;math&gt;\displaystyle x_0 (t) = \sum^N_{n=1} x_{n,0} h_n (t), t \in [0,T] &lt;/math&gt;<br /> <br /> where t ∈ R denotes the continuous time index, N ∈ N is the number of transmitted symbols in each transmission period, T &gt; 0 is the interval of one period, xn,0 ∈ {+1, −1} are independent and identically distributed (i.i.d.) binary symbols [i.e., binary phase shift keying (BPSK)], and hn (t) (n = 1,...,N) are the modulation pulses.<br /> <br /> Reformulated as a convex optimization problem and repeating Newton’s method with absolute values, the solution approximates can be achieved.<br /> &lt;math&gt;\displaystyle \min_{z \in R^N} (\lambda \Vert y - Hz \Vert^2_2 + \frac{1}{2} \Vert z - 1_N \Vert_1 + \frac{1}{2} \Vert z + 1_N \Vert_1 ) &lt;/math&gt;<br /> <br /> <br /> == Conclusion ==<br /> The presence of an absolute value within the objective function prevents the use of certain optimization methods. Solving these problems requires that the function be manipulated in order to continue with linear programming techniques like the simplex method.<br /> <br /> == References ==<br /> &lt;references /&gt;<br /> <br /> <br /> <br /> <br /> # https://www.ise.ncsu.edu/fuzzy-neural/wp-content/uploads/sites/9/2019/08/LP-Abs-Value.pdf<br /> # Vanderbei R.J. (2008) Financial Applications. In: Linear Programming. International Series in Operations Research &amp; Management Science, vol 114. Springer, Boston, MA. &lt;nowiki&gt;https://doi.org/10.1007/978-0-387-74388-2_13&lt;/nowiki&gt; https://link.springer.com/chapter/10.1007/978-0-387-74388-2_13<br /> # Sasahara, Hampei &amp; Hayashi, Kazunori &amp; Nagahara, Masaaki. (2016). Symbol Detection for Faster-Than-Nyquist Signaling by Sum-of-Absolute-Values Optimization. IEEE Signal Processing Letters. PP. 1-1. 10.1109/LSP.2016.2625839. https://www.researchgate.net/publication/309745511_Symbol_Detection_for_Faster-Than-Nyquist_Signaling_by_Sum-of-Absolute-Values_Optimization</div> Bramado https://optimization.cbe.cornell.edu/index.php?title=Optimization_with_absolute_values&diff=2472 Optimization with absolute values 2020-12-13T15:19:43Z <p>Bramado: </p> <hr /> <div>Authors: Matthew Chan (mdc297), Yilian Yin (yy896), Brian Amado (ba392), Peter (pmw99), Dewei Xiao (dx58) - SYSEN 5800 Fall 2020<br /> <br /> Steward: Fengqi You<br /> <br /> == Introduction ==<br /> Absolute values can make it relatively difficult to determine the optimal solution when handled without first converting to standard form. This conversion of the objective function is a good first step in solving optimization problems with absolute values. As a result, one can go on to solve the problem using linear programing techniques. With the addition of a new variable (ex: &lt;math&gt;\textstyle X^a &lt;/math&gt;) in the objective function the problem is considered nonlinear. Additional constraints must be added to find the optimal solution.<br /> <br /> == Method ==<br /> <br /> === Defining Absolute Values ===<br /> An absolute value of a real number can be described as its distance away from zero, or the non-negative magnitude of the number. &lt;ref&gt; Mendelson, Elliott, Schaum's Outline of Beginning Calculus, McGraw-Hill Professional, 2008. https://books.google.com/books?id=A8hAm38zsCMC&amp;pg=PA2#v=onepage&amp;q&amp;f=false &lt;/ref&gt; Thus,<br /> <br /> &lt;math&gt;\displaystyle |x|={\begin{cases}-x,&amp;{\text{if }}x&lt;0\\x,&amp;{\text{if }}x\geq 0\end{cases}}&lt;/math&gt;<br /> <br /> Absolute values can exist in linear optimization problems in two primary instances: in constraints and in the objective function. &lt;ref&gt; &quot;Absolute Values.&quot; ''lp_solve'', http://lpsolve.sourceforge.net/. Accessed 20 Nov. 2020. &lt;/ref&gt;<br /> <br /> === Absolute Values in Constraints ===<br /> Within constraints, absolute value relations can be transformed into one of the following forms:<br /> <br /> &lt;math&gt; \begin{align}<br /> |X| &amp;= 0 \\<br /> |X| &amp;\le C \\<br /> |X| &amp;\ge C<br /> \end{align} &lt;/math&gt;<br /> <br /> Where &lt;math&gt;\textstyle X&lt;/math&gt; is a linear combination (&lt;math&gt;\textstyle ax_1 + bx_2 + ...&lt;/math&gt; where &lt;math&gt;\textstyle a, b&lt;/math&gt; are constants) and &lt;math&gt;\textstyle C&lt;/math&gt; is a constant &lt;math&gt;\textstyle &gt; 0&lt;/math&gt;.<br /> <br /> ==== Form when &lt;math&gt;\displaystyle |X| = 0&lt;/math&gt; ====<br /> In this form, the only possible solution is if &lt;math&gt;\displaystyle X = 0&lt;/math&gt; simplifying the constraint. Note that this solution also occurs if the constraint is in the form &lt;math&gt;\displaystyle |X| \le 0&lt;/math&gt; due to the same conclusion that the only possible solution is &lt;math&gt;\textstyle X = 0&lt;/math&gt;.<br /> <br /> ==== Form when &lt;math&gt;\displaystyle |X| \le C&lt;/math&gt; ====<br /> The second form a linear constraint can exist in is &lt;math&gt;\displaystyle |X|\leq C&lt;/math&gt;. In this case, an equivalent feasible solution can be described by splitting the constraint into two:<br /> <br /> &lt;math&gt; \begin{align}<br /> X &amp;\leq C \\<br /> -X &amp;\leq C<br /> \end{align} &lt;/math&gt;<br /> <br /> The solution can be understood visually since &lt;math&gt;\textstyle X&lt;/math&gt; must lie between &lt;math&gt;\textstyle -C&lt;/math&gt; and &lt;math&gt;\textstyle C&lt;/math&gt;, as shown below:<br /> <br /> [[File:Number Line X Less Than C.png|none|thumb]]<br /> <br /> ==== Form when &lt;math&gt;\displaystyle |X| \ge C&lt;/math&gt; ====<br /> Visually, the solution space for the last form is the complement of the second solution above, resulting in the following representation:[[File:Number Line for X Greater Than C.png|none|thumb]]In expression form, the solutions can be written as:<br /> <br /> &lt;math&gt; \begin{align}<br /> X &amp;\geq C \\<br /> -X &amp;\geq C<br /> \end{align} &lt;/math&gt;<br /> <br /> As seen visually, the feasible region has a gap and thus non-convex. The expressions also make it impossible for both to simultaneously hold true. This means that it is not possible to transform constraints in this form to linear equations. &lt;ref&gt; ''Optimization Methods in Management Science / Operations Research.'' Massachusetts Institute of Technology, Spring 2013, https://ocw.mit.edu/courses/sloan-school-of-management/15-053-optimization-methods-in-management-science-spring-2013/tutorials/MIT15_053S13_tut04.pdf. Accessed 20 Nov. 2020. &lt;/ref&gt; <br /> <br /> An approach to reach a solution for this particular case exists in the form of Mixed-Integer Linear Programming, where only one of the equations above is “active”.<br /> <br /> The inequality can be reformulated into the following:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X + N*Y \ge C \\<br /> -&amp;X + N*(1-Y) \ge C \\<br /> &amp;Y = 0, 1 <br /> \end{align} &lt;/math&gt;<br /> <br /> With this new set of constraints, a large constant &lt;math&gt;\textstyle N&lt;/math&gt; is introduced, along with a binary variable &lt;math&gt;\textstyle Y&lt;/math&gt;. So long as &lt;math&gt;\textstyle N&lt;/math&gt; is sufficiently larger than the upper bound of &lt;math&gt;\textstyle X + C&lt;/math&gt;, the large constant multiplied with the binary variable ensures that one of the constraints must be satisfied. For instance, if &lt;math&gt;\textstyle Y = 0&lt;/math&gt;, the new constraints will resolve to:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X \ge C \\<br /> -&amp;X + N \ge C<br /> \end{align} &lt;/math&gt;<br /> <br /> Since &lt;math&gt;\textstyle N&lt;/math&gt; is sufficiently large, the latter constraint will always be satisfied, leaving only one relation active: &lt;math&gt;\textstyle X \ge C&lt;/math&gt;. Functionally, this allows for the XOR logical operation of &lt;math&gt;\textstyle X \geq C&lt;/math&gt; and &lt;math&gt;\textstyle -X \geq C&lt;/math&gt;.<br /> <br /> === Absolute Values in Objective Functions ===<br /> In objective functions, to leverage transformations of absolute functions, all constraints must be linear.<br /> <br /> Similar to the case of absolute values in constraints, there are different approaches to the reformation of the objective function, depending on the satisfaction of sign constraints. The satisfaction of sign constraints is when the coefficient signs of the absolute terms must all be either:<br /> <br /> * Positive for a minimization problem<br /> * Negative for a maximization problem<br /> <br /> ==== Sign Constraints are Satisfied ====<br /> At a high level, the transformation works similarly to the second case of absolute value in constraints – aiming to bound the solution space for the absolute value term with a new variable, &lt;math&gt;\textstyle Z&lt;/math&gt;.<br /> <br /> If &lt;math&gt;\textstyle |X|&lt;/math&gt; is the absolute value term in our objective function, two additional constraints are added to the linear program:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X\leq Z \\<br /> -&amp;X\leq Z <br /> \end{align} &lt;/math&gt;<br /> <br /> The &lt;math&gt;\textstyle |X|&lt;/math&gt; term in the objective function is then replaced by &lt;math&gt;\textstyle Z&lt;/math&gt;, relaxing the original function into a collection of linear constraints.<br /> <br /> ==== Sign Constraints are Not Satisfied ====<br /> In order to transform problems where the coefficient signs of the absolute terms do not fulfill the conditions above, a similar conclusion is reached to that of the last case for absolute values in constraints – the use of integer variables is needed to reach an LP format. <br /> <br /> The following constraints need to be added to the problem:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X + N*Y \ge Z \\<br /> -&amp;X + N*(1-Y) \ge Z \\<br /> &amp;X \le Z \\<br /> -&amp;X \le Z \\<br /> &amp;Y = 0, 1<br /> \end{align} &lt;/math&gt;<br /> <br /> Again, &lt;math&gt;\textstyle N&lt;/math&gt; is a large constant, &lt;math&gt;\textstyle Z&lt;/math&gt; is a replacement variable for &lt;math&gt;\textstyle |X|&lt;/math&gt; in the objective function, and &lt;math&gt;\textstyle Y&lt;/math&gt; is a binary variable. The first two constraints ensure that one and only one constraint is active while the other will be automatically satisfied, following the same logic as above. The third and fourth constraints ensure that &lt;math&gt;\textstyle Z&lt;/math&gt; must be equal to &lt;math&gt;\textstyle |X|&lt;/math&gt; and has either a positive or negative value. For instance, for the case of &lt;math&gt;\textstyle Y = 0&lt;/math&gt;, the new constraints will resolve to:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X \ge Z \\<br /> -&amp;X + N \ge Z \\<br /> &amp;X \le Z \\<br /> -&amp;X \le Z<br /> \end{align} &lt;/math&gt;<br /> <br /> As &lt;math&gt;\textstyle N&lt;/math&gt; is sufficiently large (&lt;math&gt;\textstyle N&lt;/math&gt; must be at least &lt;math&gt;\textstyle 2|X|&lt;/math&gt; for this approach), the second constraint must be satisfied. Since &lt;math&gt;\textstyle Z&lt;/math&gt; is non-negative, the fourth constraint must also be satisfied. The remaining constraints, &lt;math&gt;\textstyle X \ge Z&lt;/math&gt; and &lt;math&gt;\textstyle X \le Z&lt;/math&gt; can only be satisfied when &lt;math&gt;\textstyle Z = X&lt;/math&gt; and is of non-negative signage. Together, these constraints will allow for the selection of the largest &lt;math&gt;\textstyle |X|&lt;/math&gt; for maximization problems (or smallest for minimization problems).<br /> <br /> === Absolute Values in Nonlinear Optimization Problems ===<br /> The addition of a new variable &lt;math&gt; (X_a) &lt;/math&gt; to an objective function with absolute value quantities forms a nonlinear optimization problem. The absolute value quantities would require that the objective function be reformation before proceeding. Additional constraints must be added to account for the added variable.<br /> <br /> ==Numerical Example==<br /> '''Example when All Sign Constraints are Satisfied'''<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{2|x_1| + 3|x_2| + |x_3|} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14<br /> \end{align}&lt;/math&gt;<br /> <br /> The absolute value quantities will be replaced with single variables:<br /> <br /> &lt;math&gt;|x_1| = U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt;|x_2| = U_2&lt;/math&gt;<br /> <br /> &lt;math&gt;|x_3| = U_3&lt;/math&gt;<br /> <br /> We must introduce additional constraints to ensure we do not lose any information by doing this substitution:<br /> <br /> &lt;math&gt; -U_1 \le x_1 \le U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_2 \le x_2 \le U_2 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_3 \le x_3 \le U_3 &lt;/math&gt;<br /> <br /> The problem has now been reformulated as a linear programming problem that can be solved normally:<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{ 2U_1 + 3U_2 + U_3} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14 \\<br /> -&amp;U_1 \le x_1 \le U_1 \\<br /> -&amp;U_2 \le x_2 \le U_2 \\<br /> -&amp;U_3 \le x_3 \le U_3 <br /> \end{align}&lt;/math&gt;<br /> <br /> The optimum value for the objective function is &lt;math&gt;3.5&lt;/math&gt;, which occurs when &lt;math&gt;x_1 = 0 &lt;/math&gt; and &lt;math&gt;x_2 = 0 &lt;/math&gt; and &lt;math&gt;x_3 = 3.5 &lt;/math&gt;.<br /> <br /> '''Example when Sign Constraints are not Satisfied'''<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{2|x_1| + 3|x_2| - |x_3|} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14<br /> \end{align}&lt;/math&gt;<br /> <br /> The absolute value quantities will be replaced with single variables:<br /> <br /> &lt;math&gt;|x_1| = U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt;|x_2| = U_2&lt;/math&gt;<br /> <br /> &lt;math&gt;|x_3| = U_3&lt;/math&gt;<br /> <br /> We must introduce additional constraints to ensure we do not lose any information by doing this substitution:<br /> <br /> &lt;math&gt; \begin{align}<br /> -&amp;U_1 \le x_1 \le U_1 \\<br /> -&amp;U_2 \le x_2 \le U_2 \\<br /> &amp;x_3 + M*Y \ge U_3 \\<br /> -&amp;x_3 + M*(1-Y) \ge U_3 \\<br /> &amp;x_3 \le U_3 \\<br /> -&amp;x_3 \le U_3 \\<br /> &amp;Y = 0,1<br /> \end{align}&lt;/math&gt;<br /> <br /> The problem has now been reformulated as a linear programming problem that can be solved normally:<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{ 2U_1 + 3U_2 - U_3} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14 \\<br /> -&amp;U_1 \le x_1 \le U_1 \\<br /> -&amp;U_2 \le x_2 \le U_2 \\<br /> &amp;x_3 + M*Y \ge U_3 \\<br /> -&amp;x_3 + M*(1-Y) \ge U_3 \\<br /> &amp;x_3 \le U_3 \\<br /> -&amp;x_3 \le U_3 \\<br /> &amp;Y = 0,1 <br /> \end{align}&lt;/math&gt;<br /> <br /> The optimum value for the objective function is &lt;math&gt;-3.5&lt;/math&gt;, which occur when &lt;math&gt;x_1 = 0 &lt;/math&gt; and &lt;math&gt;x_2 = 0 &lt;/math&gt; and &lt;math&gt;x_3 = 3.5 &lt;/math&gt;.<br /> <br /> == Applications ==<br /> <br /> <br /> Consider the problem &lt;math&gt;Ax=b; \quad max \quad z= x c,jx,i&lt;/math&gt;. This problem cannot, in general, be solved with the simplex method. The problem has a simplex method solution (with unrestricted basis entry) only if c, are nonpositive (non-negative for minimizing problems).<br /> <br /> The primary application of absolute-value functionals in linear programming has been for absolute-value or L(i)-metric regression analysis. Such application is always a minimization problem with all C(j) equal to 1 so that the required conditions for valid use of the simplex method are met. <br /> <br /> By reformulating the original problem into a Mixed-Integer Linear Program (MILP), we can utilize known programs to solve for the optimal solution(s). <br /> <br /> === Application in Financial: Portfolio Selection===<br /> Under this topic, the same tricks played in the Numerical Example section to perform '''Reduction to a Linear Programming Problem''' will be applied here again, to reform the problem into a MILP in order to solve the problem. An example is given as below. <br /> <br /> <br /> <br /> A portfolio is determined by what fraction of one's assets to put into each investment. It can be denoted as a collection of nonnegative numbers &lt;math&gt;\textstyle x_j&lt;/math&gt;, where &lt;math&gt; j = 1, 2,...,n &lt;/math&gt;. Because each &lt;math&gt; \textstyle x_j &lt;/math&gt;stands for a portion of the assets, it sums to one. In order to get a highest reward through finding a right mix of assets, let &lt;math&gt;\mu&lt;/math&gt;, the positive parameter, denote the importance of risk relative to the return, and &lt;math&gt;/textstyle Rj&lt;/math&gt; denote the return in the next time period on investment &lt;math&gt;j, j = 1, 2,..., n&lt;/math&gt;. The total return one would obtain from the investment is &lt;math&gt;R = \sum_{j}\!x_j\!R_j &lt;/math&gt;. The expected return is &lt;math&gt;\mathbb{E}\!R = \sum_{j}\!x_j\mathbb{E}\!R_j &lt;/math&gt;. And the Mean Absolute Deviation from the Mean (MAD) is &lt;math&gt;\mathbb{E}\left\vert \!R - \mathbb{E}\!R \right\vert = \mathbb{E}\left\vert \sum_{j}\!x_j\tilde{R}_j \right\vert &lt;/math&gt;. <br /> <br /> maximize &lt;math display=&quot;inline&quot;&gt;\mu\sum_j\!x_j\mathbb{E}\!R_j - \mathbb{E}\left\vert \sum_j \!x_j\tilde{R}_j \right\vert &lt;/math&gt;<br /> <br /> subject to &lt;math&gt;\sum_j\!x_j = 1&lt;/math&gt;<br /> <br /> &lt;math&gt;x_j \geq 0&lt;/math&gt; &lt;math&gt; j = 1,2,..n.&lt;/math&gt;<br /> <br /> where &lt;math&gt;\tilde{R}_j = \!R_j - \mathbb{E}\!R_j &lt;/math&gt;<br /> <br /> <br /> <br /> Very obviously, this problem is not a linear programming problem yet. Similar to the numerical example showed above, the right thing to do is to replace each absolute value with a new variable and impose inequality constraints to ensure that the new variable is the appropriate absolute value once an optimal value is obtained. To simplify the program, an average of the historical returns can be taken in order to get the mean expected return: &lt;math&gt;r_j = \mathbb{E}\!R_j = \left ( \frac{1}{T} \right ) \sum_{t=1}^T \!R_j(t)<br /> &lt;/math&gt;. Thus the objective function is turned into: &lt;math&gt;\mu\sum_{j}\!x_j\!r_j - \left ( \frac{1}{T} \right ) \sum_{t=1}^T\left\vert \sum_{j} \!x_j \bigl(R_j (t) - \!r_j\bigr) \right\vert <br /> &lt;/math&gt;<br /> <br /> Now, replace &lt;math&gt;\left\vert \sum_{j} \!x_j \bigl(R_j (t) - \!r_j\bigr) \right\vert <br /> &lt;/math&gt; with a new variable &lt;math&gt;y_t<br /> &lt;/math&gt;and thus the problem can be rewrote as:<br /> <br /> <br /> maximize &lt;math&gt;\mu \sum_j \!x_j\!r_j - \left ( \frac{1}{T} \right ) \sum_{t=1}^T \!y_t<br /> <br /> &lt;/math&gt;<br /> <br /> subject to &lt;math&gt;-\!y_t \leq \sum_{j} \!x_j \bigl(R_j (t) - \!r_j\bigr) \leq y_t <br /> &lt;/math&gt;. t = 1, 2,...,T<br /> <br /> where &lt;math&gt;\sum_j \!x_j = 1<br /> <br /> &lt;/math&gt;<br /> <br /> &lt;math&gt;x_j\geq 0<br /> <br /> &lt;/math&gt;. j = 1, 2,...,n<br /> <br /> &lt;math&gt;y_t \geq 0<br /> <br /> &lt;/math&gt;. t = 1, 2,...,T<br /> <br /> <br /> So finally, after some simplifications methods and some tricks applied, the original problem is converted into a linear programming which is easier to be solved further.<br /> <br /> <br /> ===Data Transfer Rate===<br /> Faster-than-nyquist, or FTNS, is a framework to transmit signals beyond the Nyquist rate. The refence to this section proposed a 24.7% faster symbol rate by utilizing Sum-of-Absolute-Values optimization.<br /> <br /> The initial model is defined as follows:<br /> &lt;math&gt;\displaystyle x_0 (t) = \sum^N_{n=1} x_{n,0} h_n (t), t \in [0,T] &lt;/math&gt;<br /> <br /> where t ∈ R denotes the continuous time index, N ∈ N is the number of transmitted symbols in each transmission period, T &gt; 0 is the interval of one period, xn,0 ∈ {+1, −1} are independent and identically distributed (i.i.d.) binary symbols [i.e., binary phase shift keying (BPSK)], and hn (t) (n = 1,...,N) are the modulation pulses.<br /> <br /> Reformulated as a convex optimization problem and repeating Newton’s method with absolute values, the solution approximates can be achieved.<br /> &lt;math&gt;\displaystyle \min_{z \in R^N} (\lambda \Vert y - Hz \Vert^2_2 + \frac{1}{2} \Vert z - 1_N \Vert_1 + \frac{1}{2} \Vert z + 1_N \Vert_1 ) &lt;/math&gt;<br /> <br /> <br /> == Conclusion ==<br /> The presence of an absolute value within the objective function prevents the use of certain optimization methods. Solving these problems requires that the function be manipulated in order to continue with linear programming techniques like the simplex method.<br /> <br /> == References ==<br /> &lt;references /&gt;<br /> <br /> <br /> <br /> <br /> # https://www.ise.ncsu.edu/fuzzy-neural/wp-content/uploads/sites/9/2019/08/LP-Abs-Value.pdf<br /> # Vanderbei R.J. (2008) Financial Applications. In: Linear Programming. International Series in Operations Research &amp; Management Science, vol 114. Springer, Boston, MA. &lt;nowiki&gt;https://doi.org/10.1007/978-0-387-74388-2_13&lt;/nowiki&gt; https://link.springer.com/chapter/10.1007/978-0-387-74388-2_13<br /> # Sasahara, Hampei &amp; Hayashi, Kazunori &amp; Nagahara, Masaaki. (2016). Symbol Detection for Faster-Than-Nyquist Signaling by Sum-of-Absolute-Values Optimization. IEEE Signal Processing Letters. PP. 1-1. 10.1109/LSP.2016.2625839. https://www.researchgate.net/publication/309745511_Symbol_Detection_for_Faster-Than-Nyquist_Signaling_by_Sum-of-Absolute-Values_Optimization</div> Bramado https://optimization.cbe.cornell.edu/index.php?title=Optimization_with_absolute_values&diff=2471 Optimization with absolute values 2020-12-13T15:13:09Z <p>Bramado: /* Introduction */</p> <hr /> <div>Authors: Matthew Chan (mdc297), Yilian Yin (yy896), Brian Amado (ba392), Peter (pmw99), Dewei Xiao (dx58) - SYSEN 5800 Fall 2020<br /> <br /> Steward: Fengqi You<br /> <br /> == Introduction ==<br /> Absolute values can make it relatively difficult to determine the optimal solution when handled without first converting to standard form. This conversion of the objective function is a good first step in solving optimization problems with absolute values. As a result, one can go on to solve the problem using linear programing techniques. With the addition of a new variable (ex: &lt;math&gt;\textstyle X^a &lt;/math&gt;) in the objective function the problem is considered nonlinear. Additional constraints must be added to find the optimal solution.<br /> <br /> == Method ==<br /> <br /> === Defining Absolute Values ===<br /> An absolute value of a real number can be described as its distance away from zero, or the non-negative magnitude of the number. &lt;ref&gt; Mendelson, Elliott, Schaum's Outline of Beginning Calculus, McGraw-Hill Professional, 2008. https://books.google.com/books?id=A8hAm38zsCMC&amp;pg=PA2#v=onepage&amp;q&amp;f=false &lt;/ref&gt; Thus,<br /> <br /> &lt;math&gt;\displaystyle |x|={\begin{cases}-x,&amp;{\text{if }}x&lt;0\\x,&amp;{\text{if }}x\geq 0\end{cases}}&lt;/math&gt;<br /> <br /> Absolute values can exist in linear optimization problems in two primary instances: in constraints and in the objective function. &lt;ref&gt; &quot;Absolute Values.&quot; ''lp_solve'', http://lpsolve.sourceforge.net/. Accessed 20 Nov. 2020. &lt;/ref&gt;<br /> <br /> === Absolute Values in Constraints ===<br /> Within constraints, absolute value relations can be transformed into one of the following forms:<br /> <br /> &lt;math&gt; \begin{align}<br /> |X| &amp;= 0 \\<br /> |X| &amp;\le C \\<br /> |X| &amp;\ge C<br /> \end{align} &lt;/math&gt;<br /> <br /> Where &lt;math&gt;\textstyle X&lt;/math&gt; is a linear combination (&lt;math&gt;\textstyle ax_1 + bx_2 + ...&lt;/math&gt; where &lt;math&gt;\textstyle a, b&lt;/math&gt; are constants) and &lt;math&gt;\textstyle C&lt;/math&gt; is a constant &lt;math&gt;\textstyle &gt; 0&lt;/math&gt;.<br /> <br /> ==== Form when &lt;math&gt;\displaystyle |X| = 0&lt;/math&gt; ====<br /> In this form, the only possible solution is if &lt;math&gt;\displaystyle X = 0&lt;/math&gt; simplifying the constraint. Note that this solution also occurs if the constraint is in the form &lt;math&gt;\displaystyle |X| \le 0&lt;/math&gt; due to the same conclusion that the only possible solution is &lt;math&gt;\textstyle X = 0&lt;/math&gt;.<br /> <br /> ==== Form when &lt;math&gt;\displaystyle |X| \le C&lt;/math&gt; ====<br /> The second form a linear constraint can exist in is &lt;math&gt;\displaystyle |X|\leq C&lt;/math&gt;. In this case, an equivalent feasible solution can be described by splitting the constraint into two:<br /> <br /> &lt;math&gt; \begin{align}<br /> X &amp;\leq C \\<br /> -X &amp;\leq C<br /> \end{align} &lt;/math&gt;<br /> <br /> The solution can be understood visually since &lt;math&gt;\textstyle X&lt;/math&gt; must lie between &lt;math&gt;\textstyle -C&lt;/math&gt; and &lt;math&gt;\textstyle C&lt;/math&gt;, as shown below:<br /> <br /> [[File:Number Line X Less Than C.png|none|thumb]]<br /> <br /> ==== Form when &lt;math&gt;\displaystyle |X| \ge C&lt;/math&gt; ====<br /> Visually, the solution space for the last form is the complement of the second solution above, resulting in the following representation:[[File:Number Line for X Greater Than C.png|none|thumb]]In expression form, the solutions can be written as:<br /> <br /> &lt;math&gt; \begin{align}<br /> X &amp;\geq C \\<br /> -X &amp;\geq C<br /> \end{align} &lt;/math&gt;<br /> <br /> As seen visually, the feasible region has a gap and thus non-convex. The expressions also make it impossible for both to simultaneously hold true. This means that it is not possible to transform constraints in this form to linear equations. &lt;ref&gt; ''Optimization Methods in Management Science / Operations Research.'' Massachusetts Institute of Technology, Spring 2013, https://ocw.mit.edu/courses/sloan-school-of-management/15-053-optimization-methods-in-management-science-spring-2013/tutorials/MIT15_053S13_tut04.pdf. Accessed 20 Nov. 2020. &lt;/ref&gt; <br /> <br /> An approach to reach a solution for this particular case exists in the form of Mixed-Integer Linear Programming, where only one of the equations above is “active”.<br /> <br /> The inequality can be reformulated into the following:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X + N*Y \ge C \\<br /> -&amp;X + N*(1-Y) \ge C \\<br /> &amp;Y = 0, 1 <br /> \end{align} &lt;/math&gt;<br /> <br /> With this new set of constraints, a large constant &lt;math&gt;\textstyle N&lt;/math&gt; is introduced, along with a binary variable &lt;math&gt;\textstyle Y&lt;/math&gt;. So long as &lt;math&gt;\textstyle N&lt;/math&gt; is sufficiently larger than the upper bound of &lt;math&gt;\textstyle X + C&lt;/math&gt;, the large constant multiplied with the binary variable ensures that one of the constraints must be satisfied. For instance, if &lt;math&gt;\textstyle Y = 0&lt;/math&gt;, the new constraints will resolve to:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X \ge C \\<br /> -&amp;X + N \ge C<br /> \end{align} &lt;/math&gt;<br /> <br /> Since &lt;math&gt;\textstyle N&lt;/math&gt; is sufficiently large, the latter constraint will always be satisfied, leaving only one relation active: &lt;math&gt;\textstyle X \ge C&lt;/math&gt;. Functionally, this allows for the XOR logical operation of &lt;math&gt;\textstyle X \geq C&lt;/math&gt; and &lt;math&gt;\textstyle -X \geq C&lt;/math&gt;.<br /> <br /> === Absolute Values in Objective Functions ===<br /> In objective functions, to leverage transformations of absolute functions, all constraints must be linear.<br /> <br /> Similar to the case of absolute values in constraints, there are different approaches to the reformation of the objective function, depending on the satisfaction of sign constraints. The satisfaction of sign constraints is when the coefficient signs of the absolute terms must all be either:<br /> <br /> * Positive for a minimization problem<br /> * Negative for a maximization problem<br /> <br /> ==== Sign Constraints are Satisfied ====<br /> At a high level, the transformation works similarly to the second case of absolute value in constraints – aiming to bound the solution space for the absolute value term with a new variable, &lt;math&gt;\textstyle Z&lt;/math&gt;.<br /> <br /> If &lt;math&gt;\textstyle |X|&lt;/math&gt; is the absolute value term in our objective function, two additional constraints are added to the linear program:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X\leq Z \\<br /> -&amp;X\leq Z <br /> \end{align} &lt;/math&gt;<br /> <br /> The &lt;math&gt;\textstyle |X|&lt;/math&gt; term in the objective function is then replaced by &lt;math&gt;\textstyle Z&lt;/math&gt;, relaxing the original function into a collection of linear constraints.<br /> <br /> ==== Sign Constraints are Not Satisfied ====<br /> In order to transform problems where the coefficient signs of the absolute terms do not fulfill the conditions above, a similar conclusion is reached to that of the last case for absolute values in constraints – the use of integer variables is needed to reach an LP format. <br /> <br /> The following constraints need to be added to the problem:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X + N*Y \ge Z \\<br /> -&amp;X + N*(1-Y) \ge Z \\<br /> &amp;X \le Z \\<br /> -&amp;X \le Z \\<br /> &amp;Y = 0, 1<br /> \end{align} &lt;/math&gt;<br /> <br /> Again, &lt;math&gt;\textstyle N&lt;/math&gt; is a large constant, &lt;math&gt;\textstyle Z&lt;/math&gt; is a replacement variable for &lt;math&gt;\textstyle |X|&lt;/math&gt; in the objective function, and &lt;math&gt;\textstyle Y&lt;/math&gt; is a binary variable. The first two constraints ensure that one and only one constraint is active while the other will be automatically satisfied, following the same logic as above. The third and fourth constraints ensure that &lt;math&gt;\textstyle Z&lt;/math&gt; must be equal to &lt;math&gt;\textstyle |X|&lt;/math&gt; and has either a positive or negative value. For instance, for the case of &lt;math&gt;\textstyle Y = 0&lt;/math&gt;, the new constraints will resolve to:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X \ge Z \\<br /> -&amp;X + N \ge Z \\<br /> &amp;X \le Z \\<br /> -&amp;X \le Z<br /> \end{align} &lt;/math&gt;<br /> <br /> As &lt;math&gt;\textstyle N&lt;/math&gt; is sufficiently large (&lt;math&gt;\textstyle N&lt;/math&gt; must be at least &lt;math&gt;\textstyle 2|X|&lt;/math&gt; for this approach), the second constraint must be satisfied. Since &lt;math&gt;\textstyle Z&lt;/math&gt; is non-negative, the fourth constraint must also be satisfied. The remaining constraints, &lt;math&gt;\textstyle X \ge Z&lt;/math&gt; and &lt;math&gt;\textstyle X \le Z&lt;/math&gt; can only be satisfied when &lt;math&gt;\textstyle Z = X&lt;/math&gt; and is of non-negative signage. Together, these constraints will allow for the selection of the largest &lt;math&gt;\textstyle |X|&lt;/math&gt; for maximization problems (or smallest for minimization problems).<br /> <br /> ==Numerical Example==<br /> '''Example when All Sign Constraints are Satisfied'''<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{2|x_1| + 3|x_2| + |x_3|} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14<br /> \end{align}&lt;/math&gt;<br /> <br /> The absolute value quantities will be replaced with single variables:<br /> <br /> &lt;math&gt;|x_1| = U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt;|x_2| = U_2&lt;/math&gt;<br /> <br /> &lt;math&gt;|x_3| = U_3&lt;/math&gt;<br /> <br /> We must introduce additional constraints to ensure we do not lose any information by doing this substitution:<br /> <br /> &lt;math&gt; -U_1 \le x_1 \le U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_2 \le x_2 \le U_2 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_3 \le x_3 \le U_3 &lt;/math&gt;<br /> <br /> The problem has now been reformulated as a linear programming problem that can be solved normally:<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{ 2U_1 + 3U_2 + U_3} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14 \\<br /> -&amp;U_1 \le x_1 \le U_1 \\<br /> -&amp;U_2 \le x_2 \le U_2 \\<br /> -&amp;U_3 \le x_3 \le U_3 <br /> \end{align}&lt;/math&gt;<br /> <br /> The optimum value for the objective function is &lt;math&gt;3.5&lt;/math&gt;, which occurs when &lt;math&gt;x_1 = 0 &lt;/math&gt; and &lt;math&gt;x_2 = 0 &lt;/math&gt; and &lt;math&gt;x_3 = 3.5 &lt;/math&gt;.<br /> <br /> '''Example when Sign Constraints are not Satisfied'''<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{2|x_1| + 3|x_2| - |x_3|} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14<br /> \end{align}&lt;/math&gt;<br /> <br /> The absolute value quantities will be replaced with single variables:<br /> <br /> &lt;math&gt;|x_1| = U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt;|x_2| = U_2&lt;/math&gt;<br /> <br /> &lt;math&gt;|x_3| = U_3&lt;/math&gt;<br /> <br /> We must introduce additional constraints to ensure we do not lose any information by doing this substitution:<br /> <br /> &lt;math&gt; \begin{align}<br /> -&amp;U_1 \le x_1 \le U_1 \\<br /> -&amp;U_2 \le x_2 \le U_2 \\<br /> &amp;x_3 + M*Y \ge U_3 \\<br /> -&amp;x_3 + M*(1-Y) \ge U_3 \\<br /> &amp;x_3 \le U_3 \\<br /> -&amp;x_3 \le U_3 \\<br /> &amp;Y = 0,1<br /> \end{align}&lt;/math&gt;<br /> <br /> The problem has now been reformulated as a linear programming problem that can be solved normally:<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{ 2U_1 + 3U_2 - U_3} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14 \\<br /> -&amp;U_1 \le x_1 \le U_1 \\<br /> -&amp;U_2 \le x_2 \le U_2 \\<br /> &amp;x_3 + M*Y \ge U_3 \\<br /> -&amp;x_3 + M*(1-Y) \ge U_3 \\<br /> &amp;x_3 \le U_3 \\<br /> -&amp;x_3 \le U_3 \\<br /> &amp;Y = 0,1 <br /> \end{align}&lt;/math&gt;<br /> <br /> The optimum value for the objective function is &lt;math&gt;-3.5&lt;/math&gt;, which occur when &lt;math&gt;x_1 = 0 &lt;/math&gt; and &lt;math&gt;x_2 = 0 &lt;/math&gt; and &lt;math&gt;x_3 = 3.5 &lt;/math&gt;.<br /> <br /> == Applications ==<br /> <br /> <br /> Consider the problem &lt;math&gt;Ax=b; \quad max \quad z= x c,jx,i&lt;/math&gt;. This problem cannot, in general, be solved with the simplex method. The problem has a simplex method solution (with unrestricted basis entry) only if c, are nonpositive (non-negative for minimizing problems).<br /> <br /> The primary application of absolute-value functionals in linear programming has been for absolute-value or L(i)-metric regression analysis. Such application is always a minimization problem with all C(j) equal to 1 so that the required conditions for valid use of the simplex method are met. <br /> <br /> By reformulating the original problem into a Mixed-Integer Linear Program (MILP), we can utilize known programs to solve for the optimal solution(s). <br /> <br /> === Application in Financial: Portfolio Selection===<br /> Under this topic, the same tricks played in the Numerical Example section to perform '''Reduction to a Linear Programming Problem''' will be applied here again, to reform the problem into a MILP in order to solve the problem. An example is given as below. <br /> <br /> <br /> <br /> A portfolio is determined by what fraction of one's assets to put into each investment. It can be denoted as a collection of nonnegative numbers &lt;math&gt;\textstyle x_j&lt;/math&gt;, where &lt;math&gt; j = 1, 2,...,n &lt;/math&gt;. Because each &lt;math&gt; \textstyle x_j &lt;/math&gt;stands for a portion of the assets, it sums to one. In order to get a highest reward through finding a right mix of assets, let &lt;math&gt;\mu&lt;/math&gt;, the positive parameter, denote the importance of risk relative to the return, and &lt;math&gt;/textstyle Rj&lt;/math&gt; denote the return in the next time period on investment &lt;math&gt;j, j = 1, 2,..., n&lt;/math&gt;. The total return one would obtain from the investment is &lt;math&gt;R = \sum_{j}\!x_j\!R_j &lt;/math&gt;. The expected return is &lt;math&gt;\mathbb{E}\!R = \sum_{j}\!x_j\mathbb{E}\!R_j &lt;/math&gt;. And the Mean Absolute Deviation from the Mean (MAD) is &lt;math&gt;\mathbb{E}\left\vert \!R - \mathbb{E}\!R \right\vert = \mathbb{E}\left\vert \sum_{j}\!x_j\tilde{R}_j \right\vert &lt;/math&gt;. <br /> <br /> maximize &lt;math display=&quot;inline&quot;&gt;\mu\sum_j\!x_j\mathbb{E}\!R_j - \mathbb{E}\left\vert \sum_j \!x_j\tilde{R}_j \right\vert &lt;/math&gt;<br /> <br /> subject to &lt;math&gt;\sum_j\!x_j = 1&lt;/math&gt;<br /> <br /> &lt;math&gt;x_j \geq 0&lt;/math&gt; &lt;math&gt; j = 1,2,..n.&lt;/math&gt;<br /> <br /> where &lt;math&gt;\tilde{R}_j = \!R_j - \mathbb{E}\!R_j &lt;/math&gt;<br /> <br /> <br /> <br /> Very obviously, this problem is not a linear programming problem yet. Similar to the numerical example showed above, the right thing to do is to replace each absolute value with a new variable and impose inequality constraints to ensure that the new variable is the appropriate absolute value once an optimal value is obtained. To simplify the program, an average of the historical returns can be taken in order to get the mean expected return: &lt;math&gt;r_j = \mathbb{E}\!R_j = \left ( \frac{1}{T} \right ) \sum_{t=1}^T \!R_j(t)<br /> &lt;/math&gt;. Thus the objective function is turned into: &lt;math&gt;\mu\sum_{j}\!x_j\!r_j - \left ( \frac{1}{T} \right ) \sum_{t=1}^T\left\vert \sum_{j} \!x_j \bigl(R_j (t) - \!r_j\bigr) \right\vert <br /> &lt;/math&gt;<br /> <br /> Now, replace &lt;math&gt;\left\vert \sum_{j} \!x_j \bigl(R_j (t) - \!r_j\bigr) \right\vert <br /> &lt;/math&gt; with a new variable &lt;math&gt;y_t<br /> &lt;/math&gt;and thus the problem can be rewrote as:<br /> <br /> <br /> maximize &lt;math&gt;\mu \sum_j \!x_j\!r_j - \left ( \frac{1}{T} \right ) \sum_{t=1}^T \!y_t<br /> <br /> &lt;/math&gt;<br /> <br /> subject to &lt;math&gt;-\!y_t \leq \sum_{j} \!x_j \bigl(R_j (t) - \!r_j\bigr) \leq y_t <br /> &lt;/math&gt;. t = 1, 2,...,T<br /> <br /> where &lt;math&gt;\sum_j \!x_j = 1<br /> <br /> &lt;/math&gt;<br /> <br /> &lt;math&gt;x_j\geq 0<br /> <br /> &lt;/math&gt;. j = 1, 2,...,n<br /> <br /> &lt;math&gt;y_t \geq 0<br /> <br /> &lt;/math&gt;. t = 1, 2,...,T<br /> <br /> <br /> So finally, after some simplifications methods and some tricks applied, the original problem is converted into a linear programming which is easier to be solved further.<br /> <br /> <br /> ===Data Transfer Rate===<br /> Faster-than-nyquist, or FTNS, is a framework to transmit signals beyond the Nyquist rate. The refence to this section proposed a 24.7% faster symbol rate by utilizing Sum-of-Absolute-Values optimization.<br /> <br /> The initial model is defined as follows:<br /> &lt;math&gt;\displaystyle x_0 (t) = \sum^N_{n=1} x_{n,0} h_n (t), t \in [0,T] &lt;/math&gt;<br /> <br /> where t ∈ R denotes the continuous time index, N ∈ N is the number of transmitted symbols in each transmission period, T &gt; 0 is the interval of one period, xn,0 ∈ {+1, −1} are independent and identically distributed (i.i.d.) binary symbols [i.e., binary phase shift keying (BPSK)], and hn (t) (n = 1,...,N) are the modulation pulses.<br /> <br /> Reformulated as a convex optimization problem and repeating Newton’s method with absolute values, the solution approximates can be achieved.<br /> &lt;math&gt;\displaystyle \min_{z \in R^N} (\lambda \Vert y - Hz \Vert^2_2 + \frac{1}{2} \Vert z - 1_N \Vert_1 + \frac{1}{2} \Vert z + 1_N \Vert_1 ) &lt;/math&gt;<br /> <br /> <br /> == Conclusion ==<br /> The presence of an absolute value within the objective function prevents the use of certain optimization methods. Solving these problems requires that the function be manipulated in order to continue with linear programming techniques like the simplex method.<br /> <br /> == References ==<br /> &lt;references /&gt;<br /> <br /> <br /> <br /> <br /> # https://www.ise.ncsu.edu/fuzzy-neural/wp-content/uploads/sites/9/2019/08/LP-Abs-Value.pdf<br /> # Vanderbei R.J. (2008) Financial Applications. In: Linear Programming. International Series in Operations Research &amp; Management Science, vol 114. Springer, Boston, MA. &lt;nowiki&gt;https://doi.org/10.1007/978-0-387-74388-2_13&lt;/nowiki&gt; https://link.springer.com/chapter/10.1007/978-0-387-74388-2_13<br /> # Sasahara, Hampei &amp; Hayashi, Kazunori &amp; Nagahara, Masaaki. (2016). Symbol Detection for Faster-Than-Nyquist Signaling by Sum-of-Absolute-Values Optimization. IEEE Signal Processing Letters. PP. 1-1. 10.1109/LSP.2016.2625839. https://www.researchgate.net/publication/309745511_Symbol_Detection_for_Faster-Than-Nyquist_Signaling_by_Sum-of-Absolute-Values_Optimization</div> Bramado https://optimization.cbe.cornell.edu/index.php?title=Optimization_with_absolute_values&diff=2470 Optimization with absolute values 2020-12-13T15:09:48Z <p>Bramado: </p> <hr /> <div>Authors: Matthew Chan (mdc297), Yilian Yin (yy896), Brian Amado (ba392), Peter (pmw99), Dewei Xiao (dx58) - SYSEN 5800 Fall 2020<br /> <br /> Steward: Fengqi You<br /> <br /> == Introduction ==<br /> Absolute values can make it relatively difficult to determine the optimal solution when handled without first converting to standard form. This conversion of the objective function is a good first step in solving optimization problems with absolute values. As a result, one can go on to solve the problem using linear programing techniques. Within the objective function, if anything is nonlinear (ex: &lt;math&gt;\textstyle |X^2| &lt;/math&gt;) the problem is considered nonlinear. Additional constraints must be added.<br /> <br /> == Method ==<br /> <br /> === Defining Absolute Values ===<br /> An absolute value of a real number can be described as its distance away from zero, or the non-negative magnitude of the number. &lt;ref&gt; Mendelson, Elliott, Schaum's Outline of Beginning Calculus, McGraw-Hill Professional, 2008. https://books.google.com/books?id=A8hAm38zsCMC&amp;pg=PA2#v=onepage&amp;q&amp;f=false &lt;/ref&gt; Thus,<br /> <br /> &lt;math&gt;\displaystyle |x|={\begin{cases}-x,&amp;{\text{if }}x&lt;0\\x,&amp;{\text{if }}x\geq 0\end{cases}}&lt;/math&gt;<br /> <br /> Absolute values can exist in linear optimization problems in two primary instances: in constraints and in the objective function. &lt;ref&gt; &quot;Absolute Values.&quot; ''lp_solve'', http://lpsolve.sourceforge.net/. Accessed 20 Nov. 2020. &lt;/ref&gt;<br /> <br /> === Absolute Values in Constraints ===<br /> Within constraints, absolute value relations can be transformed into one of the following forms:<br /> <br /> &lt;math&gt; \begin{align}<br /> |X| &amp;= 0 \\<br /> |X| &amp;\le C \\<br /> |X| &amp;\ge C<br /> \end{align} &lt;/math&gt;<br /> <br /> Where &lt;math&gt;\textstyle X&lt;/math&gt; is a linear combination (&lt;math&gt;\textstyle ax_1 + bx_2 + ...&lt;/math&gt; where &lt;math&gt;\textstyle a, b&lt;/math&gt; are constants) and &lt;math&gt;\textstyle C&lt;/math&gt; is a constant &lt;math&gt;\textstyle &gt; 0&lt;/math&gt;.<br /> <br /> ==== Form when &lt;math&gt;\displaystyle |X| = 0&lt;/math&gt; ====<br /> In this form, the only possible solution is if &lt;math&gt;\displaystyle X = 0&lt;/math&gt; simplifying the constraint. Note that this solution also occurs if the constraint is in the form &lt;math&gt;\displaystyle |X| \le 0&lt;/math&gt; due to the same conclusion that the only possible solution is &lt;math&gt;\textstyle X = 0&lt;/math&gt;.<br /> <br /> ==== Form when &lt;math&gt;\displaystyle |X| \le C&lt;/math&gt; ====<br /> The second form a linear constraint can exist in is &lt;math&gt;\displaystyle |X|\leq C&lt;/math&gt;. In this case, an equivalent feasible solution can be described by splitting the constraint into two:<br /> <br /> &lt;math&gt; \begin{align}<br /> X &amp;\leq C \\<br /> -X &amp;\leq C<br /> \end{align} &lt;/math&gt;<br /> <br /> The solution can be understood visually since &lt;math&gt;\textstyle X&lt;/math&gt; must lie between &lt;math&gt;\textstyle -C&lt;/math&gt; and &lt;math&gt;\textstyle C&lt;/math&gt;, as shown below:<br /> <br /> [[File:Number Line X Less Than C.png|none|thumb]]<br /> <br /> ==== Form when &lt;math&gt;\displaystyle |X| \ge C&lt;/math&gt; ====<br /> Visually, the solution space for the last form is the complement of the second solution above, resulting in the following representation:[[File:Number Line for X Greater Than C.png|none|thumb]]In expression form, the solutions can be written as:<br /> <br /> &lt;math&gt; \begin{align}<br /> X &amp;\geq C \\<br /> -X &amp;\geq C<br /> \end{align} &lt;/math&gt;<br /> <br /> As seen visually, the feasible region has a gap and thus non-convex. The expressions also make it impossible for both to simultaneously hold true. This means that it is not possible to transform constraints in this form to linear equations. &lt;ref&gt; ''Optimization Methods in Management Science / Operations Research.'' Massachusetts Institute of Technology, Spring 2013, https://ocw.mit.edu/courses/sloan-school-of-management/15-053-optimization-methods-in-management-science-spring-2013/tutorials/MIT15_053S13_tut04.pdf. Accessed 20 Nov. 2020. &lt;/ref&gt; <br /> <br /> An approach to reach a solution for this particular case exists in the form of Mixed-Integer Linear Programming, where only one of the equations above is “active”.<br /> <br /> The inequality can be reformulated into the following:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X + N*Y \ge C \\<br /> -&amp;X + N*(1-Y) \ge C \\<br /> &amp;Y = 0, 1 <br /> \end{align} &lt;/math&gt;<br /> <br /> With this new set of constraints, a large constant &lt;math&gt;\textstyle N&lt;/math&gt; is introduced, along with a binary variable &lt;math&gt;\textstyle Y&lt;/math&gt;. So long as &lt;math&gt;\textstyle N&lt;/math&gt; is sufficiently larger than the upper bound of &lt;math&gt;\textstyle X + C&lt;/math&gt;, the large constant multiplied with the binary variable ensures that one of the constraints must be satisfied. For instance, if &lt;math&gt;\textstyle Y = 0&lt;/math&gt;, the new constraints will resolve to:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X \ge C \\<br /> -&amp;X + N \ge C<br /> \end{align} &lt;/math&gt;<br /> <br /> Since &lt;math&gt;\textstyle N&lt;/math&gt; is sufficiently large, the latter constraint will always be satisfied, leaving only one relation active: &lt;math&gt;\textstyle X \ge C&lt;/math&gt;. Functionally, this allows for the XOR logical operation of &lt;math&gt;\textstyle X \geq C&lt;/math&gt; and &lt;math&gt;\textstyle -X \geq C&lt;/math&gt;.<br /> <br /> === Absolute Values in Objective Functions ===<br /> In objective functions, to leverage transformations of absolute functions, all constraints must be linear.<br /> <br /> Similar to the case of absolute values in constraints, there are different approaches to the reformation of the objective function, depending on the satisfaction of sign constraints. The satisfaction of sign constraints is when the coefficient signs of the absolute terms must all be either:<br /> <br /> * Positive for a minimization problem<br /> * Negative for a maximization problem<br /> <br /> ==== Sign Constraints are Satisfied ====<br /> At a high level, the transformation works similarly to the second case of absolute value in constraints – aiming to bound the solution space for the absolute value term with a new variable, &lt;math&gt;\textstyle Z&lt;/math&gt;.<br /> <br /> If &lt;math&gt;\textstyle |X|&lt;/math&gt; is the absolute value term in our objective function, two additional constraints are added to the linear program:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X\leq Z \\<br /> -&amp;X\leq Z <br /> \end{align} &lt;/math&gt;<br /> <br /> The &lt;math&gt;\textstyle |X|&lt;/math&gt; term in the objective function is then replaced by &lt;math&gt;\textstyle Z&lt;/math&gt;, relaxing the original function into a collection of linear constraints.<br /> <br /> ==== Sign Constraints are Not Satisfied ====<br /> In order to transform problems where the coefficient signs of the absolute terms do not fulfill the conditions above, a similar conclusion is reached to that of the last case for absolute values in constraints – the use of integer variables is needed to reach an LP format. <br /> <br /> The following constraints need to be added to the problem:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X + N*Y \ge Z \\<br /> -&amp;X + N*(1-Y) \ge Z \\<br /> &amp;X \le Z \\<br /> -&amp;X \le Z \\<br /> &amp;Y = 0, 1<br /> \end{align} &lt;/math&gt;<br /> <br /> Again, &lt;math&gt;\textstyle N&lt;/math&gt; is a large constant, &lt;math&gt;\textstyle Z&lt;/math&gt; is a replacement variable for &lt;math&gt;\textstyle |X|&lt;/math&gt; in the objective function, and &lt;math&gt;\textstyle Y&lt;/math&gt; is a binary variable. The first two constraints ensure that one and only one constraint is active while the other will be automatically satisfied, following the same logic as above. The third and fourth constraints ensure that &lt;math&gt;\textstyle Z&lt;/math&gt; must be equal to &lt;math&gt;\textstyle |X|&lt;/math&gt; and has either a positive or negative value. For instance, for the case of &lt;math&gt;\textstyle Y = 0&lt;/math&gt;, the new constraints will resolve to:<br /> <br /> &lt;math&gt; \begin{align}<br /> &amp;X \ge Z \\<br /> -&amp;X + N \ge Z \\<br /> &amp;X \le Z \\<br /> -&amp;X \le Z<br /> \end{align} &lt;/math&gt;<br /> <br /> As &lt;math&gt;\textstyle N&lt;/math&gt; is sufficiently large (&lt;math&gt;\textstyle N&lt;/math&gt; must be at least &lt;math&gt;\textstyle 2|X|&lt;/math&gt; for this approach), the second constraint must be satisfied. Since &lt;math&gt;\textstyle Z&lt;/math&gt; is non-negative, the fourth constraint must also be satisfied. The remaining constraints, &lt;math&gt;\textstyle X \ge Z&lt;/math&gt; and &lt;math&gt;\textstyle X \le Z&lt;/math&gt; can only be satisfied when &lt;math&gt;\textstyle Z = X&lt;/math&gt; and is of non-negative signage. Together, these constraints will allow for the selection of the largest &lt;math&gt;\textstyle |X|&lt;/math&gt; for maximization problems (or smallest for minimization problems).<br /> <br /> ==Numerical Example==<br /> '''Example when All Sign Constraints are Satisfied'''<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{2|x_1| + 3|x_2| + |x_3|} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14<br /> \end{align}&lt;/math&gt;<br /> <br /> The absolute value quantities will be replaced with single variables:<br /> <br /> &lt;math&gt;|x_1| = U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt;|x_2| = U_2&lt;/math&gt;<br /> <br /> &lt;math&gt;|x_3| = U_3&lt;/math&gt;<br /> <br /> We must introduce additional constraints to ensure we do not lose any information by doing this substitution:<br /> <br /> &lt;math&gt; -U_1 \le x_1 \le U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_2 \le x_2 \le U_2 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_3 \le x_3 \le U_3 &lt;/math&gt;<br /> <br /> The problem has now been reformulated as a linear programming problem that can be solved normally:<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{ 2U_1 + 3U_2 + U_3} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14 \\<br /> -&amp;U_1 \le x_1 \le U_1 \\<br /> -&amp;U_2 \le x_2 \le U_2 \\<br /> -&amp;U_3 \le x_3 \le U_3 <br /> \end{align}&lt;/math&gt;<br /> <br /> The optimum value for the objective function is &lt;math&gt;3.5&lt;/math&gt;, which occurs when &lt;math&gt;x_1 = 0 &lt;/math&gt; and &lt;math&gt;x_2 = 0 &lt;/math&gt; and &lt;math&gt;x_3 = 3.5 &lt;/math&gt;.<br /> <br /> '''Example when Sign Constraints are not Satisfied'''<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{2|x_1| + 3|x_2| - |x_3|} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14<br /> \end{align}&lt;/math&gt;<br /> <br /> The absolute value quantities will be replaced with single variables:<br /> <br /> &lt;math&gt;|x_1| = U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt;|x_2| = U_2&lt;/math&gt;<br /> <br /> &lt;math&gt;|x_3| = U_3&lt;/math&gt;<br /> <br /> We must introduce additional constraints to ensure we do not lose any information by doing this substitution:<br /> <br /> &lt;math&gt; \begin{align}<br /> -&amp;U_1 \le x_1 \le U_1 \\<br /> -&amp;U_2 \le x_2 \le U_2 \\<br /> &amp;x_3 + M*Y \ge U_3 \\<br /> -&amp;x_3 + M*(1-Y) \ge U_3 \\<br /> &amp;x_3 \le U_3 \\<br /> -&amp;x_3 \le U_3 \\<br /> &amp;Y = 0,1<br /> \end{align}&lt;/math&gt;<br /> <br /> The problem has now been reformulated as a linear programming problem that can be solved normally:<br /> <br /> &lt;math&gt; \begin{align}<br /> \min \quad &amp;{ 2U_1 + 3U_2 - U_3} \\<br /> s.t. \quad &amp;x_1 + 2x_2 - 3x_3 \le 8 \\<br /> &amp;2x_1 - x_2 + 4x_3= 14 \\<br /> -&amp;U_1 \le x_1 \le U_1 \\<br /> -&amp;U_2 \le x_2 \le U_2 \\<br /> &amp;x_3 + M*Y \ge U_3 \\<br /> -&amp;x_3 + M*(1-Y) \ge U_3 \\<br /> &amp;x_3 \le U_3 \\<br /> -&amp;x_3 \le U_3 \\<br /> &amp;Y = 0,1 <br /> \end{align}&lt;/math&gt;<br /> <br /> The optimum value for the objective function is &lt;math&gt;-3.5&lt;/math&gt;, which occur when &lt;math&gt;x_1 = 0 &lt;/math&gt; and &lt;math&gt;x_2 = 0 &lt;/math&gt; and &lt;math&gt;x_3 = 3.5 &lt;/math&gt;.<br /> <br /> == Applications ==<br /> <br /> <br /> Consider the problem &lt;math&gt;Ax=b; \quad max \quad z= x c,jx,i&lt;/math&gt;. This problem cannot, in general, be solved with the simplex method. The problem has a simplex method solution (with unrestricted basis entry) only if c, are nonpositive (non-negative for minimizing problems).<br /> <br /> The primary application of absolute-value functionals in linear programming has been for absolute-value or L(i)-metric regression analysis. Such application is always a minimization problem with all C(j) equal to 1 so that the required conditions for valid use of the simplex method are met. <br /> <br /> By reformulating the original problem into a Mixed-Integer Linear Program (MILP), we can utilize known programs to solve for the optimal solution(s). <br /> <br /> === Application in Financial: Portfolio Selection===<br /> Under this topic, the same tricks played in the Numerical Example section to perform '''Reduction to a Linear Programming Problem''' will be applied here again, to reform the problem into a MILP in order to solve the problem. An example is given as below. <br /> <br /> <br /> <br /> A portfolio is determined by what fraction of one's assets to put into each investment. It can be denoted as a collection of nonnegative numbers &lt;math&gt;\textstyle x_j&lt;/math&gt;, where &lt;math&gt; j = 1, 2,...,n &lt;/math&gt;. Because each &lt;math&gt; \textstyle x_j &lt;/math&gt;stands for a portion of the assets, it sums to one. In order to get a highest reward through finding a right mix of assets, let &lt;math&gt;\mu&lt;/math&gt;, the positive parameter, denote the importance of risk relative to the return, and &lt;math&gt;/textstyle Rj&lt;/math&gt; denote the return in the next time period on investment &lt;math&gt;j, j = 1, 2,..., n&lt;/math&gt;. The total return one would obtain from the investment is &lt;math&gt;R = \sum_{j}\!x_j\!R_j &lt;/math&gt;. The expected return is &lt;math&gt;\mathbb{E}\!R = \sum_{j}\!x_j\mathbb{E}\!R_j &lt;/math&gt;. And the Mean Absolute Deviation from the Mean (MAD) is &lt;math&gt;\mathbb{E}\left\vert \!R - \mathbb{E}\!R \right\vert = \mathbb{E}\left\vert \sum_{j}\!x_j\tilde{R}_j \right\vert &lt;/math&gt;. <br /> <br /> maximize &lt;math display=&quot;inline&quot;&gt;\mu\sum_j\!x_j\mathbb{E}\!R_j - \mathbb{E}\left\vert \sum_j \!x_j\tilde{R}_j \right\vert &lt;/math&gt;<br /> <br /> subject to &lt;math&gt;\sum_j\!x_j = 1&lt;/math&gt;<br /> <br /> &lt;math&gt;x_j \geq 0&lt;/math&gt; &lt;math&gt; j = 1,2,..n.&lt;/math&gt;<br /> <br /> where &lt;math&gt;\tilde{R}_j = \!R_j - \mathbb{E}\!R_j &lt;/math&gt;<br /> <br /> <br /> <br /> Very obviously, this problem is not a linear programming problem yet. Similar to the numerical example showed above, the right thing to do is to replace each absolute value with a new variable and impose inequality constraints to ensure that the new variable is the appropriate absolute value once an optimal value is obtained. To simplify the program, an average of the historical returns can be taken in order to get the mean expected return: &lt;math&gt;r_j = \mathbb{E}\!R_j = \left ( \frac{1}{T} \right ) \sum_{t=1}^T \!R_j(t)<br /> &lt;/math&gt;. Thus the objective function is turned into: &lt;math&gt;\mu\sum_{j}\!x_j\!r_j - \left ( \frac{1}{T} \right ) \sum_{t=1}^T\left\vert \sum_{j} \!x_j \bigl(R_j (t) - \!r_j\bigr) \right\vert <br /> &lt;/math&gt;<br /> <br /> Now, replace &lt;math&gt;\left\vert \sum_{j} \!x_j \bigl(R_j (t) - \!r_j\bigr) \right\vert <br /> &lt;/math&gt; with a new variable &lt;math&gt;y_t<br /> &lt;/math&gt;and thus the problem can be rewrote as:<br /> <br /> <br /> maximize &lt;math&gt;\mu \sum_j \!x_j\!r_j - \left ( \frac{1}{T} \right ) \sum_{t=1}^T \!y_t<br /> <br /> &lt;/math&gt;<br /> <br /> subject to &lt;math&gt;-\!y_t \leq \sum_{j} \!x_j \bigl(R_j (t) - \!r_j\bigr) \leq y_t <br /> &lt;/math&gt;. t = 1, 2,...,T<br /> <br /> where &lt;math&gt;\sum_j \!x_j = 1<br /> <br /> &lt;/math&gt;<br /> <br /> &lt;math&gt;x_j\geq 0<br /> <br /> &lt;/math&gt;. j = 1, 2,...,n<br /> <br /> &lt;math&gt;y_t \geq 0<br /> <br /> &lt;/math&gt;. t = 1, 2,...,T<br /> <br /> <br /> So finally, after some simplifications methods and some tricks applied, the original problem is converted into a linear programming which is easier to be solved further.<br /> <br /> <br /> ===Data Transfer Rate===<br /> Faster-than-nyquist, or FTNS, is a framework to transmit signals beyond the Nyquist rate. The refence to this section proposed a 24.7% faster symbol rate by utilizing Sum-of-Absolute-Values optimization.<br /> <br /> The initial model is defined as follows:<br /> &lt;math&gt;\displaystyle x_0 (t) = \sum^N_{n=1} x_{n,0} h_n (t), t \in [0,T] &lt;/math&gt;<br /> <br /> where t ∈ R denotes the continuous time index, N ∈ N is the number of transmitted symbols in each transmission period, T &gt; 0 is the interval of one period, xn,0 ∈ {+1, −1} are independent and identically distributed (i.i.d.) binary symbols [i.e., binary phase shift keying (BPSK)], and hn (t) (n = 1,...,N) are the modulation pulses.<br /> <br /> Reformulated as a convex optimization problem and repeating Newton’s method with absolute values, the solution approximates can be achieved.<br /> &lt;math&gt;\displaystyle \min_{z \in R^N} (\lambda \Vert y - Hz \Vert^2_2 + \frac{1}{2} \Vert z - 1_N \Vert_1 + \frac{1}{2} \Vert z + 1_N \Vert_1 ) &lt;/math&gt;<br /> <br /> <br /> == Conclusion ==<br /> The presence of an absolute value within the objective function prevents the use of certain optimization methods. Solving these problems requires that the function be manipulated in order to continue with linear programming techniques like the simplex method.<br /> <br /> == References ==<br /> &lt;references /&gt;<br /> <br /> <br /> <br /> <br /> # https://www.ise.ncsu.edu/fuzzy-neural/wp-content/uploads/sites/9/2019/08/LP-Abs-Value.pdf<br /> # Vanderbei R.J. (2008) Financial Applications. In: Linear Programming. International Series in Operations Research &amp; Management Science, vol 114. Springer, Boston, MA. &lt;nowiki&gt;https://doi.org/10.1007/978-0-387-74388-2_13&lt;/nowiki&gt; https://link.springer.com/chapter/10.1007/978-0-387-74388-2_13<br /> # Sasahara, Hampei &amp; Hayashi, Kazunori &amp; Nagahara, Masaaki. (2016). Symbol Detection for Faster-Than-Nyquist Signaling by Sum-of-Absolute-Values Optimization. IEEE Signal Processing Letters. PP. 1-1. 10.1109/LSP.2016.2625839. https://www.researchgate.net/publication/309745511_Symbol_Detection_for_Faster-Than-Nyquist_Signaling_by_Sum-of-Absolute-Values_Optimization</div> Bramado https://optimization.cbe.cornell.edu/index.php?title=Optimization_with_absolute_values&diff=1988 Optimization with absolute values 2020-12-03T01:42:45Z <p>Bramado: </p> <hr /> <div>Authors: Matthew Chan (mdc297), Yilian Yin (), Brian Amado (ba392), Peter (pmw99), Dewei Xiao (dx58) - SYSEN 5800 Fall 2020<br /> <br /> Steward: Fengqi You<br /> <br /> == Introduction ==<br /> Absolute values can make it relatively difficult to determine the optimal solution when handled without first converting to standard form. This conversion of the objective function is a good first step in solving optimization problems with absolute values. As a result, one can go on to solve the problem using linear programing techniques.<br /> <br /> == Method ==<br /> <br /> === Defining Absolute Values ===<br /> An absolute value of a real number can be described as its distance away from zero, or the non-negative magnitude of the number. Thus,<br /> <br /> &lt;math&gt;|x| = \begin{cases} -x, &amp; \text{if }x &lt; 0 \\ x, &amp; \text{if }x \ge 0 \end{cases} &lt;/math&gt;<br /> <br /> Absolute values can exist in optimization problems in two primary instances: in constraints and in the objective function.<br /> <br /> === Absolute Values in Constraints ===<br /> Within linear equations, linear constraints can exist in several forms.<br /> <br /> The first form exists as &lt;math&gt;|X| = 0 &lt;/math&gt;, where &lt;math display=&quot;inline&quot;&gt;X &lt;/math&gt; is a linear combination of variables.<br /> <br /> In this case, the only solution is if &lt;math&gt;|X| = 0 &lt;/math&gt;, simplifying the constraint to &lt;math&gt;X = 0 &lt;/math&gt;. Note that this solution also occurs if the constraint is in the form &lt;math&gt;|X| &lt;= 0 &lt;/math&gt; due to the same conclusion (only solution &lt;math&gt;X = 0 &lt;/math&gt;).<br /> <br /> The second form a linear constraint can exist in is &lt;math&gt;|X| \le C &lt;/math&gt; where &lt;math display=&quot;inline&quot;&gt;X &lt;/math&gt; remains a linear combination of variables and constant &lt;math display=&quot;inline&quot;&gt;C &gt; 0 &lt;/math&gt;.<br /> <br /> In this case, we can describe an equivalent feasible solution by splitting the inequality into<br /> <br /> &lt;math&gt;X \le C &lt;/math&gt;<br /> <br /> &lt;math&gt;-X \le C &lt;/math&gt;<br /> <br /> We can understand this visually as the solution &lt;math display=&quot;inline&quot;&gt;X &lt;/math&gt; must lie between &lt;math display=&quot;inline&quot;&gt;-C<br /> &lt;/math&gt; and &lt;math display=&quot;inline&quot;&gt;C &lt;/math&gt;, as shown below:<br /> [[File:Number Line X Less Than C.png|none|thumb]]<br /> <br /> The last case for linear constraints is when &lt;math&gt;|X| \ge C &lt;/math&gt;.<br /> <br /> Visually, the solution space is the complement of the second solution above, resulting in the following representation:<br /> [[File:Number Line for X Greater Than C.png|none|thumb]]<br /> <br /> In expression form, the solutions can be written as:<br /> <br /> &lt;math&gt;X \ge C &lt;/math&gt;<br /> <br /> &lt;math&gt;-X \ge C &lt;/math&gt;<br /> <br /> As seen visually, the feasible region has a gap and thus non-convex. The expressions also make it impossible for both to simultaneously hold true. This means that it is not possible to transform constraints in this form to linear equations. An approach to reach a solution for this particular case exists in the form of [[Mixed-Integer Linear Programming]], where only one of the equations above is “active”.<br /> <br /> === Absolute Values in Objective Functions ===<br /> In objective functions, to leverage transformations of absolute functions, several conditions must be fulfilled:<br /> <br /> # First, the constraints must all be linear.<br /> # Secondly, the coefficient signs of the absolute terms must all be:<br /> #* Positive for a minimization problem<br /> #* Negative for a maximization problem<br /> <br /> At a high level, the transformation works similarly to the second case of absolute value in constraints – we aim to bound the solution space for the absolute value term with a new variable, &lt;math display=&quot;inline&quot;&gt;Y &lt;/math&gt;.<br /> <br /> If &lt;math&gt;|X| &lt;/math&gt; is the absolute value term in our objective function, where &lt;math display=&quot;inline&quot;&gt;X &lt;/math&gt; is a linear combination of variables,<br /> <br /> Two additional constraints are added to the linear program:<br /> <br /> &lt;math&gt;X \le Y &lt;/math&gt;<br /> <br /> &lt;math&gt;-X \le Y &lt;/math&gt;<br /> <br /> This is a relaxation of the original function into a collection of linear constraints.<br /> <br /> In order to transform problems where the coefficient signs of the absolute terms do not fulfill the conditions above, a similar conclusion is reached to that of the last case for absolute values in constraints – the use of integer variables is needed to reach an LP format.<br /> <br /> ==Numerical Example==<br /> <br /> &lt;math&gt;\min{|x_1| + 2|x_2| + |x_3|} &lt;/math&gt;<br /> <br /> &lt;math&gt; \begin{align}<br /> \ s.t. x_1 + x_2 - x_3 \le 10 \\<br /> x_1 - 3x_2 + 2x_3= 12<br /> \end{align}&lt;/math&gt;<br /> <br /> We replace the absolute value quantities with a single variable:<br /> <br /> &lt;math&gt;|x_1| = U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt;|x_2| = U_2&lt;/math&gt;<br /> <br /> &lt;math&gt;|x_3| = U_3&lt;/math&gt;<br /> <br /> We must introduce additional constraints to ensure we do not lose any information by doing this substitution:<br /> <br /> &lt;math&gt; -U_1 \le x_1 \le U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_2 \le x_2 \le U_2 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_3 \le x_3 \le U_3 &lt;/math&gt;<br /> <br /> The problem has now been reformulated as a linear programming problem that can be solved normally:<br /> <br /> &lt;math&gt;\min{ U_1 + 2U_2 + U_3} &lt;/math&gt;<br /> <br /> &lt;math&gt; \begin{align}<br /> \ s.t. x_1 + x_2 - x_3 \le 10 \\<br /> x_1 - 3x_2 + 2x_3= 12<br /> \end{align}&lt;/math&gt;<br /> <br /> &lt;math&gt; -U_1 \le x_1 \le U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_2 \le x_2 \le U_2 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_3 \le x_3 \le U_3 &lt;/math&gt;<br /> <br /> The optimum value for the objective function is &lt;math&gt;6&lt;/math&gt;, which occurs when &lt;math&gt;x_1 = 0 &lt;/math&gt; and &lt;math&gt;x_2 = 0 &lt;/math&gt; and &lt;math&gt;x_3 = 6 &lt;/math&gt;.<br /> <br /> == Applications ==<br /> There are no specific applications to Optimization with Absolute Values however it is necessary to account for at times when utilizing the simplex method.<br /> <br /> Consider the problem Ax=b; max z= x c,jx,i. This problem cannot, in general, be solved with the simplex method. The problem has a simplex method solution (with unrestricted basis entry) only if c, are nonpositive (non-negative for minimizing problems).<br /> <br /> The primary application of absolute-value functionals in linear programming has been for absolute-value or L(i)-metric regression analysis. Such application is always a minimization problem with all C(j) equal to 1 so that the required conditions for valid use of the simplex method are met. <br /> <br /> By reformulating the original problem into a Mixed-Integer Linear Program (MILP), in most case we should be able to use GAMS/Pyomo/JuliaOPT to solve the problem. <br /> <br /> WIP<br /> <br /> 3<br /> <br /> * '''Application in Financial:''' '''Portfolio Selection'''<br /> Under this topic, we are going to use the same tricks we played in the Numerical Example section to perform '''Reduction to a Linear Programming Problem''', to reform the problem into a MILP, in order to solve the problem. Let's have a look of the example. <br /> <br /> <br /> <br /> A portfolio is determined by what fraction of one's assets to put into each investment. It can be denoted as a collection of nonnegative numbers xj, where j = 1, 2,...,n. Because each xj stands for a portion of the assets, it sums to one. In order to get a highest reward through finding a right mix of assets, let &lt;math&gt;\mu&lt;/math&gt;, the positive parameter, denote the importance of risk relative to the return, and Rj denote the return in the next time period on investment j, j = 1, 2,..., n. The total return one would obtain from the investment is &lt;math&gt;R = \sum_{j}\!x_j\!R_j &lt;/math&gt;. The expected return is &lt;math&gt;\mathbb{E}\!R = \sum_{j}\!x_j\mathbb{E}\!R_j &lt;/math&gt;. And the Mean Absolute Deviation from the Mean (MAD) is &lt;math&gt;\mathbb{E}\left\vert \!R - \mathbb{E}\!R \right\vert = \mathbb{E}\left\vert \sum_{j}\!x_j\tilde{R}_j \right\vert &lt;/math&gt;. <br /> <br /> maximize &lt;math display=&quot;inline&quot;&gt;\mu\sum_j\!x_j\mathbb{E}\!R_j - \mathbb{E}\left\vert \sum_j \!x_j\tilde{R}_j \right\vert &lt;/math&gt;<br /> <br /> subject to &lt;math&gt;\sum_j\!x_j = 1&lt;/math&gt;<br /> <br /> &lt;math&gt;x_j \geq 0&lt;/math&gt; &lt;math&gt; j = 1,2,..n.&lt;/math&gt;<br /> <br /> where &lt;math&gt;\tilde{R}_j = \!R_j - \mathbb{E}\!R_j &lt;/math&gt;<br /> <br /> <br /> <br /> Very obviously, this problem is not a linear programming problem yet. So we have to use the same trick we used in the numeric example, to replace each absolute value with a new variable and impose inequality constraints to ensure that the new variable is the appropriate absolute value once an optimal value is obtained. To simplify the program, we decide to take the average of the historical returns in order to get the mean expected return: &lt;math&gt;r_j = \mathbb{E}\!R_j = \left ( \frac{1}{T} \right ) \sum_{t=1}^T \!R_j(t)<br /> &lt;/math&gt;. Thus the objective function turns into: &lt;math&gt;\mu\sum_{j}\!x_j\!r_j - \left ( \frac{1}{T} \right ) \sum_{t=1}^T\left\vert \sum_{j} \!x_j \bigl(R_j (t) - \!r_j\bigr) \right\vert <br /> &lt;/math&gt;<br /> <br /> Now, replace &lt;math&gt;\left\vert \sum_{j} \!x_j \bigl(R_j (t) - \!r_j\bigr) \right\vert <br /> &lt;/math&gt; with a new variable &lt;math&gt;y_t<br /> &lt;/math&gt;and thus the problem can be rewrote as:<br /> <br /> <br /> maximize &lt;math&gt;\mu \sum_j \!x_j\!r_j - \left ( \frac{1}{T} \right ) \sum_{t=1}^T \!y_t<br /> <br /> &lt;/math&gt;<br /> <br /> subject to &lt;math&gt;-\!y_t \leq \sum_{j} \!x_j \bigl(R_j (t) - \!r_j\bigr) \leq y_t <br /> &lt;/math&gt;. t = 1, 2,...,T<br /> <br /> where &lt;math&gt;\sum_j \!x_j = 1<br /> <br /> &lt;/math&gt;<br /> <br /> &lt;math&gt;x_j\geq 0<br /> <br /> &lt;/math&gt;. j = 1, 2,...,n<br /> <br /> &lt;math&gt;y_t \geq 0<br /> <br /> &lt;/math&gt;. t = 1, 2,...,T<br /> <br /> <br /> So finally, after some simplifications methods and some tricks applied, the original problem is converted into a linear programming which is easier to be solved further.<br /> <br /> <br /> <br /> <br /> Let's see some other applications:<br /> <br /> * '''Minimizing the sum of absolute deviations'''<br /> <br /> * '''Minimizing the maximum of absolute values'''<br /> <br /> == Conclusion ==<br /> The presence of an absolute value within the objective function prevents the use of certain optimization methods. Solving these problems requires that the function be manipulated in order to continue with linear programming techniques like the simplex method.<br /> <br /> == References ==<br /> # &quot;Absolute Values.&quot; ''lp_solve'', http://lpsolve.sourceforge.net/5.1/absolute.htm. Accessed 20 Nov. 2020.<br /> # ''Optimization Methods in Management Science / Operations Research.'' Massachusetts Institute of Technology, Spring 2013, https://ocw.mit.edu/courses/sloan-school-of-management/15-053-optimization-methods-in-management-science-spring-2013/tutorials/MIT15_053S13_tut04.pdf. Accessed 20 Nov. 2020.<br /> # https://www.ise.ncsu.edu/fuzzy-neural/wp-content/uploads/sites/9/2019/08/LP-Abs-Value.pdf<br /> # Vanderbei R.J. (2008) Financial Applications. In: Linear Programming. International Series in Operations Research &amp; Management Science, vol 114. Springer, Boston, MA. &lt;nowiki&gt;https://doi.org/10.1007/978-0-387-74388-2_13&lt;/nowiki&gt;</div> Bramado https://optimization.cbe.cornell.edu/index.php?title=Optimization_with_absolute_values&diff=1553 Optimization with absolute values 2020-11-22T03:43:26Z <p>Bramado: Intro/Conclusion/Formatting</p> <hr /> <div>Authors: Matthew Chan (mdc297), Yilian Yin (), Brian Amado (ba392), Peter (pmw99), Dewei Xiao (dx58) - SYSEN 5800 Fall 2020<br /> <br /> Steward: Fengqi You<br /> <br /> == Introduction ==<br /> Absolute values can make it relatively difficult to to determine the optimal solution when handled without first converting to standard form. This conversion of the objective function is a good first step in solving optimization problems with absolute values. As a result, you can go on to solve the problem using linear programing techniques.<br /> <br /> == Method ==<br /> <br /> === Defining Absolute Values ===<br /> An absolute value of a real number can be described as its distance away from zero, or the non-negative magnitude of the number. Thus,<br /> <br /> &lt;math&gt;|x| = \begin{cases} -x, &amp; \text{if }x &lt; 0 \\ x, &amp; \text{if }x \ge 0 \end{cases} &lt;/math&gt;<br /> <br /> Absolute values can exist in optimization problems in two primary instances: in constraints and in the objective function.<br /> <br /> === Absolute Values in Constraints ===<br /> Within linear equations, linear constraints can exist in several forms.<br /> <br /> The first form exists as &lt;math&gt;|X| = 0 &lt;/math&gt;, where &lt;math display=&quot;inline&quot;&gt;X &lt;/math&gt; is a linear combination of variables.<br /> <br /> In this case, the only solution is if &lt;math&gt;|X| = 0 &lt;/math&gt;, simplifying the constraint to &lt;math&gt;X = 0 &lt;/math&gt;. Note that this solution also occurs if the constraint is in the form &lt;math&gt;|X| &lt;= 0 &lt;/math&gt; due to the same conclusion (only solution &lt;math&gt;X = 0 &lt;/math&gt;).<br /> <br /> <br /> Second form a linear constraint can exist in is &lt;math&gt;|X| \le C &lt;/math&gt; where &lt;math display=&quot;inline&quot;&gt;X &lt;/math&gt; remains a linear combination of variables and constant &lt;math display=&quot;inline&quot;&gt;C &gt; 0 &lt;/math&gt;.<br /> <br /> In this case, we can describe an equivalent feasible solution by splitting the inequality into<br /> <br /> &lt;math&gt;X \le C &lt;/math&gt;<br /> <br /> &lt;math&gt;-X \le C &lt;/math&gt;<br /> <br /> We can understand this visually as the solution &lt;math display=&quot;inline&quot;&gt;X &lt;/math&gt; must lie between &lt;math display=&quot;inline&quot;&gt;-C<br /> &lt;/math&gt; and &lt;math display=&quot;inline&quot;&gt;C &lt;/math&gt;, as shown below:<br /> [[File:Number Line X Less Than C.png|none|thumb]]<br /> <br /> <br /> The last case for linear constraints is when &lt;math&gt;|X| \ge C &lt;/math&gt;.<br /> <br /> Visually, the solution space is the complement of the second solution above, resulting in the following representation:<br /> [[File:Number Line for X Greater Than C.png|none|thumb]]<br /> <br /> <br /> In expression form, the solutions can be written as:<br /> <br /> &lt;math&gt;X \ge C &lt;/math&gt;<br /> <br /> &lt;math&gt;-X \ge C &lt;/math&gt;<br /> <br /> As seen visually, the feasible region has a gap and thus non-convex. The expressions also make it impossible for both to simultaneously hold true. This means that it is not possible to transform constraints in this form to linear equations. An approach to reach a solution for this particular case exists in the form of [[Mixed-Integer Linear Programming]], where only one of the equations above is “active”.<br /> <br /> === Absolute Values in Objective Functions ===<br /> WIP<br /> <br /> ==Numerical Example==<br /> <br /> &lt;math&gt;\min{|x_1| + 2|x_2| + |x_3|} &lt;/math&gt;<br /> <br /> &lt;math&gt; \begin{align}<br /> \ s.t. x_1 + x_2 - x_3 \le 10 \\<br /> x_1 - 3x_2 + 2x_3= 12<br /> \end{align}&lt;/math&gt;<br /> <br /> We replace the absolute value quantities with a single variable:<br /> <br /> &lt;math&gt;|x_1| = U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt;|x_2| = U_2&lt;/math&gt;<br /> <br /> &lt;math&gt;|x_3| = U_3&lt;/math&gt;<br /> <br /> We must introduce additional constraints to ensure we do not lose any information by doing this substitution:<br /> <br /> &lt;math&gt; -U_1 \le x_1 \le U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_2 \le x_2 \le U_2 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_3 \le x_3 \le U_3 &lt;/math&gt;<br /> <br /> The problem has now been reformulated as a linear programming problem that can be solved normally:<br /> <br /> &lt;math&gt;\min{ U_1 + 2U_2 + U_3} &lt;/math&gt;<br /> <br /> &lt;math&gt; \begin{align}<br /> \ s.t. x_1 + x_2 - x_3 \le 10 \\<br /> x_1 - 3x_2 + 2x_3= 12<br /> \end{align}&lt;/math&gt;<br /> <br /> &lt;math&gt; -U_1 \le x_1 \le U_1 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_2 \le x_2 \le U_2 &lt;/math&gt;<br /> <br /> &lt;math&gt; -U_3 \le x_3 \le U_3 &lt;/math&gt;<br /> <br /> The optimum value for the objective function is &lt;math&gt;6&lt;/math&gt;, which occurs when &lt;math&gt;x_1 = 0 &lt;/math&gt; and &lt;math&gt;x_2 = 0 &lt;/math&gt; and &lt;math&gt;x_3 = 6 &lt;/math&gt;.<br /> <br /> == Applications ==<br /> There are no specific applications to Optimization with Absolute Values however it is necessary to account for at times when utilizing the simplex method.<br /> <br /> <br /> <br /> Consider the problem Ax=b; max z= x c,jx,i. This problem cannot, in general, be solved with the simplex method. The problem has a simplex method solution (with unrestricted basis entry) only if c, are nonpositive (non-negative for minimizing problems).<br /> <br /> The primary application of absolute-value functionals in linear programming has been for absolute-value or L(i)-metric regression analysis. Such application is always a minimization problem with all C(j) equal to 1 so that the required conditions for valid use of the simplex method are met. <br /> <br /> WIP<br /> <br /> 3<br /> <br /> == Conclusion ==<br /> The presence of an absolute value within the objective function prevents the use of certain optimization methods. Solving these problems requires that the function be manipulated in order to continue with linear programming techniques like the simplex method.<br /> <br /> == References ==<br /> To be formatted:<br /> <br /> # http://lpsolve.sourceforge.net/5.1/absolute.htm<br /> # https://ocw.mit.edu/courses/sloan-school-of-management/15-053-optimization-methods-in-management-science-spring-2013/tutorials/MIT15_053S13_tut04.pdf<br /> # https://www.ise.ncsu.edu/fuzzy-neural/wp-content/uploads/sites/9/2019/08/LP-Abs-Value.pdf</div> Bramado https://optimization.cbe.cornell.edu/index.php?title=Optimization_with_absolute_values&diff=932 Optimization with absolute values 2020-11-10T02:27:20Z <p>Bramado: </p> <hr /> <div>Authors: Matthew Chan (mdc297), Yilian Yin (), Brian Amado (ba392), Peter (pmw99), Dewei Xiao () - SYSEN 5800 Fall 2020<br /> <br /> Steward: Fengqi You</div> Bramado