Signomial problems
Author: Matthew Hantzmon (Che_345 Spring 2015)
Introduction
Sigmoid problems are a class of optimization problems with the objective of maximizing the sum of multiple sigmoid functions. They are defined by their limits at negative and positive infinity. Similar to the unit step function the function approaches 1 as it approaches infinity and approaches -1 as it approaches negative infinity. Sigmoid functions are shaped like an “S”, having both a convex and concave portion. The concave attributes will make solving the problems require a non-linear optimization, increasing computational burden. Though sigmoid problems are harder to solve than ordinary convex programs, they have many useful real-world applications which have encouraged their development. Sigmoid problems have become especially useful in the creation of artificial neural networks that simulate learning. The functionality of sigmoid problems allow in statistical models and other artificial learning.
The S-shape of the individual sigmoid functions made them good representatives of economies of scale, or other situations where the supplier is facing a declining demand function and increased investment leads to a lower average cost to produce each good. Within the sigmoidal problem, the sigmoidal functions within the objective represent that initial increases in investment will increase profitability when there is excess demand at the current price. The inflection point in the sigmoidal functions represents the point where the marginal profit for producing another good becomes 0. At this point the decrease in marginal cost from investing in more production is matched by an equal decrease in willingness to pay by the marginal consumer. Past this point the function determines that increased investment will not increase profitability. These properties of the sigmoid functions allow them to be ideal representatives of situations where initial investment is profitable though a threshold is reached where no additional inputs will be profitable. Other situations that exhibit these characteristics and can be modeled with sigmoidal problems include election planning, lottery prize design, and bidding at an auction. Election planners always desire to find the locations where spending on advertising will have the greatest effect on election results. When designing a lotter, the company wants to design a prize value that encourages many people to buy tickets while still being net profitable for the lottery. At an auction, a bidder may not have enough capital to buy every item they may desire so it is important early on to not waste a disproportionate amount of money winning an item that is not important. A sigmoidal problem maximizing utility can help determine the threshold value to bid on each item.
Sigmoidal Functions
Generally, a sigmoid function is any function having an “S” shape. One portion of the function will be convex while the other portion will be concave. The most common example of the sigmoid function is the logistic function shown in figure 1. However, the definition of sigmoidal functions is very broad: Any function that has real values, one inflection point and has a bell shaped first derivative function. With this definition, all logistic functions, the error function, the cumulative distribution function and many more can be considered sigmoidal functions and may be included in the objective function of a sigmoidal problem. In figure 1, the coefficient 1/c is the temperature constant. As the value of c increases in the sigmoid function, the sigmoid function will approach the shape of a step function.
Sigmoidal Problems
In practice, sigmoid problems consist of an optimization where the objective function is the sum of a set of sigmoid functions. Each of the sigmoid functions only has one input variable that determines the output value. Unless a certain threshold of input value has been exceeded, usually the value of the sigmoid function will be very close to zero. As a result, the solution of a sigmoid problem will consist of all the function input values either being 0 or a number exceeding the sigmoid function threshold. This threshold is depicted as being near z in figure 2. When optimizing inputs to achieve a desired output, usually there is a scarcity of inputs that must be allocated optimally; otherwise there would be no cause for a problem. Thus, the solution of the optimization will exceed the threshold z in as many indexes as possible. Any indexes that were not able to reach z input usually would have an input of 0. This is because, below z total inputs, two functions each with one unit of input will always have a lower combined utility than one sigmoid function with two units of input.
Applications of Sigmoidal Problems
These characteristics of sigmoidal problems make them very useful for certain types of optimizations. In presidential election planning, it is always important to spend campaign advertising dollars in a way which will have the largest impact on electoral count. Since the format of our election is winner-take-all in each state, the states that have the tightest poll results would be the states where the smallest change in poll numbers would most greatly change the overall electoral count. If a campaign team were trying to optimize their advertising spending they would want to create an objective that illustrated how their advertising would have an effect on each state. The campaign would only want to spend money on states that have a feasible change of voting in favor of their candidate. Money spend on states eventually lost is completely wasted. Sigmoidal functions will behave in the same way, only recommending to spend money in a state if they can pass the threshold of getting a majority of voters. They will predict that it is better to spend a lot of money on one state they have a good chance to influence than spend half each on two states where they may end up losing both.
The same logic follows for why sigmoidal problems are useful for optimizing bidding within an auction. Utility is only gained when a desired item is won, so it is always important to set up a function that only calculates utility when the necessary bidding threshold has been passed. There is no use or utility gained from bidding half the value on every item, and sigmoidal functions would appropriately capture this fact.
Sigmoid functions can be very useful in situations where utility gained is not continuous based on proportion but is more of a binary either/or result. They are not as extreme of a binary result as a step function, and the fact that they are fully differentiable makes them much more easily solved. However, since they have concave portions, their solution is not as simple as a linear program and may require software or approximations.
Large arrays of sigmoid problems have also been combined to simulate an artificial neural networks. These neural networks are sets of sigmoid functions used to approximate interconnected neurons, which are very powerful in statistical modeling. The neural networks are created by the aggregation of sigmoidal functions using the back-propagation algorithm. The most common applications of the sigmoidal problems is the density problem. The density of a linear space with uniform convergence can be calculated for a compact set using Cybenko's approximation theorem.
Solving Sigmoidal Problems
As one of the properties of a sigmoid function is concave portions, simple simplex optimization will not suffice. As computation power has increased in recent decades, solutions of sigmoidal problems has become much more feasible using software available online. Though, solutions with the appropriate software are fairly straightforward and simple, these sorts of problems can be highly difficult to solve by hand. They can rarely be solved at all in the original format and frequently need several creative approximations to leave a problem that is computable by hand. One method that allows simpler by-hand computation is the creation of a convex envelope. Depicted in figure 3, the concave envelope is an approximation of the sigmoidal function that adds a linear portion where the function had been previously convex. This addition, greatly simplifies the function’s computational difficulty. However, it must be noticed, that this approximation will only give an upper bound on the solution as it is including area that it not part of the original constraints. Even with function simplification and approximation, usually there is a large set of functions to be optimized. Depending on the level of approximation, the solution found may not even contain a relevant level of accuracy. In general, software will be the method of choice when dealing with sigmoidal problems.
References
1. J. Birge, F. Louveaux, Introduction to Stochastic Programming, Springer, 2011.
2. Udell, Madeline, and Stephen Boyd. "Maximizing a Sum of Sigmoids." Stanford University, May 2015. Web. May 2015. <http%3A%2F%2Fweb.stanford.edu%2F~udell%2Fdoc%2Fmax_sum_sigmoids>.
3. Srivastava, Vaibhav, and Francesco Bullo. "Knapsack Problems with Sigmoid Utilities." Princeton, July 2013. Web. May 2015. <http%3A%2F%2Fwww.princeton.edu%2F~vaibhavs%2Fpapers%2F2012d-sb>.
4. Rojas, R. "Backpropagation." SpringerReference (2011): n. pag. UBerlin. UBerlin. Web.