Revision as of 16:31, 11 December 2020

Author: Eljona Pushaj, Diana Bogdanowich, Stephanie Keomany
Steward: Fengqi You

Introduction

The eight-step procedures are a simplified, multi-stage approach for determining optimal solutions in mathematical optimization. Dynamic programming, developed by Richard Bellman in the 1950s¹, is used to solve for the maximization or minimization of the objective function by transforming the problem into smaller steps and enumerating all the different possible solutions and finding the optimal solution.

In the eight-step procedure, a problem can be broken down into subproblems to solve. Using the solutions from the subproblems in a recursive manner, the solution can be determined after all the solutions of the subproblems are calculated to find the best solution, which demonstrates the principle of optimality: Any optimal policy has the property that, whatever the current state and current decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the current decision.² Such a standard framework is used so that dynamic programming store the values of the subproblems to avoid recomputing, and thus, reduce time to solve the problem.³

Theory, Methodology, and/or Algorithmic Discussion

Methodology

To solve a problem using the 8-step procedure, one must use the following steps:⁴

Step 1: Specify the stages of the problem
The stages of a dynamic programming problem can be defined as points where decisions are made. These are often denoted with the variable $n$ .

Step 2: Specify the states for each stage
The states of a problem are defined as the knowledge necessary to make a decision. This is often denoted by $s$ . Often, $C$ is set equal to the maximum value of $s$ .

Step 3: Specify the allowable actions for each state in each stage
This stage varies greatly based on the problem presented.

Step 4: Describe the optimization function using an English-language description.

Step 5: Define the boundary conditions
This helps create a starting point to finding a solution to the problem.

Step 6: Define the recurrence relation
This is often noted by a function.

Step 7: Compute the optimal value from the bottom-up
This step can be done manually or by using programming.

Step 8: Arrive at the optimal solution
This is the final step.

Numerical Example

Weight capacity of C=5 and N=2

Item types are stages: n=1,2

Remaining capacity s= 1,2,3,4,5

Boundary Conditions:

$f_{n+1}^{*}(s)=0$ , s=0,1,2,3,4,5 C=5

$U_{2}(5)\,=\,0,1,...,min\left\{a[2],\left\lfloor {\frac {5}{w[2]}}\right\rfloor \right\}$ = {0,1,2}

$f_{2}^{*}(5)=max\left\{b[2,j]+f_{3}^{*}(5-j*w[2])\right\}$ =


Unused Capacity s	$f_{1}^{*}(s)$	$f_{2}^{*}(s)$	Type 2 opt $U_{2}^{*}(s)$
5	9	9	2
4	9	9	2
3	4	4	1
2	4	4	1
1	0	0	0
0	0	0	0

Applications

The following are some applications where dynamic programming is used. The criteria for applying dynamic programming to an optimization problem are if the objective function involves maximization, minimization, or counting and if the problem is determined by finding all the solutions to find the optimal solution.

Shortest/ Longest Path Problem

In the shortest path problem, the path with the least amount of cost or value must be determined in a problem with multiple nodes in between the beginning node s to the final node e. Travelling from one node to another incurs a value or cost c(p, q), and the objective is to reach t with the smallest cost possible. The eight-step procedure can be used to determine the possible solutions which the optimal solution can be determined from.⁵

Likewise, but in a maximization function, the longest path can be determined in a problem by determining the solution with the highest cost involved to travel from node s to node e.

Knapsack Problem

The knapsack problem is an example of determining the distribution of effort or when there are limited resources to be shared with competing entities, and the goal is to maximize the benefit of the distribution. Dynamic programming is used when the increase in benefit in regard to increasing the quantity of resources is not linearly proportional. The volume may also be considered in addition to the weight of the resources. A volume constraint is added to the problem and represented in the state by stage n by an ordered pair (s, v) for remaining weight and volume. By considering d constraints, the number of states can grow exponentially with a d -dimensional state space even if the value of d is small. The problem becomes infeasible to solve and is referred to as the curse of dimensionality. However, the curse has faded due to advances in computational power.⁶

Inventory Planning Problem

In inventory management, dynamic programming is used to determine how to meet anticipated and unexpected demand in order to minimize overall costs. Tracking an inventory system involves establishing a set of policies that monitor and control the levels of inventory, determining when a stock must be replenished, and the quantity of parts to order. For example, a production schedule can be computationally solved by knowing the demand, unit production costs, and inventory supply limits in order to keep the production costs below a certain rate.⁷

Conclusion

The eight-step procedure is an approach used in dynamic programming to transform a problem into simpler problems to yield an optimal solution. The recursive nature of the procedure allows for the optimization problems to be solved using computational models that reduce time and effort and can be used in many applications across many industries.

References

1. Bellman, Richard. “The Theory of Dynamic Programming.” Bulletin of American Mathematical Society, vol. 60, 1954, pp 503–515, https://www.ams.org/journals/bull/1954-60-06/S0002-9904-1954-09848-8/S0002-9904-1954-09848-8.pdf. 18 Nov 2020.

2. Bradley, Stephen P. Applied Mathematical Programming. Addison-Wesley. 1 February 1977. 320-342. 18 Nov 2020

3. Gavin-Hughes, Sam. “Dynamic Programming for Interviews.” Byte by Byte. https://www.byte-by-byte.com/dpbook/. 18 Nov 2020

4. You, Fengqi. “Dynamic Programming” 5 Oct 2020. Online. Microsoft PowerPoint presentation.

5. Neumann K. “Dynamic Programming Basic Concepts and Applications.” Optimization in Planning and Operations of Electric Power Systems. Physica, Heidelberg, 1993, p 31-56.

6. Taylor, C. Robert. Applications Of Dynamic Programming To Agricultural Decision Problems. United States, CRC Press, 2019.

7. Bellman, Richard. “Dynamic Programming Approach to Optimal Inventory Processes with Delay in Delivery.” Quarterly of Applied Mathematics, vol 18, 1961, p. 399-403, https://www.ams.org/journals/qam/1961-18-04/S0033-569X-1961-0118516-2/S0033-569X-1961-0118516-2.pdf. 19 Nov 2020

@@ Line 18: / Line 18: @@
 '''Step 2: Specify the states for each stage''' <br />
-The states of a problem are defined as the knowledge necessary to make a decision, or <math>s</math>. We set <math>C</math> equal to the maximum value of <math>s</math>. <br />
+The states of a problem are defined as the knowledge necessary to make a decision. This is often denoted by <math>s</math>. Often, <math>C</math> is set equal to the maximum value of <math>s</math>. <br />
 <br />
 '''Step 3: Specify the allowable actions for each state in each stage''' <br />
-This can be defined as: <br />
+This stage varies greatly based on the problem presented. <br />
-<math>
-U_{n}(s)\, or\, j\, =\, 0,1,...,min\left \{ a[n], \left \lfloor \frac{s}{w[n]} \right \rfloor \right \}
-</math> <br />
 <br />
 '''Step 4: Describe the optimization function using an English-language description.''' <br />
-In this sentence, we describe the optimization function for each state, or <math>s</math>, and each stage, or <math>n</math>. This can also be called <math>f^{*}_{n}(s)</math> <br />
 <br />
 '''Step 5: Define the boundary conditions''' <br />
-This helps create a starting point to finding a solution to the problem. First, we set <math>f^{*}_{n+1}(s) = 0</math> for all values of <math>s</math>. Here, we can note that <math> s=0,...,C </math> <br />
+This helps create a starting point to finding a solution to the problem. <br />
 <br />
 '''Step 6: Define the recurrence relation''' <br />
-During this step, we make an allowable decision involving <math>j</math> items for the remaining capacity <math>s</math> for items <math>n</math>. We can write this statement as: <br />
+This is often noted by a function. <br />
-<math> f^{*}_{n}(s)= \overset{max}{j=0,1,...,min\left \{ a[n],\left \lfloor \frac{s}{w[n]} \right \rfloor \right \}} \left \{ b[n,j]+ f^{*}_{n+1}(s-j*w[n]) \right \} </math> <br />
 <br />
 '''Step 7: Compute the optimal value from the bottom-up''' <br />
-In this step, a table is made containing all <math>s</math>, <math>f^{*}_{n}(s)</math>, and optimal values for all <math>n</math> variables. This step can be done manually or by using programming. <br />
+This step can be done manually or by using programming. <br />
 <br />
 '''Step 8: Arrive at the optimal solution''' <br />
-Once the value for <math>f^{*}_{n}(s)</math> is computed, we would look at the optimal decision that corresponds to the table entry for that value. We start with the optimal value for our first <math>n</math>, calculate our remaining space <math>s</math>, and use that value to arrive at an optimal value for all <math>n</math>. <br />
+This is the final step. <br />
 =Numerical Example=

Eight step procedures: Difference between revisions