Stochastic dynamic programming

From Cornell University Computational Optimization Open Textbook - Optimization Wiki
Jump to navigation Jump to search

Authors: Bo Yuan, Ali Amadeh, Max Greenberg, Raquel Sarabia Soto and Claudia Valero De la Flor (CHEME/SYSEN 6800, Fall 2021)


In real-world decision-making problems, uncertainty is very frequently a non-negligible factor. Uncertainty may exist in the estimation of model parameters, in input or output signals, in extraneous disturbances, or in general and unaccounted-for mismatch between a model and the real system it represents. However, such decision problems are still solvable, and stochastic dynamic programming in particular serves as a powerful tool to derive optimal decision policies despite the form of uncertainty present.

Stochastic dynamic programming as a method was first described in the 1957 white paper “A Markovian Decision Process” written by Richard Bellman for the Rand Corporation.[1] Bellman’s formulation of the problem was intended to address a general industrial equipment replacement problem, in which machinery used to produce a given item has some time-dependent probability of producing defective items or breaking down entirely. Since these probability distribution functions of failure over time are assumed to be invariant and the probability of failure at time t is independent of past realizations, the Markovian decision process Bellman describes represents a comparatively simple and well-behaved stochastic dynamic problem. However, modern stochastic dynamic programming has developed into an even more versatile tool, permitting both finite- and infinite-horizon formulations, dynamic trade-offs between caution and probing in solution hunting, and multiple underlying process chains.[2][3][4]

Theory, methodology and algorithm discussion


Stochastic dynamic programming combines stochastic programming and dynamic programming. Therefore, to understand better what it is, it is better first to give two definitions:[5]

  • Stochastic programming. Unlike in a deterministic problem, where a decision’s outcome is only determined by the decision itself and all the parameters are known, in stochastic programming there is uncertainty and the decision results in a distribution of transformations.
  • Dynamic programming. It is an optimization method that consists in dividing a complex problem into easier subprobems and solving them recursively to find the optimal sub-solutions which lead to the complex problem optima.

In any stochastic dynamic programming problem, we must define the following concepts:[6]

  • Policy, which is the set of rules used to make a decision.
  • Initial vector, where and is a finite closed region.
  • Choice made, where and is a set of possible choices.
  • Stochastic vector, .
  • Distribution function , associated with and dependent on and .
  • Return, which is the expected value of the function after the final stage.

In a stochastic dynamic programming problem, we assume that is known after the decision of stage has been made and before the decision of stage has to be made.

Methodology and algorithm

Formulation in a continuous state space

First, we define the N-stage return obtained using the optimal policy and starting with vector :

Where is the function of the final state .

Second, we define the initial transformation as , and , as the state resulting from it. The return after stages will be using the optimal policy. Therefore, we can formulate the expected return due to the initial choice made in :

Having defined that, the recurrence relation can be expressed as:


Formulation in a discrete state space

The continuous formulation presented takes the following form in a discrete state space. Let the state in stage be . Under action , the system transitions from state in stage to state in stage with probability . The maximum expected return in stage can be written as:

which gives the recurrence relation that is needed in any dynamic programming model.

The formulations presented are very general and, depending on the problem characteristics, different models can be developed. For this reason, we present the algorithm of a specific model as example: a model for Approximate Dynamic Programming (ADP).

Approximate Dynamic Programming (ADP)

Approximate dynamic programming (ADP) is an algorithm strategy for solving complex problems that can be stochastic. Since the topic of this page is Stochastic Dynamic Programming, we will discuss ADP from this perspective.

To develop the ADP algorithm, we present the Bellman’s equation using the expectation form.

Where and .

The variables used and their meanings are the following:

  • State of the system,
  • Function . It represents the policy to make a decision
  • Transition function, . It describes the transition from state to state .
  • Action taken in state ,
  • Information observed after taking action ,
  • gives the expected value of being in state at time and making a decision following the optimal policy.

The goal of ADP is to replace the value of with a statistical approximation . Therefore, after iteration , we have an approximation . Another feature of ADP is that it steps forward in time. To go from one iteration to the following, we define our decision function as:

Next, we define as the value of that solves this problem and , as the estimate value of being in state :

Finally, using the approximation lookup table, is defined. In a lookup approximation, for each state we have a that gives an approximated value of being in .

where is known as a stepsize

Therefore, a generic ADP algorithm applied to stochastic dynamic programming can be summarized as:[7]

1. We initialize for all states , and we choose an initial state and set .

2. For t=0,1,2.., we have to solve:

If :

If :

3. Set as long as .

Numerical examples

Gamblling example

We have 4 chips, and we want to bet them to maximize the probability of ending with 8 chips after a maximum of 4 bets. In each bet, we can bet as many chips as we have. The probability of losing is 0.4 (where we lose every chip we have bet) and the probability of wining is 0.6 (where we get twice the number of chips we have bet).


The stages of this problem are the 4 bets we can make (represented by ), where we decide how many chips to bet (action ). The state () are the number of chips we have before making a bet in each stage.

Bet 4, State s:

For , obviously, it does not matter whether to bet or not as there is no chance we can have eight chips at the end.

Bet 3, State s:

For , obviously, it does not matter whether to bet or not as there is no chance we can have eight chips at the end.

Bet 2, State s:

Bet 1, State s:

For this bet, we know that we have 4 chips, :

The probability of having eight chips is , and the sequence of optimal actions depends on not only the initial state, but also the disturbance (win/lose)

Applications: Problems

A stock-option model

This model was created to maximize the expected profit that we can obtain in N days (stages) from selling/buying stocks. This is considered a finite-stage model because we know in advance for how many days are we calculating the expected profit.

First, we define the stock price on the th day as . We assume the following:

Where … are independent of and between them, and identically distributed with distribution .

Second, we also assume that we have the chance to buy a stock at a fixed price and this stock can be sold at price . We then define as the maximal expected profit, and it satisfies the following optimality equation:

And the boundary condition is the following:

General gambling problem

Consider an unfair coin flip game where the probability of the coin landing on heads is and the probability of landing in tails is , and the game is each you flip head you win the amount you bet and each time you flip tails you lose the amount you bet as well. A gambler starts with a certain amount of money, cannot bet more money than he has and can play a limited number of games (the bet must be nonnegative). Our goal is to maximize the logarithm of his expected payout by using an optimal betting strategy. To find this we will use stochastic dynamic programming. Let be the number of bets, be the amount of money we have on each step, α the percentage of our money that we will bet, the probability of winning and the probability of losing :

With the boundary condition:

For the first bet:

The maximum of the previous equation is obtained at , so:


For :

Once again, the optimal bet will be , so:

It can be easily shown using induction that:

Being always the best bet .

Secretary problem

Suppose that we are to employ one secretary from N unknown candidates whom we can sequentially see and tell if the new secretary has a higher value than the ones already seen. The conditions are:

  • We can see each secretary only once
  • In each stage, we can
    • accept that secretary and the selection process will stop
    • reject that secretary and observe the next one without being able to go back to it.
  • For each candidate, we can measure if he/she is better than those previously seen.

Our goal is to develop a process to maximize the probability of employing the best secretary.

In each stage, we have 4 possible states:

  • if the secretary is already employed and is the best we have seen so far
  • if the secretary is already employed and is not the best we have seen so far
  • if the secretary is not yet employed, and the candidate we are seeing is the best we have seen so far
  • if the secretary is not yet employed, and the candidate we are seeing is not the best we have seen so far

Also, in each stage, we can make different decisions:

  • : accept (employ) the candidate we are seeing
  • : reject the candidate we are seeing
  • if we do nothing

Obviously, decisions “” and “” can be made only if the state is different from or . Thus, the set of allowable actions can be written as:

It goes without saying that if , we will always pass and reject the candidate.

Denoting each stage by , we know that the reward that can be obtained in each stage before the final stage is zero regardless of the state and the action taken. Thus, we can write:

In the final stage, if the best secretary has been already employed or if the secretary is the best we have seen so far, the reward is equal to , otherwise, the reward is equal to zero. Hence, we have:

The state transition probability matrix in stage under action can be defined as:

Now, we can start formulating the Stochastic Dynamic Programming problem. In stage , we have:

For stage , the value function depends on the state. For , Hence, the optimal value function under the only possible action can be written as:

For , the next state can be either with probability or with probability . Hence, the optimal value function under the only possible action can be written as:

For , the next state can be either with probability or with probability . Hence, the optimal value function under the only possible action can be written as:

For , the next state can be either or . The optimal value function under the two possible actions and can be written as:

Thus, the optimal action policy in stage and at each state can be written as follows:

It can be easily shown that the optimal value functions at state are monotonically non-increasing in t meaning:

The monotonicity of implies that the problem has a unique optimal stopping time, which can be obtained as:

Figure 1 illustrates the variation in and for .

Figure 1. Variation in and for

We know that:

If ,

Using induction, it is easy to show that:

Hence, for we have:

It can be easily shown that:

Knowing and the definition of , we can claim:

Therefore, we can write:

It can be concluded that:

As is an integer, we can write:

It is easy to show that as , and the probability of picking the best secretary .

Applications: Fields

Stochastic dynamic programming formulations have been proposed for a wide range of practical decision-making problems. These include, but are not limited to, water reservoir management,[8] water quality management,[9] production planning,[10] breeding policy in livestock herds,[11] energy allocation and reserve management,[12] drought mitigation,[13] fuel consumption optimization for hybrid vehicles,[14] and management of land for endangered species.[15] Here, we go over some of these problems in more detail.

Water resources and reservoir operation

Water needs to be stored in reservoirs for hydropower generation. The effect of storing water on downstream ecosystems has to be considered when constructing dams.[8] In fact, it is of paramount importance to let some water flow into the downstream to make sure the system is ecologically sustainable. The optimal flowrate released into the downstream, known as the minimum environmental flow, should be determined as it directly affects the hydropower generation and the associated benefit. Stochastic Dynamic Programming (SDP) has established itself as a powerful technique for analyzing hydropower systems and scheduling reservoir operation. The key state of such problems is the reservoir storage level. The objective is to maximize the associated revenue that can be obtained from hydropower generation. The SDP problem determines the optimal release rate in each stage. The problem is affected by the inherent uncertainty in the streamflows throughout the river basin.[16]

Production planning

The problem of production planning involves determining the production and, if applicable, the loss of the demand in each stage to minimize the total production and storage cost during a time horizon. The production capacity limits, warehouse capacity limits, lot size, and demand act as the constraints of this problem. Uncertainty is inherently available on account of erroneous predictions of the demand, processing cost, and resource availability.[10] A similar application is the problem of recycling via manufacturing, namely remanufacturing. The degree of complexity and uncertainty in those problems is even greater than that in regular production planning problems.[17] Apart from the enumerated uncertainties above, the quality of the returned products is uncertain as the products were used by different people under different conditions. In addition, the returns may be from different companies and of different types adding more uncertainty to the problem.

Animal husbandry

SDP has also been applied to the problem of determining the replacement policy that is economically optimal in swine breeding herds.[11] The objective is to maximize the total profit that can be achieved given the number of existing sows in the herd and also the number of the replacement gilts over a planning horizon. As every other dynamic programming problem, the process starts from the last stage and proceeds backwards. The value that can be achieved from the available sows at the final stage is their slaughter value. The decision is to either keep or replace a sow. The problem is prone to uncertainty owing to involuntary reasons for culling sows such as disease or death and different reproduction characteristic between individual sows available in the herd.


SDP has been utilized to deal with the problem of weed control. Sells has found SDP more powerful than simulation-based approaches in determining the optimal weed control strategies.[18] The weed seedbanks of the available weeds can be regarded as the state vector. The decision to be made can be the type of crop to cultivate, time of planting, cultivations prior to planting, type of herbicide, rate of herbicide used, and time of spraying herbicide. The objective function to be minimized can be the total expected cost over a long planning horizon. Uncertainty needs to be considered in such problems due to uncertain performance of herbicides.

Energy systems

SDP has been applied to the storage management problem. Xi et al. developed an SDP model for determining the optimal use of a residential energy storage for different applications, namely meeting the energy demand of a building and providing regulation reserves to the grid, under uncertainty.[12] The objective is to maximize the total expected profit that can be obtained over a planning horizon by participating in both the energy and ancillary services markets. The decision variables are the amount of energy that needs to be charged and discharged during each time step. The optimal decision depends on the price in the energy and ancillary services markets, whether a power outage has happened or not, and the building energy demand. The prices in the two markets and the uncertain building energy demand expose the problem to uncertainty. As another example, SDP was deployed for managing a hybrid energy storage of an electric vehicle under the uncertainty arising from the driving cycle.[19] The state vector in this problem consisted of the velocity, power, and state of charge of the added double layer capacitors (making the storage hybrid). The control input was the battery current, and the objective was to minimize the total expected cost comprising a term associated with the loss of energy and a penalty term penalizing the deviation from the medium state of charge. The penalty term was important to be included in the formulation of the cost function as the prediction horizon of the SDP was short.


Stochastic Dynamic Programming is a technique that recursively solves stochastic optimization problems. Given a known state at the beginning of a discrete time period, Stochastic Dynamic Programming determines the optimal action that maximizes (minimizes) the expected total reward (cost) that can be achieved in that stage and the following ones. Starting from the boundary condition (typically the last stage), the optimal action in each stage is recursively determined for different possible states. The sequence of optimal actions is then selected based on the initial state and the consequent state transitions. Clearly specifying the state and the stage is of the utmost importance in all dynamic programming problems. Unlike in deterministic problems, the probability of transition from one state to another also affects the reward (cost) obtained (incurred) and, in turn, affects the optimal action. Stochastic Dynamic Programming has been applied to a variety of applications in different research fields ranging from energy systems to animal husbandry.


  1. R. Bellman, "A Markovian Decision Process", The Rand Corporation, 1957.
  2. A. Hordijk and L. C. M. Kallenberg, "Constrained Undiscounted Stochastic Dynamic Programming", Mathematics of Operations Research, vol. 9, pp. 276-289, 1984.
  3. Y. Bar-Shalom, "Stochastic Dynamic Programming: Caution and Probing", IEEE Transactions on Automatic Control, vol. 26, pp. 1184-1195, 1981.
  4. J. Van der Wal, Stochastic Dynamic Programming, Eindhoven University of Technology, 2010.
  5. J. R. Birge and F. Louveaux, ”Introduction to Stochastic Programming” (2nd ed.), Springer, pp. 341-441, 2011.
  6. R. Bellman, ”Dynamic Programming” (6th ed.), Princeton University Press, pp. 65-115, 1972.
  7. W. B. Powell, ”What You Should Know About Approximate Dynamic Programming”, Wiley InterScience, pp. 239-249, 2008.
  8. 8.0 8.1 K. B. Cho, ”Stochastic Dynamic Programming (SDP) and Sample Stochastic Dynamic Programming (SSDP) for Optimization of Korean Hydropower Plant”, Computers and Operations Research, vol. 36, pp. 2418–2428, 2017.
  9. H. Cardwell and H. Hugh Ellis, "Stochastic Dynamic Programming Models for Water Quality Management", Water Resources Research, vol. 29, pp. 803-813, 1993.
  10. 10.0 10.1 M. P. Cristobal, L. F. Escudero, and J. F. Monge, ”On stochastic dynamic programming for solving large-scale planning problems under uncertainty”, Computers and Operations Research, vol. 36, pp. 2418–2428, 2009.
  11. 11.0 11.1 R. B. M. Huirne, ”Stochastic dynamic programming to support sow replacement decisions”, European Journal of Operational Research, vol. 67, pp. 161–171, 1993.
  12. 12.0 12.1 X. Xi, R. Sioshansi, and V. Marano, ”A stochastic dynamic programming model for co-optimization of distributed energy storage”, Energy Systems, vol. 5, pp. 475–505, 2014.
  13. H. Eum, Y. Kim and R. N. Palmer, "Optimal Drought Management Using Sampling Stochastic Dynamic Programming with a Hedging Rule", Journal of Water Resources Planning and Management, vol. 137, pp. 113-122, 2009
  14. L. Johannesson, M. Asbogard and B. Egardt, "Assessing the Potential of Predictive Control for Hybrid Vehicle Powertrains Using Stochastic Dynamic Programming," IEEE Transactions on Intelligent Transportation Systems, vol. 8, pp. 71-83, 2007.
  15. M. Westphal, "The Use of Stochastic Dynamic Programming in Optimal Landscape Reconstruction for Metapopulations", Ecological Applications, vol. 13, pp. 543-555, 2003.
  16. J. Kelman, ”Sampling stochastic dynamic programming applied to reservoir operation”, Water Resources Research, vol 26, pp. 447–454, 1990.
  17. C. Li, ”A stochastic dynamic programming based model for uncertain production planning of re-manufacturing system”, International Journal of Production Research, vol. 47, pp. 3657–3668, 2009.
  18. J. E. Sells, ”Optimising weed management using stochastic dynamic programming to take account of uncertain herbicide performance”, Agricultural Systems, vol. 48, pp. 271-296, 1995.
  19. C. Romaus, K. Gathmann and J. Böcker, ”Optimal energy management for a hybrid energy storage system for electric vehicles based on stochastic dynamic programming”, presented at the IEEE Vehicle Power and Propulsion conference, 2010.