Today we looked at some stopping problems. Many problems are like this. They are present in financial mathematics in the theory of option pricing (Black-Scholes). The holder of an American option is allowed to exercise the right to buy (or sell) the underlying asset at a predetermined price at any time before or at the expiry date. The valuation of American options is essentially an optimal stopping problem.
You can read more about Bruss's odds algorithm in his paper: "Sum the odds to one and stop" Annals of Probability, Vol. 28, 1384–1391, (2000). I showed you how to use this to address the Secretary Problem. A very interesting related problem is the "last arrivals problem". Consider a shopkeeper who would like to go home early just after he has served the last potential customer of the day.
I will say more about the mine prospecting problem of Section 6.5 next time.
Let me finish with some hints about questions on Examples Sheets 1 and 2. Question 6 on Sheet 1 can be solved by realizing that there are only two stationary Markov policies. You simple need to decide which is best. Question 7 is a simple stopping problem that can be solved using a OSLA rule. Question 8 is tricky. You should use Theorem 4.2 to prove that $F(x)=x$ when $x\geq x^*=(1-q)/(q\lambda)$. But the difficult part of this question is to convince yourself that an optimal policy exists when $x<x^*$.
Question 9 on Sheet 1 is important because it should convince you that it is possible to solve stochastic dynamic programming equations using linear programming (at least if both state space and action space are finite.) It there are $n$ states and $m$ actions (controls) available in each state then we will have an LP in $n$ variables, with $nm$ linear inequality constraints.
When on sheet 2 you do Question 1 you might think about using Bruss's Odds Algorithm. Although you cannot apply the algorithm directly, you should be able to solve this problem using the same scheme of proof as we used to prove the optimality of the odds algorithm in this lecture.
You can read more about Bruss's odds algorithm in his paper: "Sum the odds to one and stop" Annals of Probability, Vol. 28, 1384–1391, (2000). I showed you how to use this to address the Secretary Problem. A very interesting related problem is the "last arrivals problem". Consider a shopkeeper who would like to go home early just after he has served the last potential customer of the day.
I will say more about the mine prospecting problem of Section 6.5 next time.
Let me finish with some hints about questions on Examples Sheets 1 and 2. Question 6 on Sheet 1 can be solved by realizing that there are only two stationary Markov policies. You simple need to decide which is best. Question 7 is a simple stopping problem that can be solved using a OSLA rule. Question 8 is tricky. You should use Theorem 4.2 to prove that $F(x)=x$ when $x\geq x^*=(1-q)/(q\lambda)$. But the difficult part of this question is to convince yourself that an optimal policy exists when $x<x^*$.
Question 9 on Sheet 1 is important because it should convince you that it is possible to solve stochastic dynamic programming equations using linear programming (at least if both state space and action space are finite.) It there are $n$ states and $m$ actions (controls) available in each state then we will have an LP in $n$ variables, with $nm$ linear inequality constraints.
When on sheet 2 you do Question 1 you might think about using Bruss's Odds Algorithm. Although you cannot apply the algorithm directly, you should be able to solve this problem using the same scheme of proof as we used to prove the optimality of the odds algorithm in this lecture.
