Lecture 14

On the basis of today's lecture you should be able to do Examples Sheet 3, #3, 4, 5.

I started the lecture with examples of two methods applied to solution of the continuous-time LQR problem.

Method 1 is the solution of this problem using the Hamilton Jacobi Bellman equation.

Method 2 foreshadows the use of Pontryagin's Maximum Principle, which we shall discuss in lectures 15, 16. You will need this for #6, 7, 8, 9, which should all be possible after lecture 15.

Question #10 I will probably do in lecture 16.

In the fishing example we were essentially using Theorem 14.1 to show that an optimal policy is one for which,
$$u=\begin{cases} 0\\ a(\bar x)\\ u_{\max} \end{cases}\text{as }\ x\begin{array}{c} <\\ =\\ > \end{array}\bar x$$
and that a necessary condition is that $a'(\bar x)=\alpha$. At the end of page 59 of the notes, $F(x)$ is computed at a function of $T$, the time at which $x$ first becomes equal to $\bar x$. A simple calculation, which I did not do in lecture, is that $F_{xx}<0$, so $F$ is concave. All this information confirms that $F$ satisfies the HJB equation (14.2).

Notice that Figure 2 on page 59 is showing $u_{\max}\geq a(x)$ for all $x$, so fishing at maximum rate does cause the population to decrease (unless initially $a(x)=u_{\max}$).