Sustainable fishing

At the end of Lecture 13 I introduced the Hamilton-Jacobi-Bellman equation, which is a continuous time version of the optimality equation. In Section 13.3 (which I did not discuss in lectures) there is a presentation of the HJB equation for the continuous-time version of our well-studied LQ regulation problem. This is essentially Examples Sheet 3, Question 3. So you can review that section when you do the examples sheet. It is straightforward.

I have just now added to the notes a Section 13.4. This section is not examinable, but I thought you might find it interesting. The HJB equation is used in this example to deduce how fisherman might either fish to extinction, or not, a potentially sustainable fish population. The population follows dynamics of  $\dot x=a(x)-u$, when $x>0$, where $u$ is the rate at which fish are extracted. Fishermen are trying to maximize
$$\int_0^\infty u(t)  e^{-\alpha t}dt.$$ Whether the fish population is completely wiped out, or sustained, depends on whether the discounting rate $\alpha$ is greater or less than the rate at which the fish population can grow when it is small, i.e. $a'(0)$. If $\alpha< a'(0)$ then the population will converge to a sustainable positive level $\bar x$  and at which the optimal fishing rate is $\bar u$ and $\dot x=a(\bar x)-\bar u=0$.

We are not going to spend much time thinking about how to solve HJB equations directly, because the theory of Pontryagin's Maximum Principle that we will meet in Lecture 14 is more powerful. However, Questions 4 and 5 are about find a solution to the HJB equation, and you begin these questions by writing down the infinitesimal form of the optimality equation.

Lecture 13

The name "Kalman filter" refers to the estimation equation (13.1) and takes its name from Rudolf Kalman (1930 –), who developed it in the years 1958-64. He also coined the terms controllable and observable, and gave the criteria that we have seen in previous lectures. The fact that a system is controllable iff the matrix $[B\ AB\ \cdots\ A^{n-1}B]$ is of full rank is sometimes called Kalman's criteria. In the IEEE biography of Kalman it is stated

The Kalman filter, and its later extensions to nonlinear problems, represents perhaps the most widely applied by-product of modern control theory. It has been used in space vehicle navigation and control (e.g. the Apollo vehicle), radar tracking algorithms for ABM applications, process control, and socioeconomic systems.

The theory in this lecture is admittedly quite tricky - partly because the notation. As a test of memory, can you say what roles in the theory are taken by each of these?

 $x_t$,  $u_t$, $A$, $B$, $\epsilon_t$, $y_t$, $C$, $\eta_t$, $\hat x_t$, $\Delta_t$, $\xi_t$, $\zeta_t$, $R$, $S$, $Q$, $K_t$, $\Pi_t$, $N$, $L$, $M$, $H_t$,  $V_t$. 

 You will understand the ideas better once you have worked through the details of a scalar example (in which $n=m=p=1$). You do this in Example Sheet 3 Question 2. I also think that Question 1 is helpful in gaining an appreciation of the duality between control and estimation problems.

You will not be asked to reproduce the statement or proofs of Theorem 13.1 or 13.2 in examinations. You should simply know that $\hat{x}_t$ is computed from $\hat{x}_{t-1}$ and $y_t$ in the linear manner specified by (13.1), and that the covariance matrix $V_t$ satisfies a Riccati equation. You are not expected to memorize Riccati equations.

Notice that the Riccati equation for $V_t$, i.e. $V_t = g\, V_{t-1}$ runs in the opposite time direction to the one we had for $\Pi_t$ in lecture 10, where $\Pi_{t-1} = f\, \Pi_t$. We are given $V_0$ and $\Pi_h$.

Coda on the puzzle

Here is an amusing sporting fact that resonates with our puzzle in which a decision that is the same under conditions of A true and A false, may not be the right decision when the truth status of A is unknown.

John Howard (London School of Economics)  tells me that in cricket it can happen that a batsman attempts to drive the ball but appears to miss (or he may have snicked it). The ball then hits his pad, bounces off, and is caught by a slip fielder. There is an appeal for lbw (leg before wicket). The umpire would have given him out "lbw" if he was sure he had not hit the ball. He would be out "caught" if he had hit the ball. But a batsman cannot be given out unless a definite reason is given, and if the umpire is not sure which of these two it was, then batsman is not out.

I am told that a similar incident happened in the recently concluded India-Australia Test, where the bowler appealed for lbw or caught, the umpire upheld the appeal for lbw, but the batsman was upset thinking he was given caught out as he had not edged the ball. 

You can read more about this in the article, Ricky Ponting and the Judges, by Ian Rumfitt, who writes:

"In the first innings of the final Ashes Test of 2009, Ricky Ponting faced a ball which was deflected, off something, into the wicket-keeper’s hands. The English XI appealed, and in the agonizingly long time that it took Asad Rauf to decide, Jonathan Agnew (commentating on Test Match Special) reasoned as follows: ‘Either the ball hit Ponting’s bat or it hit his pads. If it hit his bat, he is out caught behind. If it hit his pads, he is out lbw. So, either way, he is out’. Rauf, however, appeared to disagree with Agnew’s reasoning, and Ponting stayed at the wicket."