A student has asked, which of these is the correct way to write the DP equation for a deterministic plant equation $x_{t+1}=a(x_t,u_t,t)$.
\begin{align*}
F(x_t,t)& = \inf_{u_t} \Big[c(x_t,u_t) + F(a(x_t,u_t,t),t+1) \Big] \tag{1}\\
F(x,t)& = \inf_{u} \Big[c(x,u) + F(a(x,u,t),t+1) \Big]\tag{2}
\end{align*}
In fact, both are correct, since they express the same thing.
Recall that $F(x,t)$ is defined as $\inf_\pi C_t$, when $C_t$ is the total cost incurred over times $t,t+1,\dotsc,h$, given that we start in state $x$ at time $t$. Thus the two arguments of $F$ are the state and time. So (1) and (2) are saying the same thing. You may find it helpful to write $x_t$ as a reminder that the first argument is the state at time $t$, or you may prefer to just write $x$, since this is simpler. The choice is yours.
When I first encountered this subject the lecturer, Peter Whittle, was in the habit of aggressively omitting subscripts of $t$. I found that sometimes confusing, and I would go through my notes replacing $x$ by $x_t$. But actually, once one understands the meaning of (1) and (2) there is really no need to be confused. We can see (2) as more beautiful, and as an example of the dictum:
The typical stochastic DP equation is
\begin{equation}
F(x,t) = \inf_u \Big[ c(x,u,t) + E[F(x_{t+1},t+1) \mid x_t=x, u_t=u] \Big].\tag{3}
\end{equation}
Now it is helpful to show by $E[\ \cdot \ \mid x_t=x, u_t=u]$ that we are taking an expectation that is conditional on certain information, namely that $x_t=x$ and $u_t=u$. Remember that in (3) the random variable is $x_{t+1}$.
\begin{align*}
F(x_t,t)& = \inf_{u_t} \Big[c(x_t,u_t) + F(a(x_t,u_t,t),t+1) \Big] \tag{1}\\
F(x,t)& = \inf_{u} \Big[c(x,u) + F(a(x,u,t),t+1) \Big]\tag{2}
\end{align*}
In fact, both are correct, since they express the same thing.
Recall that $F(x,t)$ is defined as $\inf_\pi C_t$, when $C_t$ is the total cost incurred over times $t,t+1,\dotsc,h$, given that we start in state $x$ at time $t$. Thus the two arguments of $F$ are the state and time. So (1) and (2) are saying the same thing. You may find it helpful to write $x_t$ as a reminder that the first argument is the state at time $t$, or you may prefer to just write $x$, since this is simpler. The choice is yours.
When I first encountered this subject the lecturer, Peter Whittle, was in the habit of aggressively omitting subscripts of $t$. I found that sometimes confusing, and I would go through my notes replacing $x$ by $x_t$. But actually, once one understands the meaning of (1) and (2) there is really no need to be confused. We can see (2) as more beautiful, and as an example of the dictum:
Less is more (architect Ludwig Mies van der Rohe speaking of minimalist design)
The typical stochastic DP equation is
\begin{equation}
F(x,t) = \inf_u \Big[ c(x,u,t) + E[F(x_{t+1},t+1) \mid x_t=x, u_t=u] \Big].\tag{3}
\end{equation}
Now it is helpful to show by $E[\ \cdot \ \mid x_t=x, u_t=u]$ that we are taking an expectation that is conditional on certain information, namely that $x_t=x$ and $u_t=u$. Remember that in (3) the random variable is $x_{t+1}$.
