From time to time I will add a post to this blog which I hope will add to your appreciation of the course. This post relates to today's lecture, in which we had the idea of a state-structured model. The notion of a state is intuitive, or is it?
Warren B. Powell (a professor at Princeton and researcher in optimization) has mused upon this question. You can read his blog about this here: http://www.castlelab.princeton.edu/cso.htm
I think he makes a nice definition of a state, in these words:
A state variable is a minimally dimensioned function of history that is necessary and sufficient to compute the decision function, cost/reward/contribution function, and the transition function.
By "decision function" he means the $F(x,t)$ that I was today calling the "value function". For the other terms he is thinking of $c(x,u,t)$ and $a(x,u,t)$.
He discusses the fact that "state" can be a physical state, an information state, or a belief state.
Please be aware that a state can be a vector of more that one component. For example, consider the uncontrolled autoregressive plant equation,
$x_t = a_1 x_{t-1} + a_2 x_{t-2}$.
Here we would take the state to be $y_t=(x_t,x_{t-1})$, because $y_{t-1}$ is needed to compute $y_t$.
Warren is particularly interested reconciling the many communities of stochastic optimization, amongst which he includes
"What is a state?"
Warren B. Powell (a professor at Princeton and researcher in optimization) has mused upon this question. You can read his blog about this here: http://www.castlelab.princeton.edu/cso.htm
I think he makes a nice definition of a state, in these words:
A state variable is a minimally dimensioned function of history that is necessary and sufficient to compute the decision function, cost/reward/contribution function, and the transition function.
He discusses the fact that "state" can be a physical state, an information state, or a belief state.
Please be aware that a state can be a vector of more that one component. For example, consider the uncontrolled autoregressive plant equation,
$x_t = a_1 x_{t-1} + a_2 x_{t-2}$.
Here we would take the state to be $y_t=(x_t,x_{t-1})$, because $y_{t-1}$ is needed to compute $y_t$.
Warren is particularly interested reconciling the many communities of stochastic optimization, amongst which he includes
- Dynamic programming (including Markov decision processes)
- Stochastic programming (two-stage, multistage, chance constrained)
- Stochastic search (including simulation optimization, infinitessimal perturbation analysis, and ranking and selection)
- Optimal control (primarily engineering, but also economics and finance)
- Approximate dynamic programming and reinforcement learning
- ...
