Markov

Markov Chain vs Markov Process

Markov decision processes are an extension of Markov chains; the difference is the addition of actions (allowing choice) and rewards (giving motivation). Conversely, if only one action exists for each state and all rewards are the same (e.g., zero), a Markov decision process reduces to a Markov chain.

Recurrent vs Transient

Recurrent states

State $i$ is said to be persistent or recurrent if the probability of returning back to $i$, having started at $i$ is 1

$$P(X_n = i \text{ for some }n > 0 | X_0 = i) = 1$$

If you start at a Recurrent State, then you will for sure return back to that state at some point in the future.

Transient states

State $i$ is said to be transient if it is not guaranteed to return back to $i$, having started at ior probability of returning back to $i$ having started at $i$ is less than 1

$$P(X_n = i \text{ for some }n > 0 | X_0 = i) < 1$$

There is some positive probability that once you leave you will never return.