Discrete-Time Markov Chains
A Discrete-Time Markov Chain (DTMC) is a mathematical model that describes a system which transitions between a finite or countable number of states over discrete time intervals. The key feature of a DTMC is that the future state depends only on the current state and not on the sequence of events that preceded it, a property known as the Markov property.
In a DTMC, each state has a set of probabilities that dictate the likelihood of moving to other states in the next time step. These probabilities are typically represented in a transition matrix, which summarizes the behavior of the system and allows for analysis of long-term behavior, such as steady-state distributions.