Suppose we have a latent variable model $p(x, z; \theta)$ with $z$ being the latent variable. The density of $x$ can be obtained using marginal probability over $z$:

Now to maximize likelihood, we need to maximize the following function:

However, the above equation is not concave with respect to $\theta$. Hence, we can not use gradient ascent to find the maximum likelihood estimate.

Moreover, if $p(x; z; \theta)$ is an exponential family distribution, then taking the derivative of the above equation with respect to $\theta$ doesn't typically lead to a solvable expression.

To get around this, we imagine $Q(z)$ to be some distribution over $z$ so that $\sum_z Q(z) = 1$ and $Q(z) \geq 0$

From Jensen's inequality, we also know that our inequality becomes an equality (the bound becomes right) when the expectation is over a constant:

It turns out that this bound is tight when:

For convenience, let's define Evidence Lower Bound (ELBO) as:

Therefore,

In the $E$ step we find the estimated distributions over $z^{(i)}$ using our current value of $\theta$ and thus our current estimated distribution over $x^{(i)}$ given $z^{(i)}$. We do so using the Bayes' rule:

In the $M$ step we update the value of $\theta$ to maximizes the $\text{ELBO}$ for which it is equal to $\log p(x; \theta)$ for our current values of $\theta$.

It can be shown that the EM algorithm converges to the maximum likelihood estimate of $\theta$ as the number of iterations goes to infinity.

We saw that the for any given value of $\theta$ and any distribution over $z$, our log-likelihood $\ell(\theta) = \log p(x;\theta)$ is always greater than the $\text{ELBO}(x; Q, \theta)$.

Hence,

Moreover, since in the $M$ step, $\theta^{(t+1)}$ is chosen to maximize $\text{ELBO}(x; Q^{(t)}, \theta^ {(t)})$:

Therefore,

Finally, in the $E$ step, $Q(z)$ is choosen such that for the current value of $\theta$, the log-likelihood and the evidence lower bound are equal.

Putting it all together, we have the following

Note that the reason we wanted a tight bound on our inequality was to guarantee convergence. Otherwise after the $E$ step, our $\text{ELBO}(x; Q^{(t)}, \theta^{(t)})$ could have been lower than our log-likelihood $\ell(\theta^{(t)})$. This would've meant that $\ell(\theta^{(t+1)})$ could have been lower than $\ell(\theta^{(t)})$.

The evidence lower bound can be rewritten as:

This would mean that maximizing $\text{ELBO}$ over $\theta$ is equivalent to maximizing $p(x | z; \theta)$ since the KL-divergence term does not depend on $\theta$.

Similarly, the evidence lower bound can also be rewritten as:

This shows us that $\text{ELBO}$ is maximized for a given value of $\theta$ when the KL divergence between $Q(z)$ and $p(z | x; \theta)$ is minimized, which is when they are equal. This is exactly what we saw while deriving the algorithm above.