Probabilistic Models

Reference:

Bishop C M. Pattern recognition and machine learning[M]. springer, 2006.

Slides of CS4220, TUD

Bayesian Inference

Bayes’ Theorem for Gaussian Variables (2.3.1-2.3.3)

Suppose $\mathbf{x}$ is a $D$-dimensional vector with Gaussian distribution $\mathcal{N}(\mathbf{x}|\mathbf{\mu },\mathbf{\Sigma })$ and that we partition $\mathbf{x}$ into two disjoint subsets ${\mathbf{x}}_a$ and ${\mathbf{x}}_b$. Without loss of generality, we can take ${\mathbf{x}}_a$ to form the first $M$ components of $\mathbf{x}$, with ${\mathbf{x}}_b$ comprising the remaining $D-M$ components, i.e.
$$\mathbf{x}=\left(\begin{array}{c}{\mathbf{x}}_a\\ {\mathbf{x}}_b\end{array}\right)\text{\hspace{1em}(BI.1)}$$
We also define corresponding partitions of the mean vector $\mathbf{\mu }$ given by
$$\mathbf{\mu }=\left(\begin{array}{c}{\mathbf{\mu }}_a\\ {\mathbf{\mu }}_b\end{array}\right)\text{\hspace{1em}(BI.2)}$$
and of the covariance matrix $\mathbf{\Sigma }$ given by
$$\mathbf{\Sigma }=\left(\begin{array}{cc}{\mathbf{\Sigma }}_{aa}& {\mathbf{\Sigma }}_{ab}\\ {\mathbf{\Sigma }}_{ba}& {\mathbf{\Sigma }}_{bb}\end{array}\right)\text{\hspace{1em}(BI.3)}$$
Note that the symmetry ${\mathbf{\Sigma }}^{\mathrm{T}}=\mathbf{\Sigma }$ of the covariance matrix implies that ${\mathbf{\Sigma }}_{aa}$ and ${\mathbf{\Sigma }}_{bb}$ are symmetric, while ${\mathbf{\Sigma }}_{ba}={\mathbf{\Sigma }}_{ab}^{\mathrm{T}}$.

In many situations, it will be convenient to work with the inverse of the covariance matrix
$$\mathbf{\Lambda }\equiv {\mathbf{\Sigma }}^{-1}\text{\hspace{1em}(BI.4)}$$
which is known as the precision matrix. In fact, we shall see that some properties of Gaussian distributions are most naturally expressed in terms of the covariance, whereas others take a simpler form when viewed in terms of the precision. We therefore also introduce the partitioned form of the precision matrix
$$\mathbf{\Lambda }=\left(\begin{array}{ll}{\mathbf{\Lambda }}_{aa}& {\mathbf{\Lambda }}_{ab}\\ {\mathbf{\Lambda }}_{ba}& {\mathbf{\Lambda }}_{bb}\end{array}\right)\text{\hspace{1em}(BI.5)}$$
Because the inverse of a symmetric matrix is also symmetric, we see that ${\mathbf{\Lambda }}_{aa}$ and ${\mathbf{\Lambda }}_{bb}$ are symmetric, while ${\mathbf{\Lambda }}_{ab}^T={\mathbf{\Lambda }}_{ba}$.

An important property of the multivariate Gaussian distribution is that if two sets of variables are jointly Gaussian, then the conditional distribution of one set conditioned on the other is again Gaussian. Similarly, the marginal distribution of either set is also Gaussian.

Using the expression above, since ${\mathbf{x}}_a$ and ${\mathbf{x}}_b$ are jointly Gaussian, then the conditional distributions $p({\mathbf{x}}_a|{\mathbf{x}}_b)$ and $p({\mathbf{x}}_b|{\mathbf{x}}_a)$, and the marginal distributions $p({\mathbf{x}}_a)$ and $p({\mathbf{x}}_b)$ are also Gaussian. We will show how to prove this.

Conditional Gaussian distributions

Let us begin by finding an expression for the conditional distribution $p({\mathbf{x}}_a|{\mathbf{x}}_b)$. It can be evaluated from the joint distribution $p(\mathbf{x})=p({\mathbf{x}}_a,{\mathbf{x}}_b)$ by
$$p({\mathbf{x}}_a|{\mathbf{x}}_b)=\frac{p({\mathbf{x}}_a,{\mathbf{x}}_b)}{p({\mathbf{x}}_b)}=\frac{p({\mathbf{x}}_a,{\mathbf{x}}_b)}{\int p({\mathbf{x}}_a,{\mathbf{x}}_b)d{\mathbf{x}}_a}$$
where
$$p(\mathbf{x})=p({\mathbf{x}}_1,{\mathbf{x}}_2)=\frac{1}{\sqrt{(2\pi {)}^N|\mathbf{\Sigma }|}}\exp \left(-\frac{1}{2}(\mathbf{x}-\mathbf{\mu }{)}^T{\mathbf{\Sigma }}^{-1}(\mathbf{x}-\mathbf{\mu })\right)$$
It can be viewed as fixing ${\mathbf{x}}_b$ to the observed value and normalizing the resulting expression to obtain a valid probability distribution over ${\mathbf{x}}_a$. Therefore, we can first ignore $p({\mathbf{x}}_b)$ and focus on the quadratic form in the exponent of the Gaussian distribution $p({\mathbf{x}}_a,{\mathbf{x}}_b)$ and then reinstating the normalization coefficient at the end of the calculation.

If we make use of the partitioning $(BI.1)$, $(BI.2)$ and $(BI.5)$, we obtain
$$\begin{array}{rl}& -\frac{1}{2}(\mathbf{x}-\mathbf{\mu }{)}^T{\mathbf{\Sigma }}^{-1}(\mathbf{x}-\mathbf{\mu })=\\ & -\frac{1}{2}({\mathbf{x}}_a-{\mathbf{\mu }}_a{)}^T{\mathbf{\Lambda }}_{aa}^{-1}({\mathbf{x}}_a-{\mathbf{\mu }}_a)-\frac{1}{2}({\mathbf{x}}_a-{\mathbf{\mu }}_a{)}^T{\mathbf{\Lambda }}_{ab}^{-1}({\mathbf{x}}_b-{\mathbf{\mu }}_b)\\ & -\frac{1}{2}({\mathbf{x}}_b-{\mathbf{\mu }}_b{)}^T{\mathbf{\Lambda }}_{ba}^{-1}({\mathbf{x}}_a-{\mathbf{\mu }}_a)-\frac{1}{2}({\mathbf{x}}_b-{\mathbf{\mu }}_b{)}^T{\mathbf{\Lambda }}_{bb}^{-1}({\mathbf{x}}_b-{\mathbf{\mu }}_b)\end{array}\text{\hspace{1em}(BI.6)}$$
We see that as a function of ${\mathbf{x}}_a$, this is again a quadratic form, and hence the corresponding conditional distribution $p({\mathbf{x}}_a|{\mathbf{x}}_b)$ will be Gaussian.

Because this distribution is completely characterized by its mean and its covariance, our goal will be to identify expressions for the mean and covariance of $p({\mathbf{x}}_a|{\mathbf{x}}_b)$ by inspection of $(BI.6)$. Such problems can be solved straightforwardly by noting that the exponent in a general Gaussian distribution $\mathcal{N}(\mathbf{x}|\mathbf{\mu },\mathbf{\Sigma })$ can be written
$$-\frac{1}{2}(\mathbf{x}-\mathbf{\mu }{)}^T{\mathbf{\Sigma }}^{-1}(\mathbf{x}-\mathbf{\mu })=-\frac{1}{2}{\mathbf{x}}^T{\mathbf{\Sigma }}^{-1}\mathbf{x}+{\mathbf{x}}^T{\mathbf{\Sigma }}^{-1}\mathbf{\mu }+\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{t}\text{\hspace{1em}(BI.7)}$$
Thus if we take our general quadratic form and express it in the form given by the right-hand side of $(BI.7)$, then we can immediately equate the matrix of coefficients entering the second order term in $\mathbf{x}$ to the inverse covariance matrix ${\mathbf{\Sigma }}^{-1}$ and the coefficient of the linear term in $\mathbf{x}$ to ${\mathbf{\Sigma }}^{-1}\mathbf{\mu }$, from which we can obtain $\mathbf{\mu }$.

Now let us apply this procedure to $(BI.6)$. Denote the mean and covariance of this distribution by ${\mathbf{\mu }}_{a|b}$ and ${\mathbf{\Sigma }}_{a|b}$, respectively. If we pick out all terms that are second order in ${\mathbf{x}}_a$, we have
$$-\frac{1}{2}{\mathbf{x}}_a^T{\mathbf{\Lambda }}_{aa}{\mathbf{x}}_a$$
from which we can immediately conclude that the covariance (inverse precision) of $p({\mathbf{x}}_a|{\mathbf{x}}_b)$ is given by
$${\mathbf{\Sigma }}_{a|b}={\mathbf{\Lambda }}_{aa}^{-1}\text{\hspace{1em}(BI.8)}$$
Now consider all of the terms in $(BI.6)$ that are linear in ${\mathbf{x}}_a$
x a T { Λ a a μ a − Λ a b ( x b − μ b ) } \mathbf{x}_{a}^{\mathrm{T}}\left\{\mathbf{\Lambda}_{a a} \boldsymbol{\mu}_{a}-\boldsymbol{\Lambda}_{a b}\left(\mathbf{x}_{b}-\boldsymbol{\mu}_{b}\right)\right\} xaT​{Λaa​μa​−Λab​(xb​−μb​)}
where we have used ${\mathbf{\Lambda }}_{ba}^{\mathrm{T}}={\mathbf{\Lambda }}_{ab}.$ From our discussion of the general form $(BI.7)$ the coefficient of ${\mathbf{x}}_a$ in this expression must equal ${\mathbf{\Sigma }}_{a|b}^{-1}{\mathbf{\mu }}_{a|b}$ and hence
μ a ∣ b = Σ a ∣ b { Λ a a μ a − Λ a b ( x b − μ b ) } = B I . 8 μ a − Λ a a − 1 Λ a b ( x b − μ b ) (BI.9) \begin{aligned} \boldsymbol{\mu}_{a | b} =\boldsymbol{\Sigma}_{a | b}\left\{\boldsymbol{\Lambda}_{a a} \boldsymbol{\mu}_{a}-\boldsymbol{\Lambda}_{a b}\left(\mathbf{x}_{b}-\boldsymbol{\mu}_{b}\right)\right\} \stackrel{BI.8}{=}\boldsymbol{\mu}_{a}-\boldsymbol{\Lambda}_{a a}^{-1} \boldsymbol{\Lambda}_{a b}\left(\mathbf{x}_{b}-\boldsymbol{\mu}_{b}\right) \end{aligned}\tag{BI.9} μa∣b​=Σa∣b​{Λaa​μa​−Λab​(xb​−μb​)}=BI.8μa​−Λaa−1​Λab​(xb​−μb​)​(BI.9)
We can also express these results in terms of the corresponding partitioned covariance matrix. To do this, we make use of the following identity for the inverse of a partitioned matrix
$${\left(\begin{array}{cc}\mathbf{A}& \mathbf{B}\\ \mathbf{C}& \mathbf{D}\end{array}\right)}^{-1}=\left(\begin{array}{cc}\mathbf{M}& -\mathbf{M}\mathbf{B}{\mathbf{D}}^{-1}\\ -{\mathbf{D}}^{-1}\mathbf{C}\mathbf{M}& {\mathbf{D}}^{-1}+{\mathbf{D}}^{-1}{\mathbf{C}\mathbf{M}\mathbf{B}\mathbf{D}}^{-1}\end{array}\right)\text{\hspace{1em}(BI.10)}$$
where we have defined
$$\mathbf{M}={\left(\mathbf{A}-\mathbf{B}{\mathbf{D}}^{-1}\mathbf{C}\right)}^{-1}\text{\hspace{1em}(BI.11)}$$
The quantity ${\mathrm{M}}^{-1}$ is known as the Schur complement of the matrix on the left-hand side of $(BI.10)$ with respect to the submatrix $\mathbf{D}$. Using the definition
$${\left(\begin{array}{cc}{\mathbf{\Sigma }}_{aa}& {\mathbf{\Sigma }}_{ab}\\ {\mathbf{\Sigma }}_{ba}& {\mathbf{\Sigma }}_{bb}\end{array}\right)}^{-1}=\left(\begin{array}{cc}{\mathbf{\Lambda }}_{aa}& {\mathbf{\Lambda }}_{ab}\\ {\mathbf{\Lambda }}_{ba}& {\mathbf{\Lambda }}_{bb}\end{array}\right)$$
and making use of $(BI.10),$ we have
$$\begin{array}{rl}{\mathbf{\Lambda }}_{aa}& ={\left({\mathbf{\Sigma }}_{aa}-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}\right)}^{-1}\\ {\mathbf{\Lambda }}_{ab}& =-{\left({\mathbf{\Sigma }}_{aa}-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}\right)}^{-1}{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}\end{array}$$
From this we obtain the following expressions for the mean and covariance of the conditional distribution $p({\mathbf{x}}_a|{\mathbf{x}}_b)$
$${\mathbf{\mu }}_{a|b}={\mathbf{\mu }}_a+{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}({\mathbf{x}}_b-{\mathbf{\mu }}_b)\text{\hspace{1em}(BI.12)}$$

$${\mathbf{\Sigma }}_{a|b}={\mathbf{\Sigma }}_{aa}-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}\text{\hspace{1em}(BI.13)}$$

Marginal Gaussian distributions

Now we turn to a discussion of the marginal distribution given by
$$p({\mathbf{x}}_a)=\int p({\mathbf{x}}_a,{\mathbf{x}}_b)d{\mathbf{x}}_b\text{\hspace{1em}(BI.14)}$$
Once again, our strategy for evaluating this distribution efficiently will be to focus on the quadratic form in the exponent of the joint distribution $p({\mathbf{x}}_a,{\mathbf{x}}_b)$ and thereby to identify the mean and covariance of the marginal distribution $p({\mathbf{x}}_a)$.

The quadratic form for the joint distribution can be expressed, using the partitioned precision matrix, in the form $(BI.6)$. Because our goal is to integrate out ${\mathbf{x}}_b$, this is most easily achieved by first considering the terms involving ${\mathbf{x}}_b$ and then completing the square in order to facilitate integration. Picking out just those terms that involve ${\mathbf{x}}_b$, we have
$$-\frac{1}{2}{\mathbf{x}}_b^{\mathrm{T}}{\mathbf{\Lambda }}_{bb}{\mathbf{x}}_b+{\mathbf{x}}_b^T\mathbf{m}=-\frac{1}{2}{\left({\mathbf{x}}_b-{\mathbf{\Lambda }}_{bb}^{-1}\mathbf{m}\right)}^{\mathrm{T}}{\mathbf{\Lambda }}_{bb}\left({\mathbf{x}}_b-{\mathbf{\Lambda }}_{bb}^{-1}\mathbf{m}\right)+\frac{1}{2}{\mathbf{m}}^{\mathrm{T}}{\mathbf{\Lambda }}_{bb}^{-1}\mathbf{m}\text{\hspace{1em}(BI.15)}$$
where we have defined
$$\mathbf{m}={\mathbf{\Lambda }}_{bb}{\mathbf{\mu }}_b-{\mathbf{\Lambda }}_{ba}\left({\mathbf{x}}_a-{\mathbf{\mu }}_a\right)\text{\hspace{1em}(BI.16)}$$
We see that the dependence on ${\mathbf{x}}_b$ has been cast into the standard quadratic form of a Gaussian distribution corresponding to the first term on the right-hand side of $(BI.15)$ plus a term that does not depend on ${\mathbf{x}}_b$ (but that does depend on ${\mathbf{x}}_a$). Thus, when we take the exponential of this quadratic form, we see that the integration over ${\mathbf{x}}_b$ required by $(BI.14)$ will take the form
$$\begin{array}{rl}& \int \exp \left\{-\frac{1}{2}{\left({\mathbf{x}}_b-{\mathbf{\Lambda }}_{bb}^{-1}\mathbf{m}\right)}^{\mathrm{T}}{\mathbf{\Lambda }}_{bb}\left({\mathbf{x}}_b-{\mathbf{\Lambda }}_{bb}^{-1}\mathbf{m}\right)\right\}\mathrm{d}{\mathbf{x}}_b\\ =& \sqrt{(2\pi {)}^N|{\mathbf{\Lambda }}_{bb}^{-1}|}\int \frac{1}{\sqrt{(2\pi {)}^N|{\mathbf{\Lambda }}_{bb}^{-1}|}}\exp \left\{-\frac{1}{2}{\left({\mathbf{x}}_b-{\mathbf{\Lambda }}_{bb}^{-1}\mathbf{m}\right)}^{\mathrm{T}}{\mathbf{\Lambda }}_{bb}\left({\mathbf{x}}_b-{\mathbf{\Lambda }}_{bb}^{-1}\mathbf{m}\right)\right\}\mathrm{d}{\mathbf{x}}_b\\ =& \sqrt{(2\pi {)}^N|{\mathbf{\Lambda }}_{bb}^{-1}|}\end{array}\text{\hspace{1em}(BI.17)}$$
This integration is easily performed by noting that it is the integral over an unnormalized Gaussian, and so the result will be the reciprocal of the normalization coefficient. Thus, by completing the square with respect to ${\mathbf{x}}_b$, we can integrate out ${\mathbf{x}}_b$ and the only term remaining from the contributions on the left-hand side of $(BI.15)$ that depends on ${\mathbf{x}}_a$ is $1/2{\mathbf{m}}^{\mathrm{T}}{\mathbf{\Lambda }}_{bb}^{-1}\mathbf{m}$ in which $\mathbf{m}$ is given by $(BI.16)$. Combining this term with the remaining terms from $(BI.6)$ that depends on ${\mathbf{x}}_a$, we obtain
$$\begin{array}{rl}\frac{1}{2}[{\mathbf{\Lambda }}_{bb}& {{\mathbf{\mu }}_b-{\mathbf{\Lambda }}_{ba}\left({\mathbf{x}}_a-{\mathbf{\mu }}_a\right)]}^{\mathrm{T}}{\mathbf{\Lambda }}_{bb}^{-1}\left[{\mathbf{\Lambda }}_{bb}{\mathbf{\mu }}_b-{\mathbf{\Lambda }}_{ba}\left({\mathbf{x}}_a-{\mathbf{\mu }}_a\right)\right]\\ & -\frac{1}{2}{\mathbf{x}}_a^{\mathrm{T}}{\mathbf{\Lambda }}_{aa}{\mathbf{x}}_a+{\mathbf{x}}_a^{\mathrm{T}}\left({\mathbf{\Lambda }}_{aa}{\mathbf{\mu }}_a+{\mathbf{\Lambda }}_{ab}{\mathbf{\mu }}_b\right)+\text{ const }\\ =& -\frac{1}{2}{\mathbf{x}}_a^{\mathrm{T}}\left({\mathbf{\Lambda }}_{aa}-{\mathbf{\Lambda }}_{ab}{\mathbf{\Lambda }}_{bb}^{-1}{\mathbf{\Lambda }}_{ba}\right){\mathbf{x}}_a\\ & +{\mathbf{x}}_a^{\mathrm{T}}{\left({\mathbf{\Lambda }}_{aa}-{\mathbf{\Lambda }}_{ab}{\mathbf{\Lambda }}_{bb}^{-1}{\mathbf{\Lambda }}_{ba}\right)}^{-1}{\mathbf{\mu }}_a+\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{t}\end{array}\text{\hspace{1em}(BI.18)}$$
Again, by comparison with $(BI.7)$, we see that the covariance of the marginal distribution of $p({\mathbf{x}}_a)$ is given by
$${\mathbf{\Sigma }}_a={\left({\mathbf{\Lambda }}_{aa}-{\mathbf{\Lambda }}_{ab}{\mathbf{\Lambda }}_{bb}^{-1}{\mathbf{\Lambda }}_{ba}\right)}^{-1}\text{\hspace{1em}(BI.19)}$$
Similarly, the mean is given by
$${\mathbf{\Sigma }}_a\left({\mathbf{\Lambda }}_{aa}-{\mathbf{\Lambda }}_{ab}{\mathbf{\Lambda }}_{bb}^{-1}{\mathbf{\Lambda }}_{ba}\right){\mathbf{\mu }}_a\stackrel{BI.19}{=}{\mathbf{\mu }}_a\text{\hspace{1em}(BI.20)}$$
The covariance in $(BI.19)$ is expressed in terms of the partitioned precision matrix. We can rewrite this in terms of the corresponding partitioning of the covariance matrix as we did for the conditional distribution. Making use of $(BI.10),$ we then have
$${\left({\mathbf{\Lambda }}_{aa}-{\mathbf{\Lambda }}_{ab}{\mathbf{\Lambda }}_{bb}^{-1}{\mathbf{\Lambda }}_{ba}\right)}^{-1}={\mathbf{\Sigma }}_{aa}\text{\hspace{1em}(BI.21)}$$
Thus we obtain the intuitively satisfying result that the marginal distribution $p\left({\mathbf{x}}_a\right)$ has mean and covariance given by
$$\mathbb{E}\left[{\mathbf{x}}_a\right]={\mathbf{\mu }}_a\text{\hspace{1em}(BI.22)}$$
$$\mathrm{cov}\left[{\mathbf{x}}_a\right]={\mathbf{\Sigma }}_{aa}\text{\hspace{1em}(BI.23)}$$

We see that for a marginal distribution, the mean and covariance are most simply expressed in terms of the partitioned covariance matrix, in contrast to the conditional distribution for which the partitioned precision matrix gives rise to simpler expressions.

Partitioned Gaussians

We summarize the result for the marginal and conditional distributions of a partitioned Gaussian here.

Theorem: Partitioned Gaussians

Given a joint Gaussian distribution $\mathcal{N}(\mathbf{x}|\mathbf{\mu },\mathbf{\Sigma })$ with $\mathbf{\Lambda }\equiv {\mathbf{\Sigma }}^{-1}$ and
$$\mathbf{x}=\left(\begin{array}{c}{\mathbf{x}}_a\\ {\mathbf{x}}_b\end{array}\right),\mathbf{\mu }=\left(\begin{array}{c}{\mathbf{\mu }}_a\\ {\mathbf{\mu }}_b\end{array}\right)$$

$$\mathbf{\Sigma }=\left(\begin{array}{cc}{\mathbf{\Sigma }}_{aa}& {\mathbf{\Sigma }}_{ab}\\ {\mathbf{\Sigma }}_{ba}& {\mathbf{\Sigma }}_{bb}\end{array}\right),\mathbf{\Lambda }=\left(\begin{array}{ll}{\mathbf{\Lambda }}_{aa}& {\mathbf{\Lambda }}_{ab}\\ {\mathbf{\Lambda }}_{ba}& {\mathbf{\Lambda }}_{bb}\end{array}\right)$$

Conditional distribution:
$$\begin{gathered} p({\mathbf{x}}_a|{\mathbf{x}}_b)=\mathcal{N}(\mathbf{x}|{\mathbf{\mu }}_{a|b},{\mathbf{\Sigma }}_{a|b}) \\ {\mathbf{\mu }}_{a|b}={\mathbf{\mu }}_a-{\mathbf{\Lambda }}_{aa}^{-1}{\mathbf{\Lambda }}_{ab}\left({\mathbf{x}}_b-{\mathbf{\mu }}_b\right)={\mathbf{\mu }}_a+{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}({\mathbf{x}}_b-{\mathbf{\mu }}_b) \\ {\mathbf{\Sigma }}_{a|b}={\mathbf{\Lambda }}_{aa}^{-1}={\mathbf{\Sigma }}_{aa}-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba} \end{gathered}$$
Marginal distribution:
$$p({\mathbf{x}}_a)=\mathcal{N}({\mathbf{x}}_a|{\mathbf{\mu }}_a,{\mathbf{\Sigma }}_{aa})$$

Bayes’ theorem for Gaussian variables

In [Conditional Gaussian distributions](#Conditional Gaussian distributions) and [Marginal Gaussian distributions](#Marginal Gaussian distributions), we considered a Gaussian $p(\mathbf{x})$ in which we partitioned the vector $\mathbf{x}$ into two subvectors $\mathbf{x}=({\mathbf{x}}_a,{\mathbf{x}}_b)$ and then found expressions for the conditional distribution $p({\mathbf{x}}_a|{\mathbf{x}}_b)$ and the marginal distribution $p({\mathbf{x}}_a)$.

Here we shall suppose that we are given a Gaussian marginal distribution $p(\mathbf{x})$ and a Gaussian conditional distribution $p(\mathbf{y}|\mathbf{x})$ in which $p(\mathbf{y}|\mathbf{x})$ has a mean that is a linear function of $\mathbf{x}$, and a covariance which is independent of $\mathbf{x}$, i.e.,
$$p(\mathbf{x})=\mathcal{N}(\mathbf{x}|\mathbf{\mu },{\mathbf{\Lambda }}^{-1})\text{\hspace{1em}(BI.24)}$$

$$p(\mathbf{y}|\mathbf{x})=\mathcal{N}(\mathbf{y}|\mathbf{A}\mathbf{x}+\mathbf{b},{\mathbf{L}}^{-1})\text{\hspace{1em}(BI.25)}$$

where $\mathbf{\Lambda }$ and $\mathbf{L}$ are precision matrices.

We wish to find the marginal distribution $p(\mathbf{y})$ and the conditional distribution $p(\mathbf{x}|\mathbf{y})$. For that, we will first find the joint distribution for $\mathbf{x}$ and $\mathbf{y}$, and then use the conclusions in [Partitioned Gaussians](#Partitioned Gaussians) to obtain $p(\mathbf{y})$ and $p(\mathbf{x}|\mathbf{y})$.

Define
$$\mathbf{z}=\left(\begin{array}{c}\mathbf{x}\\ \mathbf{y}\end{array}\right)\text{\hspace{1em}(BI.26)}$$
and then consider the log of the joint distribution
$$\begin{array}{rl}\ln p(\mathbf{z})=& \ln p(\mathbf{x})+\ln p(\mathbf{y}\mid \mathbf{x})\\ =& -\frac{1}{2}(\mathbf{x}-\mathbf{\mu }{)}^{\mathrm{T}}\mathbf{\Lambda }(\mathbf{x}-\mathbf{\mu })\\ & -\frac{1}{2}(\mathbf{y}-\mathbf{A}\mathbf{x}-\mathbf{b}{)}^{\mathrm{T}}\mathbf{L}(\mathbf{y}-\mathbf{A}\mathbf{x}-\mathbf{b})+\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{t}\end{array}\text{\hspace{1em}(BI.27)}$$
As before, we see that this is a quadratic function of the components of $\mathbf{z},$ and hence $p(\mathbf{z})$ is Gaussian distribution. To find the precision of this Gaussian, we consider the second order terms in $(BI.27)$, which can be written as
$$\begin{array}{r}-\frac{1}{2}{\mathbf{x}}^{\mathrm{T}}\left(\mathbf{\Lambda }+{\mathbf{A}}^{\mathrm{T}}\mathbf{L}\mathbf{A}\right)\mathbf{x}-\frac{1}{2}{\mathbf{y}}^{\mathrm{T}}\mathbf{L}\mathbf{y}+\frac{1}{2}{\mathbf{y}}^{\mathrm{T}}\mathbf{L}\mathbf{A}\mathbf{x}+\frac{1}{2}{\mathbf{x}}^{\mathrm{T}}{\mathbf{A}}^{\mathrm{T}}\mathbf{L}\mathbf{y}\\ =-\frac{1}{2}{\left(\begin{array}{l}\mathbf{x}\\ \mathbf{y}\end{array}\right)}^{\mathrm{T}}\left(\begin{array}{cc}\mathbf{\Lambda }+{\mathbf{A}}^{\mathrm{T}}\mathbf{L}\mathbf{A}& -{\mathbf{A}}^{\mathrm{T}}\mathbf{L}\\ -\mathbf{L}\mathbf{A}& \mathbf{L}\end{array}\right)\left(\begin{array}{l}\mathbf{x}\\ \mathbf{y}\end{array}\right)=-\frac{1}{2}{\mathbf{z}}^{\mathrm{T}}\mathbf{R}\mathbf{z}\end{array}\text{\hspace{1em}(BI.28)}$$
and so the Gaussian distribution over $\mathbf{z}$ has precision (inverse covariance) matrix given by
$$\mathbf{R}=\left(\begin{array}{cc}\mathbf{\Lambda }+{\mathbf{A}}^{\mathrm{T}}\mathbf{L}\mathbf{A}& -{\mathbf{A}}^{\mathrm{T}}\mathbf{L}\\ -\mathbf{L}\mathbf{A}& \mathbf{L}\end{array}\right)\text{\hspace{1em}(BI.29)}$$
The covariance matrix is found by taking the inverse of the precision, which can be done using the matrix inversion formula $(BI.10)$ to give
$$\mathrm{cov}[\mathbf{z}]={\mathbf{R}}^{-1}=\left(\begin{array}{cc}{\mathbf{\Lambda }}^{-1}& {\mathbf{\Lambda }}^{-1}{\mathbf{A}}^{\mathrm{T}}\\ \mathbf{A}{\mathbf{\Lambda }}^{-1}& {\mathbf{L}}^{-1}+\mathbf{A}{\mathbf{\Lambda }}^{-1}{\mathbf{A}}^{\mathrm{T}}\end{array}\right)\text{\hspace{1em}(BI.30)}$$
Similarly, we can find the mean of the Gaussian distribution over $\mathbf{z}$ by identifying the linear terms in $(BI.27)$, which are given by
$${\mathbf{x}}^{\mathrm{T}}\mathbf{\Lambda }\mathbf{\mu }-{\mathbf{x}}^{\mathrm{T}}{\mathbf{A}}^{\mathrm{T}}\mathbf{L}\mathbf{b}+{\mathbf{y}}^{\mathrm{T}}\mathbf{L}\mathbf{b}={\left(\begin{array}{l}\mathbf{x}\\ \mathbf{y}\end{array}\right)}^{\mathrm{T}}\left(\begin{array}{c}\mathbf{\Lambda }\mathbf{\mu }-{\mathbf{A}}^{\mathrm{T}}\mathbf{L}\mathbf{b}\\ \mathbf{L}\mathbf{b}\end{array}\right)\text{\hspace{1em}(BI.31)}$$
Using our earlier result $(BI.7)$ obtained by completing the square over the quadratic form of a multivariate Gaussian, we find that the mean of $z$ is given by
$$\mathbb{E}[\mathbf{z}]={\mathbf{R}}^{-1}\left(\begin{array}{c}\mathbf{\Lambda }\mu -{\mathbf{A}}^{\mathrm{T}}\mathbf{L}\mathbf{b}\\ \mathbf{L}\mathbf{b}\end{array}\right)\stackrel{BI.29}{=}\left(\begin{array}{c}\mathbf{\mu }\\ \mathbf{A}\mu +\mathbf{b}\end{array}\right)\text{\hspace{1em}(BI.32)}$$
Now we can obtain the mean and covariance of the marginal distribution $p(\mathbf{y})$ using the conclusions in [Partitioned Gaussians](#Partitioned Gaussians):
$$\begin{array}{rl}\mathbb{E}[\mathbf{y}]& =\mathbf{A}\mathbf{\mu }+\mathbf{b}\end{array}\text{\hspace{1em}(BI.33)}$$
$$\begin{array}{rl}\mathrm{cov}[\mathbf{y}]& ={\mathbf{L}}^{-1}+\mathbf{A}{\mathbf{\Lambda }}^{-1}{\mathbf{A}}^{\mathrm{T}}\end{array}\text{\hspace{1em}(BI.34)}$$

A special case of this result is when $\mathbf{A}=\mathbf{I}$, in which case it reduces to the convolution of two Gaussians, for which we see that the mean of the convolution is the sum of the mean of the two Gaussians, and the covariance of the convolution is the sum of their covariances.

The conditional distribution $p(\mathbf{x}|\mathbf{y})$ has mean and covariance given by
$$\begin{array}{rl}\mathbb{E}[\mathbf{x}|\mathbf{y}]& ={\left(\mathbf{\Lambda }+{\mathbf{A}}^{\mathrm{T}}\mathbf{L}\mathbf{A}\right)}^{-1}\left\{{\mathbf{A}}^{\mathrm{T}}\mathbf{L}(\mathbf{y}-\mathbf{b})+\mathbf{\Lambda }\mathbf{\mu }\right\}\end{array}\text{\hspace{1em}(BI.35)}$$
$$\begin{array}{rl}\mathrm{cov}[\mathbf{x}|\mathbf{y}]& ={\left(\mathbf{\Lambda }+{\mathbf{A}}^{\mathrm{T}}\mathbf{L}\mathbf{A}\right)}^{-1}\end{array}\text{\hspace{1em}(BI.36)}$$

Having found the marginal and conditional distributions, we effectively expressed the joint distribution $p(\mathbf{z})=p(\mathbf{x})p(\mathbf{y}|\mathbf{x})$ in the form $p(\mathbf{x}|\mathbf{y})p(\mathbf{y}).$ These results are summarized below.

Theorem: Marginal and Conditional Gaussians

Given a marginal Gaussian distribution for $\mathbf{x}$ and a conditional Gaussian distribution for $\mathbf{y}$ given $\mathbf{x}$ in the form
$$\begin{array}{rl}p(\mathbf{x})& =\mathcal{N}\left(\mathbf{x}|\mathbf{\mu },{\mathbf{\Lambda }}^{-1}\right)\\ p(\mathbf{y}|\mathbf{x})& =\mathcal{N}\left(\mathbf{y}|\mathbf{A}\mathbf{x}+\mathbf{b},{\mathbf{L}}^{-1}\right)\end{array}$$
the marginal distribution of $\mathbf{y}$ and the conditional distribution of $\mathbf{x}$ given $\mathbf{y}$ are given by
$$\begin{array}{rl}p(\mathbf{y})& =\mathcal{N}\left(\mathbf{y}|\mathbf{A}\mathbf{\mu }+\mathbf{b},{\mathbf{L}}^{-1}+\mathbf{A}{\mathbf{\Lambda }}^{-1}{\mathbf{A}}^{\mathrm{T}}\right)\\ p(\mathbf{x}|\mathbf{y})& =\mathcal{N}\left(\mathbf{x}|\mathbf{\Sigma }\left\{{\mathbf{A}}^{\mathrm{T}}\mathbf{L}(\mathbf{y}-\mathbf{b})+\mathbf{\Lambda }\mathbf{\mu }\right\},\mathbf{\Sigma }\right)\end{array}$$
where
$$\mathbf{\Sigma }={\left(\mathbf{\Lambda }+{\mathbf{A}}^{\mathrm{T}}\mathbf{L}\mathbf{A}\right)}^{-1}$$

Bayesian Linear Regression (3.3.1-3.3.2)

Consider a data set of inputs X = { x 1 , . . . , x N } \mathbf X = \{\mathbf x_1,..., \mathbf x_N\} X={x1​,...,xN​} with corresponding target values $\mathbf{t}=[t_1,...,t_N{]}^T$. The expression for the likelihood function, which is a function of the adjustable parameters $\mathbf{w}$ and $\beta$, has the form
$$p(\mathbf{t}|\mathbf{X},\mathbf{w},\beta )=\prod _{n=1}^N\mathcal{N}({\mathbf{w}}^T\varphi ({\mathbf{x}}_n),{\beta }^{-1})=\mathcal{N}(\mathbf{\Phi }\mathbf{w},{\beta }^{-1}\mathbf{I})\text{\hspace{1em}(BI.37)}$$
where $\mathbf{\Phi }=[\varphi ({\mathbf{x}}_1)\cdots \varphi ({\mathbf{x}}_N){]}^T$.

We begin our discussion of the Bayesian treatment of linear regression by introducing a prior probability distribution over the model parameters $\mathbf{w}$. For the moment, we shall treat the noise precision parameter $\beta$ as a known constant.

Parameter distribution

To simplify the notation, we denote the likelihood function $p(\mathbf{t}|\mathbf{X},\mathbf{w},\beta )$ defined in $(BI.37)$ as $p(\mathbf{t}|\mathbf{w})$. Note that $p(\mathbf{t}|\mathbf{w})$ is the exponential of a quadratic function of $\mathbf{w}$, the corresponding conjugate prior is therefore given by a Gaussian distribution of the form
$$p(\mathbf{w})=\mathcal{N}(\mathbf{w}|{\mathbf{m}}_0,{\mathbf{S}}_0)\text{\hspace{1em}(BI.38)}$$
having mean ${\mathbf{m}}_0$ and covariance ${\mathbf{S}}_0$.

Next we compute the posterior distribution, which is proportional to the product of the likelihood function and the prior. Due to the choice of a conjugate Gaussian prior distribution, the posterior will also be Gaussian. By applying the conclusions in [Bayes’ theorem for Gaussian variables](#Bayes’ theorem for Gaussian variables), where $\mathbf{x}=\mathbf{w}$, $\mathbf{y}=\mathbf{t}$, $\mathbf{A}=\mathbf{\Phi }$,$\mathbf{b}=0$, $\mathbf{L}=\beta \mathbf{I}$, $\mathbf{\mu }={\mathbf{m}}_0$, and $\mathbf{\Lambda }={\mathbf{S}}_0^{-1}$, we have
$$p(\mathbf{w}|\mathbf{t})=\mathcal{N}(\mathbf{w}|{\mathbf{m}}_N.{\mathbf{S}}_N)\text{\hspace{1em}(BI.39)}$$

$${\mathbf{m}}_N={\mathbf{S}}_N({\mathbf{S}}_0^{-1}{\mathbf{m}}_0+\beta {\mathbf{\Phi }}^T\mathbf{t})\text{\hspace{1em}(BI.40)}$$

$${\mathbf{S}}_N^{-1}={\mathbf{S}}_0^{-1}+\beta {\mathbf{\Phi }}^T\mathbf{\Phi }\text{\hspace{1em}(BI.41)}$$

Note that because the posterior distribution is Gaussian, its mode coincides with its mean. Thus the maximum posterior weight vector is simply given by ${\mathbf{w}}_{\mathrm{M}\mathrm{A}\mathrm{P}}={\mathbf{m}}_N$.

We shall consider a particular form of Gaussian prior in order to simplify the treatment. Specifically, we consider a zero-mean isotropic Gaussian governed by a single precision parameter $\alpha$ so that
$$p(\mathbf{w}|\alpha )=\mathcal{N}(\mathbf{w}|0,{\alpha }^{-1}\mathbf{I})\text{\hspace{1em}(BI.42)}$$
and the corresponding posterior distribution over $\mathbf{w}$ is then given by $(BI.39)$ with
$${\mathbf{m}}_N=\beta {\mathbf{S}}_N{\mathbf{\Phi }}^T\mathbf{t}\text{\hspace{1em}(BI.43)}$$

$${\mathbf{S}}_N^{-1}=\alpha \mathbf{I}+\beta {\mathbf{\Phi }}^T\mathbf{\Phi }\text{\hspace{1em}(BI.44)}$$

Predictive distribution

In practice, we are not usually interested in the value of $\mathbf{w}$ itself but rather in making predictions of $t$ for new values of $\mathbf{x}$. This requires that we evaluate the predictive distribution defined by
$$p(t|\mathbf{x},\mathbf{t},\alpha ,\beta )=\int p(t|\mathbf{x},\mathbf{w},\beta )p(\mathbf{w}|\mathbf{x},\mathbf{t},\alpha ,\beta )d\mathbf{w}\text{\hspace{1em}(BI.45)}$$
where the conditional distribution $p(t|\mathbf{x},\mathbf{w},\beta )$ is given by
$$p(t|\mathbf{x},\mathbf{w},\beta )=\mathcal{N}(t|{\mathbf{w}}^T\varphi (\mathbf{x}),{\beta }^{-1})\text{\hspace{1em}(BI.46)}$$
and $p(\mathbf{w}|\mathbf{x},\mathbf{t},\alpha ,\beta )$ is given by $(BI.39),(BI.43),(BI.44)$.

Similarly, using the conclusions in [Bayes’ theorem for Gaussian variables](#Bayes’ theorem for Gaussian variables), viewing $p(\mathbf{w}|\mathbf{x},\mathbf{t},\alpha ,\beta )$ as the prior and $p(t|\mathbf{x},\mathbf{w},\beta )$ the likelihood, we obtain the marginal distribution $p(t|\mathbf{x},\mathbf{t},\alpha ,\beta )$ from $(BI.33)$ and $(BI.34)$
$$p(t|\mathbf{x},\mathbf{t},\alpha ,\beta )=\mathcal{N}\left(t|{\mathbf{m}}_N^T\varphi (\mathbf{x}),{\sigma }_N^2(\mathbf{x})\right)\text{\hspace{1em}(BI.47)}$$
where the variance ${\sigma }_N^2(\mathbf{x})$ of the predictive distribution is given by
$${\sigma }_N^2(\mathbf{x})=\frac{1}{\beta }+\varphi (\mathbf{x}{)}^{\mathrm{T}}{\mathbf{S}}_N\mathbf{\varphi }(\mathbf{x})\text{\hspace{1em}(BI.48)}$$
Note: To see this better, simplify the notation of $(BI.45)$ as
$$p(t|\mathbf{t})=\int p(t,\mathbf{w}|\mathbf{t})d\mathbf{w}=\int p(t|\mathbf{w},\mathbf{t})p(\mathbf{w}|\mathbf{t})d\mathbf{w}$$
The first term in $(BI.48)$ represents the noise on the data whereas the second term reflects the uncertainty associated with the parameters w. Because the noise process and the distribution of $\mathbf{w}$ are independent Gaussians, their variances are additive.

As an illustration of the predictive distribution for Bayesian linear regression models, let us discuss the synthetic sinusoidal data set example. In Figure 3.8, we fit a model comprising a linear combination of Gaussian basis functions to data sets of various sizes and then look at the corresponding posterior distributions.

Here the green curves correspond to the function $\sin (2\pi x)$ from which the data points were generated (with the addition of Gaussian noise). Data sets of size $N=1,N=2,N=4$, and $N=25$ are shown in the four plots by the blue circles. For each plot, the red curve shows the mean of the corresponding Gaussian predictive distribution, and the red shaded region spans one standard deviation either side of the mean. Note that the predictive uncertainty depends on $\mathbf{x}$ and is smallest in the neighborhood of the data points. Also note that the level of uncertainty decreases as more data points are observed.

The plots in Figure 3.8 only show the point-wise predictive variance as a function of $\mathbf{x}$. In order to gain insight into the covariance between the predictions at different values of $\mathbf{x}$, we can draw samples from the posterior distribution over $\mathbf{w}$, and then plot the corresponding functions $y(\mathbf{x},\mathbf{w})$, as shown in Figure 3.9.

Graphic Models (8.2.1-8.2.2)

Bayesian Networks

In order to motivate the use of directed graphs to describe probability distributions, consider first an arbitrary joint distribution $p(a,b,c)$ over three variables $a,b,$ and $c$. We can write the joint distribution in the form
$$p(a,b,c)=p(c|a,b)p(a,b)=p(c|a,b)p(b|a)p(a)\text{\hspace{1em}(GM.1)}$$
Note that at this stage, we do not need to specify anything further about these variables, such as whether they are discrete or continuous.

We now represent the right-hand side of $(GM.1)$ in terms of a simple graphical model as follows

Example:

The joint distribution corresponding to Fig. 8.2 is given by
$$p(x_1)p(x_2)p(x_3)p(x_4|x_1,x_2,x_3)p(x_5|x_1,x_3)p(x_6|x_4)p(x_7|x_4,x_5)$$
An important concept for probability distributions over multiple variables is that of conditional independence. Consider three variables $a,b,$ and $c$, and suppose that the conditional distribution of $a$,given $b$ and $c$, is such that it does not depend on the value of $b$, so that
$$p(a|b,c)=p(a|c)\text{\hspace{1em}(GM.2)}$$
We say that a is conditionally independent of b given c. This can be expressed in a slightly different way if we consider the joint distribution of a and b conditioned on c, which we can write in the form
$$p(a,b|c)=p(a|b,c)p(b|c)=p(a|c)p(b|c)\text{\hspace{1em}(GM.3)}$$
Thus we see that, conditioned on $c$, the joint distribution of $a$ and $b$ factorizes into the product of the marginal distribution of $a$ and the marginal distribution of $b$ (again both conditioned on $c$). This says that the variables $a$ and $b$ are statistically independent, given $c$.

We shall sometimes use a shorthand notation for conditional independence in which
$$a\perp b\mid c\text{\hspace{1em}(GM.4)}$$

Three Example Graphs

We begin our discussion of the conditional independence properties of directed graphs by considering three simple examples each involving graphs having just three nodes. Together, these will motivate and illustrate the key concepts of d-separation.

• tail-to-tail

The joint distribution corresponding to this graph is
$$p(a,b,c)=p(a|c)p(b|c)p(c)\text{\hspace{1em}(GM.5)}$$
If none of the variables are observed, then we can investigate whether $a$ and $b$ are independent by marginalizing both sides of $(GM.5)$ with respect to $c$ to give
$$p(a,b)=\sum _{c}p(a|c)p(b|c)p(c)\text{\hspace{1em}(GM.6)}$$
In general, this does not factorize into the product $p(a)p(b)$, and so
$$a/\perp b\mid \varnothing \text{\hspace{1em}(GM.7)}$$

Now suppose we condition on the variable $c$, as represented by the graph above. From $(GM.5)$, we can easily write down the conditional distribution of $a$ and $b$,given $c$, in the form
$$p(a,b|c)=\frac{p(a,b,c)}{p(c)}=p(a|c)p(b|c)\text{\hspace{1em}(GM.8)}$$
and so we obtain the conditional independence property
$$a\perp b\mid c\text{\hspace{1em}(GM.4)}$$

• Chain

The joint distribution corresponding to this graph is
$$p(a,b,c)=p(a)p(c|a)p(b|c)\text{\hspace{1em}(GM.9)}$$
First of all, suppose that none of the variables are observed. Again, we can test to see if $a$ and $b$ are independent by marginalizing over $c$ to give
$$p(a,b)=p(a)\sum _{c}p(c|a)p(b|c)=p(a)p(b|a)\text{\hspace{1em}(GM.10)}$$
which in general does not factorize into $p(a)p(b)$, and so
$$a/\perp b\mid \varnothing \text{\hspace{1em}(GM.7)}$$
as before.

Now suppose we condition on node $c$. Using Bayes’ theorem, together with $(GM.9),$ we obtain
$$\begin{array}{rl}p(a,b|c)& =\frac{p(a,b,c)}{p(c)}\\ & =\frac{p(a)p(c|a)p(b|c)}{p(c)}\\ & =p(a|c)p(b|c)\end{array}\text{\hspace{1em}(GM.11)}$$
and so again we obtain the conditional independence property
$$a\perp b\mid c\text{\hspace{1em}(GM.4)}$$

• Collider/head-to head

The joint distribution can again be written as
$$p(a,b,c)=p(a)p(b)p(c|a,b)\text{\hspace{1em}(GM.12)}$$
Consider first the case where none of the variables are observed. Marginalizing both sides of $(GM.12)$ over $c$ we obtain
$$p(a,b)=p(a)p(b)\text{\hspace{1em}(GM.13)}$$
and so $a$ and $b$ are independent with no variables observed, in contrast to the two previous examples. We can write this result as
$$a\perp b\mid \varnothing \text{\hspace{1em}(GM.14)}$$

Now suppose we condition on $c$. The conditional distribution of $a$ and $b$ is then given by
$$\begin{array}{rl}p(a,b|c)& =\frac{p(a,b,c)}{p(c)}\\ & =\frac{p(a)p(b)p(c|a,b)}{p(c)}\end{array}\text{\hspace{1em}(GM.15)}$$
which in general does not factorize into the product $p(a)p(b),$ and so
$$a/\perp b\mid c\text{\hspace{1em}(GM.16)}$$

D-separation

We now give a general statement of the d-separation property for directed graphs. Consider a general directed graph in which $A,B,$ and $C$ are arbitrary nonintersecting sets of nodes (whose union may be smaller than the complete set of nodes in the graph). We wish to ascertain whether a particular conditional independence statement $A\perp B\mid C$ is implied by a given directed acyclic graph.

To do so, we consider all possible paths from any node in $A$ to any node in $B$. Any such path is said to be blocked if it includes a node such that either

• the arrows on the path meet either head-to-tail or tail-to-tail at the node, and the node is in the set $C,$ or
• the arrows meet head-to-head at the node, and neither the node, nor any of its descendants, is in the set $C$.

If all paths are blocked, then $A$ is said to be $\mathrm{d}$-separated from $B$ by $C,$ and the joint distribution over all of the variables in the graph will satisfy $A\perp B\mid C$.

The concept of d-separation is illustrated in Figure $8.22.$ In graph (a), the path from $a$ to $b$ is not blocked by node $f$ because it is a tail-to-tail node for this path and is not observed, nor is it blocked by node $e$ because, although the latter is a head-to-head node, it has a descendant $c$ because is in the conditioning set. Thus the conditional independence statement $a\perp b\mid c$ does $not$ follow from this graph. In graph (b), the path from $a$ to $b$ is blocked by node $f$ because this is a tail-to-tail node that is observed, and so the conditional independence property $a\perp b\mid f$ will be satisfied by any distribution that factorizes according to this graph. Note that this path is also blocked by node e because $e$ is a head-to-head node and neither it nor its descendant are in the conditioning set.