Let and be independent zero-mean real Gaussian vectors of respective length and with respective invertible covariance matrices and . Let be a full-rank matrix and define by
If we are given the vector we know that will be Gaussian with mean and covariance .
However suppose we are given and wish to find the conditional distribution of . Here we may think of as a hidden variable and as the observed variable. In this case the result is a little more involved. If we recover the results of this earlier blog post, is jointly Gaussian and so is Gaussian with mean
(Here is the cross-covariance of and , while .)
Using the fact that , , ,
and , (2) and (3) become
In this post we also derive the following alternative expressions (also described here) which the covariances appear as inverse matrices.
Note that in the scalar case with variances and (5) and (7) become the identity
In the case where is a scalar and is a diagonal matrix, is a weighted sum of the elements of vector and (8) becomes the inverse of a sum of inverses of variances (inverse-variance weighting).
One can check algebraically that the expressions (4),(6) and (5),(7) are equivalent in the matrix case, or we may proceed as follows.
Let be the covariance matrix of the joint vector . Then since we have and
Then as the Gaussian vector is zero-mean with covariance , the joint pdf of and is proportional to .
The key step now is to make use of the following identity (also see this explanation) based on completion of squares:
where and are matrices, defined similarly to and in the scalar equation
The second term in the right side of (11) is proportional to the pdf of (being Gaussian) , so the first term must be proportional to the conditional pdf . We are left to find and .
From (11), is the top-left block of while is the top-right block of . To find these blocks, consider the block matrix equation
This is the same as the system of equations
Multiplying (13) by gives
Subtracting this from (14) gives
Since the coefficients of and in (15) are the bottom-left and bottom-right blocks of respectively. We may then write and so from (15)
Also we have from (13)
Using (16) this becomes
Note that analogous to (16) we could have written
Comparing coefficients of , of (18) and (19) gives
In the same way that , , which for is simply as we saw before.
Hence from (20),
and from the -coefficient of (18),
Hence as desired, and equations (6)-(8) have been verified.