PCA without full factorization?

• Question

• I have a question about the PCA example:

http://research.microsoft.com/en-us/um/cambridge/projects/infernet/docs/Bayesian%20PCA%20and%20Factor%20Analysis.aspx

Are the approximate VB posteriors of Z and W fully factorized? That is, are all elements in Z (and W) mutually independent?

How to modify the example so that VMP would estimate the component-wise covariance matrices?

Wednesday, November 5, 2014 5:14 PM

All replies

• Yes, they are fully factorized.  If you don't want this, you need to rewrite the model to use inner products of Vectors (Variable<Vector> types and Variable.Inner), like so:
```            Range N = data.Range0.Named("N");
Range D = data.Range1.Named("D");
Range d = new Range(10).Named("d");

Variable<Gaussian> priorMu = Variable.New<Gaussian>().Named("PriorMu");
Variable<Gamma> priorPrec = Variable.New<Gamma>().Named("PriorPrec");
priorMu.ObservedValue = Gaussian.FromMeanAndPrecision(0.0, 1.0);
priorPrec.ObservedValue = Gamma.FromShapeAndScale(2.0, 1 / 2.0);

// Mixing matrix
VariableArray<Vector> W = Variable.Array<Vector>(D).Named("W");
var Alpha = Variable.WishartFromShapeAndScale(2, PositiveDefiniteMatrix.IdentityScaledBy(d.SizeAsInt, .5)).Named("Alpha");
W[D] = Variable.VectorGaussianFromMeanAndPrecision(Vector.Zero(d.SizeAsInt), Alpha).ForEach(D);

// Latent variables
var Z = Variable.Array<Vector>(N).Named("Z");
Z[N] = Variable.VectorGaussianFromMeanAndPrecision(Vector.Zero(d.SizeAsInt), PositiveDefiniteMatrix.Identity(d.SizeAsInt)).ForEach(N);

// Multiply
var ZtimesW = Variable.Array<double>(N, D).Named("ZTimesW");
ZtimesW[N, D] = Variable.InnerProduct(Z[N], W[D]);

// Bias
VariableArray<double> mu = Variable.Array<double>(D).Named("mu");
mu[D] = Variable.Random<double, Gaussian>(priorMu).ForEach(D);
VariableArray2D<double> ZWplusMu = Variable.Array<double>(N, D).Named("ZWplusMu");
ZWplusMu[N, D] = ZtimesW[N, D] + mu[D];

// Observation noise
VariableArray<double> pi = Variable.Array<double>(D).Named("pi");
pi[D] = Variable.Random<double, Gamma>(priorPrec).ForEach(D);

// The observation
data[N, D] = Variable.GaussianFromMeanAndPrecision(ZWplusMu[N, D], pi[D]);```

Also see this post.
Tuesday, November 11, 2014 2:29 PM
• Thanks a lot! How to initialize W randomly? ﻿randomGaussianArray returns invalid type for W and I couldn't find help from API..

I'm getting this error:

`The type `MicrosoftResearch.Infer.Distributions.IDistribution<double[,]>' cannot be used as type parameter `TDist' in the generic type or method `MicrosoftResearch.Infer.Models.Variable<MicrosoftResearch.Infer.Maths.Vector[]>.InitialiseTo<TDist>(TDist)'. There is no implicit reference conversion from `MicrosoftResearch.Infer.Distributions.IDistribution<double[,]>' to `MicrosoftResearch.Infer.Distributions.IDistribution<MicrosoftResearch.Infer.Maths.Vector[]>'`

Monday, November 17, 2014 1:05 PM
• ```        public static IDistribution<Vector[]> RandomGaussianVectorArray(int N, int C)
{
VectorGaussian[] array = new VectorGaussian[N];
for (int i = 0; i < N; i++)
{
Vector mean = Vector.Zero(C);
for (int j = 0; j < C; j++) mean[j] = Rand.Normal();
array[i] = new VectorGaussian(mean, PositiveDefiniteMatrix.Identity(C));
}
return Distribution<Vector>.Array(array);
}```

Monday, November 17, 2014 1:21 PM