Asked by:
PCA without full factorization?

Question
-
I have a question about the PCA example:
http://research.microsoft.com/en-us/um/cambridge/projects/infernet/docs/Bayesian%20PCA%20and%20Factor%20Analysis.aspx
Are the approximate VB posteriors of Z and W fully factorized? That is, are all elements in Z (and W) mutually independent?
How to modify the example so that VMP would estimate the component-wise covariance matrices?
Wednesday, November 5, 2014 5:14 PM
All replies
-
Yes, they are fully factorized. If you don't want this, you need to rewrite the model to use inner products of Vectors (Variable<Vector> types and Variable.Inner), like so:
Range N = data.Range0.Named("N"); Range D = data.Range1.Named("D"); Range d = new Range(10).Named("d"); Variable<Gaussian> priorMu = Variable.New<Gaussian>().Named("PriorMu"); Variable<Gamma> priorPrec = Variable.New<Gamma>().Named("PriorPrec"); priorMu.ObservedValue = Gaussian.FromMeanAndPrecision(0.0, 1.0); priorPrec.ObservedValue = Gamma.FromShapeAndScale(2.0, 1 / 2.0); // Mixing matrix VariableArray<Vector> W = Variable.Array<Vector>(D).Named("W"); var Alpha = Variable.WishartFromShapeAndScale(2, PositiveDefiniteMatrix.IdentityScaledBy(d.SizeAsInt, .5)).Named("Alpha"); W[D] = Variable.VectorGaussianFromMeanAndPrecision(Vector.Zero(d.SizeAsInt), Alpha).ForEach(D); // Latent variables var Z = Variable.Array<Vector>(N).Named("Z"); Z[N] = Variable.VectorGaussianFromMeanAndPrecision(Vector.Zero(d.SizeAsInt), PositiveDefiniteMatrix.Identity(d.SizeAsInt)).ForEach(N); // Multiply var ZtimesW = Variable.Array<double>(N, D).Named("ZTimesW"); ZtimesW[N, D] = Variable.InnerProduct(Z[N], W[D]); // Bias VariableArray<double> mu = Variable.Array<double>(D).Named("mu"); mu[D] = Variable.Random<double, Gaussian>(priorMu).ForEach(D); VariableArray2D<double> ZWplusMu = Variable.Array<double>(N, D).Named("ZWplusMu"); ZWplusMu[N, D] = ZtimesW[N, D] + mu[D]; // Observation noise VariableArray<double> pi = Variable.Array<double>(D).Named("pi"); pi[D] = Variable.Random<double, Gamma>(priorPrec).ForEach(D); // The observation data[N, D] = Variable.GaussianFromMeanAndPrecision(ZWplusMu[N, D], pi[D]);
Also see this post.- Edited by Tom MinkaMicrosoft employee, Owner Tuesday, November 11, 2014 2:30 PM
Tuesday, November 11, 2014 2:29 PMOwner -
Thanks a lot! How to initialize W randomly? randomGaussianArray returns invalid type for W and I couldn't find help from API..
I'm getting this error:
The type `MicrosoftResearch.Infer.Distributions.IDistribution<double[,]>' cannot be used as type parameter `TDist' in the generic type or method `MicrosoftResearch.Infer.Models.Variable<MicrosoftResearch.Infer.Maths.Vector[]>.InitialiseTo<TDist>(TDist)'. There is no implicit reference conversion from `MicrosoftResearch.Infer.Distributions.IDistribution<double[,]>' to `MicrosoftResearch.Infer.Distributions.IDistribution<MicrosoftResearch.Infer.Maths.Vector[]>'
- Edited by Jaakko Luttinen Monday, November 17, 2014 1:12 PM
Monday, November 17, 2014 1:05 PM -
public static IDistribution<Vector[]> RandomGaussianVectorArray(int N, int C) { VectorGaussian[] array = new VectorGaussian[N]; for (int i = 0; i < N; i++) { Vector mean = Vector.Zero(C); for (int j = 0; j < C; j++) mean[j] = Rand.Normal(); array[i] = new VectorGaussian(mean, PositiveDefiniteMatrix.Identity(C)); } return Distribution<Vector>.Array(array); }
- Edited by Tom MinkaMicrosoft employee, Owner Monday, November 17, 2014 1:21 PM
Monday, November 17, 2014 1:21 PMOwner