Answered by:
Unhandled Exception for Fixed-Covariance GMM

Question
-
Hi,
Using Infer.NET, I've tried and tested the Gaussian Mixture Model with distributions on mean vectors (centroids), mixture weights and covariance matrices and everything works fine (with datasets obtained from here). But for my research, as I have to deal with large datasets with very high dimensions (more than 10000 data points with each datapoint of dimension 3780), to reduce the number of free parameters, I would like to tie/share the covariance matrices and fix it to identity matrix (I'm guessing that this will approach to Bayesian-Kmeans).
I've written the code for this and it does work for some datasets but for other datasets, it gives the following error (I've also attached my code below).
Compiling model...done.
Iterating:
........Unhandled Exception: MicrosoftResearch.Infer.Maths.AllZeroException: The model has zero probability
at MicrosoftResearch.Infer.Distributions.Discrete.Normalize() in C:\infernetBuilds\12-10-2011_14-16\Runtime\Distributions\Discrete.cs:line 541
at MicrosoftResearch.Infer.Distributions.Discrete.SetToProduct(Discrete a, Discrete b) in C:\infernetBuilds\12-10-2011_14-16\Runtime\Distributions\Discrete.cs:line 366
at MicrosoftResearch.Infer.Factors.VariableVmpOp.MarginalAverageLogarithm[T](T use, T def, T result) in C:\infernetBuilds\12-10-2011_14-16\Runtime\Factors\VariableOp.cs:line 285
at MicrosoftResearch.Infer.Models.User.Model_VMP.Changed_numberOfIterationsDecreased_x_vIDistributionLtint__Gt0(Int32 numberOfIterations)
at MicrosoftResearch.Infer.Models.User.Model_VMP.Execute(Int32 numberOfIterations, Boolean initialise)
at MicrosoftResearch.Infer.Models.User.Model_VMP.Execute(Int32 numberOfIterations)
at MicrosoftResearch.Infer.InferenceEngine.Execute(IGeneratedAlgorithm ca) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 519
at MicrosoftResearch.Infer.InferenceEngine.InferAll(Boolean inferOnlySpecifiedVars, IVariable var) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 494
at MicrosoftResearch.Infer.InferenceEngine.Infer[TReturn](IVariable var) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 290
Code for Shared identity Covariance GMM
using System; using System.Collections.Generic; using System.Text; using System.IO; using MicrosoftResearch.Infer.Models; using MicrosoftResearch.Infer; using MicrosoftResearch.Infer.Distributions; using MicrosoftResearch.Infer.Maths; using MicrosoftResearch.Infer.Utils; namespace GMM_CS { class gmm_cs { static void Main(string[] args) { ////////// Read in data ///////////////// BinaryReader binReader = new BinaryReader(File.Open(@"dataset_clustering.bin", FileMode.Open)); int ndata = binReader.ReadInt32(); int dim = binReader.ReadInt32(); int nclusters = binReader.ReadInt32(); int numiters = binReader.ReadInt32(); int burnin = binReader.ReadInt32(); int thin = binReader.ReadInt32(); double[,] dataset = new double[ndata, dim]; double eta0, alpha_temp; double[] alpha = new double[nclusters]; Vector m0 = Vector.Zero(dim); PositiveDefiniteMatrix B0 = PositiveDefiniteMatrix.Identity(dim); alpha_temp = binReader.ReadDouble(); for (int i = 0; i < nclusters; i++) alpha[i] = alpha_temp; eta0 = binReader.ReadDouble(); for (int j = 0; j < dim; j++) m0[j] = binReader.ReadDouble(); for (int i = 0; i < dim; i++) for (int j = 0; j < dim; j++) B0[i, j] = binReader.ReadDouble(); for (int j = 0; j < dim; j++) for (int i = 0; i < ndata; i++) dataset[i, j] = binReader.ReadDouble(); binReader.Close(); ///////// Specify Distributions ////// // Define a range for the number of mixture components Range k = new Range(nclusters).Named("k"); // Mixture component means VariableArray<Vector> means = Variable.Array<Vector>(k).Named("means"); means[k] = Variable.VectorGaussianFromMeanAndPrecision(m0, B0).ForEach(k); // Identity covariance matrix for all components (observed) Variable<PositiveDefiniteMatrix> prec = PositiveDefiniteMatrix.Identity(dim); // Mixture weights Variable<Vector> weights = Variable.Dirichlet(k, alpha).Named("weights"); // Create a variable array which will hold the data Range n = new Range(ndata).Named("n"); VariableArray<Vector> data = Variable.Array<Vector>(n).Named("x"); // Create latent indicator variable for each data point VariableArray<int> z = Variable.Array<int>(n).Named("z"); // The mixture of Gaussians model using (Variable.ForEach(n)) { z[n] = Variable.Discrete(weights); using (Variable.Switch(z[n])) { data[n] = Variable.VectorGaussianFromMeanAndPrecision(means[z[n]], prec); } } double[] cur_row = new double[dim]; data.ObservedValue = new Vector[ndata]; for (int i = 0; i < ndata; i++) { for (int j = 0; j < dim; j++) { cur_row[j] = dataset[i, j]; } data.ObservedValue[i] = Vector.FromArray(cur_row); } // Initialise messages randomly so as to break symmetry Discrete[] zinit = new Discrete[n.SizeAsInt]; for (int i = 0; i < zinit.Length; i++) zinit[i] = Discrete.PointMass(Rand.Int(k.SizeAsInt), k.SizeAsInt); z.InitialiseTo(Distribution<int>.Array(zinit)); // The inference InferenceEngine ie = new InferenceEngine(new VariationalMessagePassing()); //GibbsSampling gs = new GibbsSampling(); //gs.BurnIn = burnin; //gs.Thin = thin; //InferenceEngine ie = new InferenceEngine(gs); //ie.NumberOfIterations = numiters; ////////////////////////////// Obtaining Output & Writing into binary file ///////////// BinaryWriter binWriter = new BinaryWriter(File.Open("output_gmm.bin", FileMode.Create)); //////////////// Mixing Weights Output //////////////////// Vector mweights = ie.Infer<Dirichlet>(weights).GetMean(); double[] mix_weights = mweights.ToArray(); for (int i = 0; i < mix_weights.Length; i++) binWriter.Write(mix_weights[i]); ////////////// Centroids Output //////////////////// VectorGaussian[] vvg = ie.Infer<VectorGaussian[]>(means); Vector cur_centroid; for (int i = 0; i < nclusters; i++) { cur_centroid = vvg[i].GetMean(); for (int j = 0; j < dim; j++) binWriter.Write(cur_centroid[j]); } ////////////////// Precision Matrices Output //////////////////// //Wishart[] precs_pre = ie.Infer<Wishart[]>(precs); //PositiveDefiniteMatrix cur_precision; //for (int i = 0; i < nclusters; i++) //{ // cur_precision = precs_pre[i].GetMean(); // for (int ii = 0; ii < dim; ii++) // for (int jj = 0; jj < dim; jj++) // binWriter.Write(cur_precision[ii, jj]); //} binWriter.Close(); Console.WriteLine("End of Inference."); //Console.ReadLine(); } } }
I'm not really sure what's causing the error.
Thanks.
- Edited by Kyaw Kyaw Htike Monday, March 12, 2012 10:15 AM
Monday, March 12, 2012 10:09 AM
Answers
-
Ah, looks like you have found the bug. DiscreteFromDirichletOp.SampleAverageLogarithm does not work with very small values of alpha. We will have this fixed in the next version. Meanwhile, you can work-around by using larger values of alpha.
- Edited by Tom MinkaMicrosoft employee, Owner Tuesday, March 13, 2012 5:12 PM
- Marked as answer by Kyaw Kyaw Htike Wednesday, March 14, 2012 10:56 PM
Tuesday, March 13, 2012 5:06 PMOwner
All replies
-
Setting both B0 and prec to the identity matrix is a pretty strong assumption about the range of the data. If the data is far outside this range, it can lead to numerical problems. Have you normalized the data to have zero mean and unit variance in each dimension?Tuesday, March 13, 2012 10:25 AMOwner
-
Hi Tom,
Thanks for prompt reply. I haven't normalized data to zero mean and unit variance..
Furthermore, for B0, I don't actually set it to identity matrix. My code says
PositiveDefiniteMatrix B0 = PositiveDefiniteMatrix.Identity(dim);
But, I change the values of B0 right after that with the following double loop:
for (int i = 0; i < dim; i++) for (int j = 0; j < dim; j++) B0[i, j] = binReader.ReadDouble();
I'm basically reading in (from the binary file) values of B0 which I calculated in Matlab. And I calculate B0 in the following way (partially taken from Kenichi Kurihara's Matlab code)
%% assume that data is my data with rows = observations and columns = features
xi0 = 0.01;
eta_p = 1;
covariance = cov(data);
if ncols > 16
[~, max_eig] = power_method(covariance);
else
max_eig = max(eig(covariance));
end
eta0 = eta_p * ncols + 1; % degrees of freedom
B0 = eta0 * max_eig * eye(ncols) * xi0; % positive definite matrix (scale)
B0 = inv(B0)'; % convert to precision and row major order
Tuesday, March 13, 2012 11:31 AM -
Since we can't reproduce the problem, your best option is to run the code under the Visual Studio debugger (see Debugging inference), inspect the variables at the point of the exception, and post those values. According to the stack trace, Visual Studio should stop on the call to MarginalAverageLogarithm. Knowing the first two arguments to that function will help diagnose the problem.
- Edited by Tom MinkaMicrosoft employee, Owner Tuesday, March 13, 2012 1:50 PM
Tuesday, March 13, 2012 1:47 PMOwner -
The exception occurs at the following line:
Vector mweights = ie.Infer<Dirichlet>(weights).GetMean();
Below is the stacktrace:
MicrosoftResearch.Infer.Maths.AllZeroException was unhandled
Message="The model has zero probability"
Source="Infer.Runtime"
StackTrace:
at MicrosoftResearch.Infer.Distributions.Discrete.Normalize() in C:\infernetBuilds\12-10-2011_14-16\Runtime\Distributions\Discrete.cs:line 543
at MicrosoftResearch.Infer.Distributions.Discrete.SetProbs(Vector probs) in C:\infernetBuilds\12-10-2011_14-16\Runtime\Distributions\Discrete.cs:line 320
at MicrosoftResearch.Infer.Factors.DiscreteFromDirichletOp.SampleAverageLogarithm(Dirichlet probs, Discrete result) in C:\infernetBuilds\12-10-2011_14-16\Runtime\Factors\DiscreteFromDirichlet.cs:line 258
at MicrosoftResearch.Infer.Models.User.Model_VMP.Changed_numberOfIterationsDecreased_x(Int32 numberOfIterations)
at MicrosoftResearch.Infer.Models.User.Model_VMP.Execute(Int32 numberOfIterations, Boolean initialise)
at MicrosoftResearch.Infer.Models.User.Model_VMP.Execute(Int32 numberOfIterations)
at MicrosoftResearch.Infer.InferenceEngine.Execute(IGeneratedAlgorithm ca) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 519
at MicrosoftResearch.Infer.InferenceEngine.InferAll(Boolean inferOnlySpecifiedVars, IVariable var) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 494
at MicrosoftResearch.Infer.InferenceEngine.Infer[TReturn](IVariable var) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 290
at GMM_CS.gmm_cs.Main(String[] args) in C:\Users\sckkh\Documents\KYAW\DOCS\TESTBIN\infer.net\a2\a1Sol\Program.cs:line 124
at System.AppDomain._nExecuteAssembly(Assembly assembly, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
InnerException:
I'm not sure if I'm doing it right but when I try to step through the debugger (by placing a breakpoint at the "execute" method in the model source code file, it gives another error at:
_hoist0 = DiscreteFromDirichletOp.SampleAverageLogarithm(this.weights_marginal_F, _hoist0);
Here is the local value for _hoist0.
+ _hoist0 {Discrete(0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)} MicrosoftResearch.Infer.Distributions.Discrete
And since this error occurs before coming to MarginalAverageLogarithm method, I'm not able look at the local variables for MarginalAverageLogarithm method.
Also, I notice that regarding the hyperparmeter of the symmetric Dirichlet prior (alpha), if I set it to something like 0.01 (or greater), the error won't occur. If I set it a very small value like 0.0001, the error will occur.
Tuesday, March 13, 2012 4:54 PM -
Ah, looks like you have found the bug. DiscreteFromDirichletOp.SampleAverageLogarithm does not work with very small values of alpha. We will have this fixed in the next version. Meanwhile, you can work-around by using larger values of alpha.
- Edited by Tom MinkaMicrosoft employee, Owner Tuesday, March 13, 2012 5:12 PM
- Marked as answer by Kyaw Kyaw Htike Wednesday, March 14, 2012 10:56 PM
Tuesday, March 13, 2012 5:06 PMOwner -
This bug is now fixed in version 2.5.Thursday, October 4, 2012 10:18 AMOwner