Unhandled Exception for Fixed-Covariance GMM

Montag, 12. März 2012 10:09

Hi,

Using Infer.NET, I've tried and tested the Gaussian Mixture Model with distributions on mean vectors (centroids), mixture weights and covariance matrices and everything works fine (with datasets obtained from here). But for my research, as I have to deal with large datasets with very high dimensions (more than 10000 data points with each datapoint of dimension 3780), to reduce the number of free parameters, I would like to tie/share the covariance matrices and fix it to identity matrix (I'm guessing that this will approach to Bayesian-Kmeans).

I've written the code for this and it does work for some datasets but for other datasets, it gives the following error (I've also attached my code below).

Compiling model...done.
Iterating:
........Unhandled Exception: MicrosoftResearch.Infer.Maths.AllZeroException: The model has zero probability
   at MicrosoftResearch.Infer.Distributions.Discrete.Normalize() in C:\infernetBuilds\12-10-2011_14-16\Runtime\Distributions\Discrete.cs:line 541
   at MicrosoftResearch.Infer.Distributions.Discrete.SetToProduct(Discrete a, Discrete b) in C:\infernetBuilds\12-10-2011_14-16\Runtime\Distributions\Discrete.cs:line 366
   at MicrosoftResearch.Infer.Factors.VariableVmpOp.MarginalAverageLogarithm[T](T use, T def, T result) in C:\infernetBuilds\12-10-2011_14-16\Runtime\Factors\VariableOp.cs:line 285
   at MicrosoftResearch.Infer.Models.User.Model_VMP.Changed_numberOfIterationsDecreased_x_vIDistributionLtint__Gt0(Int32 numberOfIterations)
   at MicrosoftResearch.Infer.Models.User.Model_VMP.Execute(Int32 numberOfIterations, Boolean initialise)
   at MicrosoftResearch.Infer.Models.User.Model_VMP.Execute(Int32 numberOfIterations)
   at MicrosoftResearch.Infer.InferenceEngine.Execute(IGeneratedAlgorithm ca) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 519
   at MicrosoftResearch.Infer.InferenceEngine.InferAll(Boolean inferOnlySpecifiedVars, IVariable var) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 494
   at MicrosoftResearch.Infer.InferenceEngine.Infer[TReturn](IVariable var) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 290

Code for Shared identity Covariance GMM

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using MicrosoftResearch.Infer.Models;
using MicrosoftResearch.Infer;
using MicrosoftResearch.Infer.Distributions;
using MicrosoftResearch.Infer.Maths;
using MicrosoftResearch.Infer.Utils;

namespace GMM_CS
{
    class gmm_cs
    {
        static void Main(string[] args)
        {

            ////////// Read in data /////////////////
            BinaryReader binReader = new BinaryReader(File.Open(@"dataset_clustering.bin", FileMode.Open));
            int ndata = binReader.ReadInt32();
            int dim = binReader.ReadInt32();
            int nclusters = binReader.ReadInt32();
            int numiters = binReader.ReadInt32();
            int burnin = binReader.ReadInt32();
            int thin = binReader.ReadInt32();
            double[,] dataset = new double[ndata, dim];
            double eta0, alpha_temp;
            double[] alpha = new double[nclusters];
            Vector m0 = Vector.Zero(dim);
            PositiveDefiniteMatrix B0 = PositiveDefiniteMatrix.Identity(dim);

            alpha_temp = binReader.ReadDouble();

            for (int i = 0; i < nclusters; i++)
                alpha[i] = alpha_temp;

            eta0 = binReader.ReadDouble();

            for (int j = 0; j < dim; j++)
                m0[j] = binReader.ReadDouble();

            for (int i = 0; i < dim; i++)
                for (int j = 0; j < dim; j++)
                    B0[i, j] = binReader.ReadDouble();

            for (int j = 0; j < dim; j++)
                for (int i = 0; i < ndata; i++)             
                    dataset[i, j] = binReader.ReadDouble(); 
            
            binReader.Close();              

            ///////// Specify Distributions //////

            // Define a range for the number of mixture components
            Range k = new Range(nclusters).Named("k");
  
            // Mixture component means
            VariableArray<Vector> means = Variable.Array<Vector>(k).Named("means");
            means[k] = Variable.VectorGaussianFromMeanAndPrecision(m0, B0).ForEach(k);

            // Identity covariance matrix for all components (observed)
            Variable<PositiveDefiniteMatrix> prec = PositiveDefiniteMatrix.Identity(dim);
            
            // Mixture weights 
            Variable<Vector> weights = Variable.Dirichlet(k, alpha).Named("weights");

            // Create a variable array which will hold the data
            Range n = new Range(ndata).Named("n");
            VariableArray<Vector> data = Variable.Array<Vector>(n).Named("x");
            // Create latent indicator variable for each data point
            VariableArray<int> z = Variable.Array<int>(n).Named("z");

            // The mixture of Gaussians model
            using (Variable.ForEach(n))
            {
                z[n] = Variable.Discrete(weights);
                using (Variable.Switch(z[n]))
                {
                    data[n] = Variable.VectorGaussianFromMeanAndPrecision(means[z[n]], prec);
                }
            }

            double[] cur_row = new double[dim];
            data.ObservedValue = new Vector[ndata];

            for (int i = 0; i < ndata; i++)
            {
                for (int j = 0; j < dim; j++)
                {
                    cur_row[j] = dataset[i, j];
                }

                data.ObservedValue[i] = Vector.FromArray(cur_row);
            }

            // Initialise messages randomly so as to break symmetry
            Discrete[] zinit = new Discrete[n.SizeAsInt];
            for (int i = 0; i < zinit.Length; i++)
                zinit[i] = Discrete.PointMass(Rand.Int(k.SizeAsInt), k.SizeAsInt);
            z.InitialiseTo(Distribution<int>.Array(zinit));

            // The inference

            InferenceEngine ie = new InferenceEngine(new VariationalMessagePassing());

            //GibbsSampling gs = new GibbsSampling();
            //gs.BurnIn = burnin;
            //gs.Thin = thin;
            //InferenceEngine ie = new InferenceEngine(gs);
            //ie.NumberOfIterations = numiters;
   

            ////////////////////////////// Obtaining Output & Writing into binary file /////////////
            BinaryWriter binWriter = new BinaryWriter(File.Open("output_gmm.bin", FileMode.Create));

            //////////////// Mixing Weights Output ////////////////////

            Vector mweights = ie.Infer<Dirichlet>(weights).GetMean();
            double[] mix_weights = mweights.ToArray();
            for (int i = 0; i < mix_weights.Length; i++)
                binWriter.Write(mix_weights[i]);


            ////////////// Centroids Output ////////////////////

            VectorGaussian[] vvg = ie.Infer<VectorGaussian[]>(means);
            Vector cur_centroid;
            for (int i = 0; i < nclusters; i++)
            {
                cur_centroid = vvg[i].GetMean();
                for (int j = 0; j < dim; j++)
                    binWriter.Write(cur_centroid[j]);
            }

            ////////////////// Precision Matrices Output ////////////////////

            //Wishart[] precs_pre = ie.Infer<Wishart[]>(precs);
            //PositiveDefiniteMatrix cur_precision;
            //for (int i = 0; i < nclusters; i++)
            //{
            //    cur_precision = precs_pre[i].GetMean();
            //    for (int ii = 0; ii < dim; ii++)
            //        for (int jj = 0; jj < dim; jj++)
            //              binWriter.Write(cur_precision[ii, jj]);
            //}

            binWriter.Close();

            Console.WriteLine("End of Inference.");
            //Console.ReadLine();

        }

        
 

      
    }
}

I'm not really sure what's causing the error.

Thanks.

Bearbeitet Kyaw Kyaw Htike Montag, 12. März 2012 10:15

Alle Antworten

Dienstag, 13. März 2012 10:25

0

Setting both B0 and prec to the identity matrix is a pretty strong assumption about the range of the data. If the data is far outside this range, it can lead to numerical problems. Have you normalized the data to have zero mean and unit variance in each dimension?
- Antworten
- Zitieren
Dienstag, 13. März 2012 11:31

0
Hi Tom,

Thanks for prompt reply. I haven't normalized data to zero mean and unit variance..

Furthermore, for B0, I don't actually set it to identity matrix. My code says
```
PositiveDefiniteMatrix B0 = PositiveDefiniteMatrix.Identity(dim);
```
But, I change the values of B0 right after that with the following double loop:
```
for (int i = 0; i < dim; i++)
for (int j = 0; j < dim; j++)
   B0[i, j] = binReader.ReadDouble();
```
I'm basically reading in (from the binary file) values of B0 which I calculated in Matlab. And I calculate B0 in the following way (partially taken from Kenichi Kurihara's Matlab code)

%% assume that data is my data with rows = observations and columns = features
xi0 = 0.01;
eta_p = 1;
covariance = cov(data);
if ncols > 16
[~, max_eig] = power_method(covariance);
else
max_eig = max(eig(covariance));
end
eta0 = eta_p * ncols + 1; % degrees of freedom
B0 = eta0 * max_eig * eye(ncols) * xi0; % positive definite matrix (scale)
B0 = inv(B0)'; % convert to precision and row major order
- Antworten
- Zitieren
Dienstag, 13. März 2012 13:47

0
Since we can't reproduce the problem, your best option is to run the code under the Visual Studio debugger (see Debugging inference ), inspect the variables at the point of the exception, and post those values. According to the stack trace, Visual Studio should stop on the call to MarginalAverageLogarithm. Knowing the first two arguments to that function will help diagnose the problem.
- Bearbeitet Tom MinkaMicrosoft Employee Dienstag, 13. März 2012 13:50
- Antworten
- Zitieren
Dienstag, 13. März 2012 16:54

0

The exception occurs at the following line:

Vector mweights = ie.Infer<Dirichlet>(weights).GetMean();

Below is the stacktrace:

MicrosoftResearch.Infer.Maths.AllZeroException was unhandled
Message="The model has zero probability"
Source="Infer.Runtime"
StackTrace:
       at MicrosoftResearch.Infer.Distributions.Discrete.Normalize() in C:\infernetBuilds\12-10-2011_14-16\Runtime\Distributions\Discrete.cs:line 543
       at MicrosoftResearch.Infer.Distributions.Discrete.SetProbs(Vector probs) in C:\infernetBuilds\12-10-2011_14-16\Runtime\Distributions\Discrete.cs:line 320
       at MicrosoftResearch.Infer.Factors.DiscreteFromDirichletOp.SampleAverageLogarithm(Dirichlet probs, Discrete result) in C:\infernetBuilds\12-10-2011_14-16\Runtime\Factors\DiscreteFromDirichlet.cs:line 258
       at MicrosoftResearch.Infer.Models.User.Model_VMP.Changed_numberOfIterationsDecreased_x(Int32 numberOfIterations)
       at MicrosoftResearch.Infer.Models.User.Model_VMP.Execute(Int32 numberOfIterations, Boolean initialise)
       at MicrosoftResearch.Infer.Models.User.Model_VMP.Execute(Int32 numberOfIterations)
       at MicrosoftResearch.Infer.InferenceEngine.Execute(IGeneratedAlgorithm ca) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 519
       at MicrosoftResearch.Infer.InferenceEngine.InferAll(Boolean inferOnlySpecifiedVars, IVariable var) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 494
       at MicrosoftResearch.Infer.InferenceEngine.Infer[TReturn](IVariable var) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 290
       at GMM_CS.gmm_cs.Main(String[] args) in C:\Users\sckkh\Documents\KYAW\DOCS\TESTBIN\infer.net\a2\a1Sol\Program.cs:line 124
       at System.AppDomain._nExecuteAssembly(Assembly assembly, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()
InnerException:

I'm not sure if I'm doing it right but when I try to step through the debugger (by placing a breakpoint at the "execute" method in the model source code file, it gives another error at:

_hoist0 = DiscreteFromDirichletOp.SampleAverageLogarithm(this.weights_marginal_F, _hoist0);

Here is the local value for _hoist0.

+       _hoist0   {Discrete(0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)}   MicrosoftResearch.Infer.Distributions.Discrete

And since this error occurs before coming to MarginalAverageLogarithm method, I'm not able look at the local variables for MarginalAverageLogarithm method.

Also, I notice that regarding the hyperparmeter of the symmetric Dirichlet prior (alpha), if I set it to something like 0.01 (or greater), the error won't occur. If I set it a very small value like 0.0001, the error will occur.
- Antworten
- Zitieren
Dienstag, 13. März 2012 17:06

0
Ah, looks like you have found the bug. DiscreteFromDirichletOp.SampleAverageLogarithm does not work with very small values of alpha. We will have this fixed in the next version. Meanwhile, you can work-around by using larger values of alpha.
- Bearbeitet Tom MinkaMicrosoft Employee Dienstag, 13. März 2012 17:12
- Als Antwort markiert Kyaw Kyaw Htike Mittwoch, 14. März 2012 22:56
- Antworten
- Zitieren
Donnerstag, 4. Oktober 2012 10:18

0

This bug is now fixed in version 2.5.
- Antworten
- Zitieren