locked
Unhandled Exception for Fixed-Covariance GMM RRS feed

  • Question

  • Hi,

    Using Infer.NET, I've tried and tested the Gaussian Mixture Model with distributions on mean vectors (centroids), mixture weights and covariance matrices and everything works fine (with datasets obtained from here). But for my research, as I have to deal with large datasets with very high dimensions (more than 10000 data points with each datapoint of dimension 3780), to reduce the number of free parameters, I would like to tie/share the covariance matrices and fix it to identity matrix (I'm guessing that this will approach to Bayesian-Kmeans).

    I've written the code for this and it does work for some datasets but for other datasets, it gives the following error (I've also attached my code below).

    Compiling model...done.
    Iterating:  
    ........Unhandled Exception: MicrosoftResearch.Infer.Maths.AllZeroException: The model has zero probability
       at MicrosoftResearch.Infer.Distributions.Discrete.Normalize() in C:\infernetBuilds\12-10-2011_14-16\Runtime\Distributions\Discrete.cs:line 541
       at MicrosoftResearch.Infer.Distributions.Discrete.SetToProduct(Discrete a, Discrete b) in C:\infernetBuilds\12-10-2011_14-16\Runtime\Distributions\Discrete.cs:line 366
       at MicrosoftResearch.Infer.Factors.VariableVmpOp.MarginalAverageLogarithm[T](T use, T def, T result) in C:\infernetBuilds\12-10-2011_14-16\Runtime\Factors\VariableOp.cs:line 285
       at MicrosoftResearch.Infer.Models.User.Model_VMP.Changed_numberOfIterationsDecreased_x_vIDistributionLtint__Gt0(Int32 numberOfIterations)
       at MicrosoftResearch.Infer.Models.User.Model_VMP.Execute(Int32 numberOfIterations, Boolean initialise)
       at MicrosoftResearch.Infer.Models.User.Model_VMP.Execute(Int32 numberOfIterations)
       at MicrosoftResearch.Infer.InferenceEngine.Execute(IGeneratedAlgorithm ca) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 519
       at MicrosoftResearch.Infer.InferenceEngine.InferAll(Boolean inferOnlySpecifiedVars, IVariable var) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 494
       at MicrosoftResearch.Infer.InferenceEngine.Infer[TReturn](IVariable var) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 290


    Code for Shared identity Covariance GMM

    using System;
    using System.Collections.Generic;
    using System.Text;
    using System.IO;
    using MicrosoftResearch.Infer.Models;
    using MicrosoftResearch.Infer;
    using MicrosoftResearch.Infer.Distributions;
    using MicrosoftResearch.Infer.Maths;
    using MicrosoftResearch.Infer.Utils;
    
    namespace GMM_CS
    {
        class gmm_cs
        {
            static void Main(string[] args)
            {
    
                ////////// Read in data /////////////////
                BinaryReader binReader = new BinaryReader(File.Open(@"dataset_clustering.bin", FileMode.Open));
                int ndata = binReader.ReadInt32();
                int dim = binReader.ReadInt32();
                int nclusters = binReader.ReadInt32();
                int numiters = binReader.ReadInt32();
                int burnin = binReader.ReadInt32();
                int thin = binReader.ReadInt32();
                double[,] dataset = new double[ndata, dim];
                double eta0, alpha_temp;
                double[] alpha = new double[nclusters];
                Vector m0 = Vector.Zero(dim);
                PositiveDefiniteMatrix B0 = PositiveDefiniteMatrix.Identity(dim);
    
                alpha_temp = binReader.ReadDouble();
    
                for (int i = 0; i < nclusters; i++)
                    alpha[i] = alpha_temp;
    
                eta0 = binReader.ReadDouble();
    
                for (int j = 0; j < dim; j++)
                    m0[j] = binReader.ReadDouble();
    
                for (int i = 0; i < dim; i++)
                    for (int j = 0; j < dim; j++)
                        B0[i, j] = binReader.ReadDouble();
    
                for (int j = 0; j < dim; j++)
                    for (int i = 0; i < ndata; i++)             
                        dataset[i, j] = binReader.ReadDouble(); 
                
                binReader.Close();              
    
                ///////// Specify Distributions //////
    
                // Define a range for the number of mixture components
                Range k = new Range(nclusters).Named("k");
      
                // Mixture component means
                VariableArray<Vector> means = Variable.Array<Vector>(k).Named("means");
                means[k] = Variable.VectorGaussianFromMeanAndPrecision(m0, B0).ForEach(k);
    
                // Identity covariance matrix for all components (observed)
                Variable<PositiveDefiniteMatrix> prec = PositiveDefiniteMatrix.Identity(dim);
                
                // Mixture weights 
                Variable<Vector> weights = Variable.Dirichlet(k, alpha).Named("weights");
    
                // Create a variable array which will hold the data
                Range n = new Range(ndata).Named("n");
                VariableArray<Vector> data = Variable.Array<Vector>(n).Named("x");
                // Create latent indicator variable for each data point
                VariableArray<int> z = Variable.Array<int>(n).Named("z");
    
                // The mixture of Gaussians model
                using (Variable.ForEach(n))
                {
                    z[n] = Variable.Discrete(weights);
                    using (Variable.Switch(z[n]))
                    {
                        data[n] = Variable.VectorGaussianFromMeanAndPrecision(means[z[n]], prec);
                    }
                }
    
                double[] cur_row = new double[dim];
                data.ObservedValue = new Vector[ndata];
    
                for (int i = 0; i < ndata; i++)
                {
                    for (int j = 0; j < dim; j++)
                    {
                        cur_row[j] = dataset[i, j];
                    }
    
                    data.ObservedValue[i] = Vector.FromArray(cur_row);
                }
    
                // Initialise messages randomly so as to break symmetry
                Discrete[] zinit = new Discrete[n.SizeAsInt];
                for (int i = 0; i < zinit.Length; i++)
                    zinit[i] = Discrete.PointMass(Rand.Int(k.SizeAsInt), k.SizeAsInt);
                z.InitialiseTo(Distribution<int>.Array(zinit));
    
                // The inference
    
                InferenceEngine ie = new InferenceEngine(new VariationalMessagePassing());
    
                //GibbsSampling gs = new GibbsSampling();
                //gs.BurnIn = burnin;
                //gs.Thin = thin;
                //InferenceEngine ie = new InferenceEngine(gs);
                //ie.NumberOfIterations = numiters;
       
    
                ////////////////////////////// Obtaining Output & Writing into binary file /////////////
                BinaryWriter binWriter = new BinaryWriter(File.Open("output_gmm.bin", FileMode.Create));
    
                //////////////// Mixing Weights Output ////////////////////
    
                Vector mweights = ie.Infer<Dirichlet>(weights).GetMean();
                double[] mix_weights = mweights.ToArray();
                for (int i = 0; i < mix_weights.Length; i++)
                    binWriter.Write(mix_weights[i]);
    
    
                ////////////// Centroids Output ////////////////////
    
                VectorGaussian[] vvg = ie.Infer<VectorGaussian[]>(means);
                Vector cur_centroid;
                for (int i = 0; i < nclusters; i++)
                {
                    cur_centroid = vvg[i].GetMean();
                    for (int j = 0; j < dim; j++)
                        binWriter.Write(cur_centroid[j]);
                }
    
                ////////////////// Precision Matrices Output ////////////////////
    
                //Wishart[] precs_pre = ie.Infer<Wishart[]>(precs);
                //PositiveDefiniteMatrix cur_precision;
                //for (int i = 0; i < nclusters; i++)
                //{
                //    cur_precision = precs_pre[i].GetMean();
                //    for (int ii = 0; ii < dim; ii++)
                //        for (int jj = 0; jj < dim; jj++)
                //              binWriter.Write(cur_precision[ii, jj]);
                //}
    
                binWriter.Close();
    
                Console.WriteLine("End of Inference.");
                //Console.ReadLine();
    
            }
    
            
     
    
          
        }
    }

    I'm not really sure what's causing the error.

    Thanks.







    Monday, March 12, 2012 10:09 AM

Answers

  • Ah, looks like you have found the bug.  DiscreteFromDirichletOp.SampleAverageLogarithm does not work with very small values of alpha.  We will have this fixed in the next version.  Meanwhile, you can work-around by using larger values of alpha.

    Tuesday, March 13, 2012 5:06 PM
    Owner

All replies

  • Setting both B0 and prec to the identity matrix is a pretty strong assumption about the range of the data.  If the data is far outside this range, it can lead to numerical problems.  Have you normalized the data to have zero mean and unit variance in each dimension?
    Tuesday, March 13, 2012 10:25 AM
    Owner
  • Hi Tom,

    Thanks for prompt reply. I haven't normalized data to zero mean and unit variance..

    Furthermore, for B0, I don't actually set it to identity matrix. My code says

    PositiveDefiniteMatrix B0 = PositiveDefiniteMatrix.Identity(dim);

    But, I change the values of B0 right after that with the following double loop:
    for (int i = 0; i < dim; i++)
    for (int j = 0; j < dim; j++)
       B0[i, j] = binReader.ReadDouble();

    I'm basically reading in (from the binary file) values of B0 which I calculated in Matlab. And I calculate B0 in the following way (partially taken from Kenichi Kurihara's Matlab code)

    %% assume that data is my data with rows = observations and columns = features
    xi0 = 0.01;
    eta_p = 1;
    covariance = cov(data);
    if ncols > 16
      [~, max_eig] = power_method(covariance);
    else
      max_eig = max(eig(covariance));
    end
    eta0 = eta_p * ncols + 1; % degrees of freedom
    B0 = eta0 * max_eig * eye(ncols) * xi0; % positive definite matrix (scale)
    B0 = inv(B0)'; % convert to precision and row major order


    Tuesday, March 13, 2012 11:31 AM
  • Since we can't reproduce the problem, your best option is to run the code under the Visual Studio debugger (see Debugging inference), inspect the variables at the point of the exception, and post those values.  According to the stack trace, Visual Studio should stop on the call to MarginalAverageLogarithm.  Knowing the first two arguments to that function will help diagnose the problem.



    Tuesday, March 13, 2012 1:47 PM
    Owner
  • The exception occurs at the following line:

    Vector mweights = ie.Infer<Dirichlet>(weights).GetMean();

    Below is the stacktrace:

    MicrosoftResearch.Infer.Maths.AllZeroException was unhandled
      Message="The model has zero probability"
      Source="Infer.Runtime"
      StackTrace:
           at MicrosoftResearch.Infer.Distributions.Discrete.Normalize() in C:\infernetBuilds\12-10-2011_14-16\Runtime\Distributions\Discrete.cs:line 543
           at MicrosoftResearch.Infer.Distributions.Discrete.SetProbs(Vector probs) in C:\infernetBuilds\12-10-2011_14-16\Runtime\Distributions\Discrete.cs:line 320
           at MicrosoftResearch.Infer.Factors.DiscreteFromDirichletOp.SampleAverageLogarithm(Dirichlet probs, Discrete result) in C:\infernetBuilds\12-10-2011_14-16\Runtime\Factors\DiscreteFromDirichlet.cs:line 258
           at MicrosoftResearch.Infer.Models.User.Model_VMP.Changed_numberOfIterationsDecreased_x(Int32 numberOfIterations)
           at MicrosoftResearch.Infer.Models.User.Model_VMP.Execute(Int32 numberOfIterations, Boolean initialise)
           at MicrosoftResearch.Infer.Models.User.Model_VMP.Execute(Int32 numberOfIterations)
           at MicrosoftResearch.Infer.InferenceEngine.Execute(IGeneratedAlgorithm ca) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 519
           at MicrosoftResearch.Infer.InferenceEngine.InferAll(Boolean inferOnlySpecifiedVars, IVariable var) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 494
           at MicrosoftResearch.Infer.InferenceEngine.Infer[TReturn](IVariable var) in C:\infernetBuilds\12-10-2011_14-16\Compiler\Infer\InferenceEngine.cs:line 290
           at GMM_CS.gmm_cs.Main(String[] args) in C:\Users\sckkh\Documents\KYAW\DOCS\TESTBIN\infer.net\a2\a1Sol\Program.cs:line 124
           at System.AppDomain._nExecuteAssembly(Assembly assembly, String[] args)
           at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
           at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
           at System.Threading.ThreadHelper.ThreadStart()
      InnerException:


      I'm not sure if I'm doing it right but when I try to step through the debugger (by placing a breakpoint at the "execute" method in the model source code file, it gives another error at:

      _hoist0 = DiscreteFromDirichletOp.SampleAverageLogarithm(this.weights_marginal_F, _hoist0);

      Here is the local value for _hoist0.

      +        _hoist0    {Discrete(0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)}    MicrosoftResearch.Infer.Distributions.Discrete

      And since this error occurs before coming to MarginalAverageLogarithm method, I'm not able look at the local variables for MarginalAverageLogarithm method.

     Also, I notice that regarding the hyperparmeter of the symmetric Dirichlet prior (alpha), if I set it to something like 0.01 (or greater), the error won't occur. If I set it a very small value like 0.0001, the error will occur.





                     
    Tuesday, March 13, 2012 4:54 PM
  • Ah, looks like you have found the bug.  DiscreteFromDirichletOp.SampleAverageLogarithm does not work with very small values of alpha.  We will have this fixed in the next version.  Meanwhile, you can work-around by using larger values of alpha.

    Tuesday, March 13, 2012 5:06 PM
    Owner
  • This bug is now fixed in version 2.5.
    Thursday, October 4, 2012 10:18 AM
    Owner