Why are Gaussian mixture model posteriors (incorrrectly?) Gaussian? RRS feed

  • Question

  • In the CyclingTime3 example in Infer.NET 101, a Gaussian mixture model is set up and some data is passed through it to infer the parameters.  The posteriors on the distributions on means come back as Gaussians.  However, mathematically, the posterior is not Gaussian.  It should converge to a bi-modal distribution with equal weight point masses at the means of the Gaussians that generate the data.

    There is some hope in that example that since the inference engine is called like this : InferenceEngine.Infer<Gaussian[]>(AverageTime), maybe the inference engine is being forced to shoe-horn the distribution into a Gaussian.  However, the Mixture of Gaussians example in the tutorials and examples section of the page has no such obvious type-cast, and yet seemingly returns a Gaussian posterior as well.  An experiment shows that if I remove this type-cast in the CyclingTime3 example, the result is still Gaussian.

    I understand that by initializing the message passing or by having different means on the prior distribution on means, you can get convergence to (seemingly) useful Gaussian posteriors.  However 1) even those posteriors are wrong.  No matter how asymmetric the priors are, the posteriors should eventually converge to a bi-modal distribution.  2) the much bigger problem in my mind : what if I want the bi-modal distribution?

    To expand on 2 : I now know that in the second-least complicated model you can create in Infer.NET, the posteriors do not converge properly.  I therefore feel like I have very little assurance that the posteriors generated in significantly more complicated models will bare any resemblance to the correct posteriors.

    Are there any flags I can flip to make the computations more exact?  Are there any general principles I can follow to know which models do converge properly or at least usefully?


    Tuesday, November 3, 2015 6:02 PM

All replies

  • This is only mentioned in the appendix of Infer.NET 101, but inference in Infer.NET is always approximate and does not have guarantees (except for special kinds of models).  Simple models like CyclingTime are a bit misleading since they make it look like it is easy to do exact calculations.  In more realistic models like a Mixture of Gaussians or Bayes Point Machine, exact inference is intractable so approximation is the only way forward.  You can sometimes improve the accuracy by changing inference algorithms as explained at Working with different inference algorithms, or by reformulating the model, but this is problem-specific.  If there is a specific model that you have in mind, you can post it on the forum for advice on how to do more accurate inference in that model.
    Wednesday, November 4, 2015 10:36 AM
  • Thanks.  I will post on the forum for model advice.

    Wednesday, November 4, 2015 3:55 PM