locked
Infer.NET: improper distribution exceptions during EP inference (Migrated from community.research.microsoft.com) RRS feed

  • Question

  • Ferrum posted on 05-14-2009 9:19 AM

    Sometimes during EP inference I get improper distribution exceptions, with negative precisions (for the Gaussians), or negative shape/scale parameters for the Gammas. What would be a general way of handling such problems?

    Friday, June 3, 2011 4:54 PM

Answers

  • Ferrum replied on 06-08-2009 6:45 AM

    Thanks John,

    having posted the message, I have realized that using VMP to initialize EP should probably make little sense on its own. I will be very much looking forward to seeing Infer.NET extended with Power EP.

    Friday, June 3, 2011 4:55 PM

All replies

  • jwinn replied on 05-17-2009 5:52 PM

    It is an unfortunate fact that correctly applying Expectation Propagation to certain models will cause improper messages to be sent.  This normally arises when EP is trying to send a message which aims to decrease the certainty in the target variable.  See this talk for a detailed discussion of the issues: http://videolectures.net/abi07_winn_ipi/

    Often you can avoid improper messages by one of the following (note this is taken from the FAQ), depending on your model: (1) using a custom initialization (for an example of custom initialization, see the Mixture of Gaussians tutorial), (2) changing priors to be less confident or  (3) reordering the data - for example, when processing data in chunks trying not to select the chunks so that each would lead to a radically different posterior distribution e.g. by processing a class at a time.

    General solutions to this problem remain an open research question.

    Best,
    John W.

    • Proposed as answer by fl0m0 Thursday, July 5, 2012 10:22 AM
    Friday, June 3, 2011 4:54 PM
  • Ferrum replied on 05-23-2009 9:18 PM

    Nice presentation, thanks John! Do you think it could potentially be helpful to use VMP to initialize EP inference (when this is possible)? More generally, how much sense would it make to locally perturb the divergence measures of power EP, or have a scheduler for \alpha in D_{\alpha}?

    Friday, June 3, 2011 4:54 PM
  • jwinn replied on 05-30-2009 4:15 AM

    I have not tried initialised EP with VMP.  One issue with this is that VMP gives overconfident solutions i.e. the estimated variance is too low.  As this tends to be the circumstance where EP needs to send improper (confidence-decreasing) messages, it may be a poor initialisation.  Possibly you could try annealing a VMP solution.  Annealing the alpha is also an interesting idea - the good news on this front is that we are planning to introduce Power EP in a future release although I can't give any dates for this yet. As I say, this topic is a matter of open research - we would be very interested to hear the results if you try any of these ideas.

    Best,
    John W.

     

    Friday, June 3, 2011 4:54 PM
  • Ferrum replied on 06-08-2009 6:45 AM

    Thanks John,

    having posted the message, I have realized that using VMP to initialize EP should probably make little sense on its own. I will be very much looking forward to seeing Infer.NET extended with Power EP.

    Friday, June 3, 2011 4:55 PM