Confused about Constraints<p>Hi!</p>
<p>I've been using Infer.NET quite a bit, but Constraints still puzzle me a bit from the mathematic standpoint. As a concrete example, I was trying to learn a model where the likelihood is defined by a softmax, whose parameters are specified by a product of
two random variables. One random variable has a Gaussian Prior, and I had a requirement that the other random variable was strictly non-negative. The obvious choice was a Gamma prior, but a GaussianGamma product does not work with VMP (neither did Truncated
Gaussian for one of the variables). But a simple ConstrainPositive on one of the random variables does the trick. I am having trouble wrapping my head around what this constraint does during inference from a mathematic standpoint. My understanding is that
this is a constraint directly on the posterior for the variable that I want to be positive, rather than on the prior. In general, I don't quite understand how constraints can be made to work (enforced) in traditional bayesian inference. </p>
<p>Another example that confuses me is the following: In the same example as above, I have several normally distributed random variables, on which I'd like like to impose a partial order, i.e. one of the random variables has to be strictly greater than all
the others. My understanding is that ConstrainTrue(A>B) should do the trick, but this also leads to issues with VMP (but I need VMP for softmax). My solution is to introduce additional boolean random variables via the BernoulliFromLogOdds factor on the
difference between normally-distributed RVs and then give an observed value to each of those boolean variables. I can control the degree to which this "constraint" is satisfied by ramping up the coefficient in the BernoulliLogOdds factor. My question
is still mostly mathematical: I understand how observations in a generative model enforce constraints "softly" -- but what does a constraint like ConstrainTrue(A>B) really mean in BayesianInference, is this a hard constraint on the posterior?</p>
<p></p>
<p>Any pointers would be greatly appreciated! Thank you! </p>© 2009 Microsoft Corporation. All rights reserved.Wed, 14 Oct 2015 12:26:58 Z94117a1e-722b-4884-9527-f85c22f6662e- https://social.microsoft.com/Forums/en-US/94117a1e-722b-4884-9527-f85c22f6662e/confused-about-constraints?forum=infer.net#94117a1e-722b-4884-9527-f85c22f6662ehttps://social.microsoft.com/Forums/en-US/94117a1e-722b-4884-9527-f85c22f6662e/confused-about-constraints?forum=infer.net#94117a1e-722b-4884-9527-f85c22f6662eIgorNeletovhttps://social.microsoft.com:443/profile/igorneletov/?type=forumConfused about Constraints<p>Hi!</p>
<p>I've been using Infer.NET quite a bit, but Constraints still puzzle me a bit from the mathematic standpoint. As a concrete example, I was trying to learn a model where the likelihood is defined by a softmax, whose parameters are specified by a product of
two random variables. One random variable has a Gaussian Prior, and I had a requirement that the other random variable was strictly non-negative. The obvious choice was a Gamma prior, but a GaussianGamma product does not work with VMP (neither did Truncated
Gaussian for one of the variables). But a simple ConstrainPositive on one of the random variables does the trick. I am having trouble wrapping my head around what this constraint does during inference from a mathematic standpoint. My understanding is that
this is a constraint directly on the posterior for the variable that I want to be positive, rather than on the prior. In general, I don't quite understand how constraints can be made to work (enforced) in traditional bayesian inference. </p>
<p>Another example that confuses me is the following: In the same example as above, I have several normally distributed random variables, on which I'd like like to impose a partial order, i.e. one of the random variables has to be strictly greater than all
the others. My understanding is that ConstrainTrue(A>B) should do the trick, but this also leads to issues with VMP (but I need VMP for softmax). My solution is to introduce additional boolean random variables via the BernoulliFromLogOdds factor on the
difference between normally-distributed RVs and then give an observed value to each of those boolean variables. I can control the degree to which this "constraint" is satisfied by ramping up the coefficient in the BernoulliLogOdds factor. My question
is still mostly mathematical: I understand how observations in a generative model enforce constraints "softly" -- but what does a constraint like ConstrainTrue(A>B) really mean in BayesianInference, is this a hard constraint on the posterior?</p>
<p></p>
<p>Any pointers would be greatly appreciated! Thank you! </p>Wed, 14 Oct 2015 09:55:33 Z2015-10-14T09:55:33Z
- https://social.microsoft.com/Forums/en-US/94117a1e-722b-4884-9527-f85c22f6662e/confused-about-constraints?forum=infer.net#9d461db3-d1e3-4e8c-9074-66b6182014f9https://social.microsoft.com/Forums/en-US/94117a1e-722b-4884-9527-f85c22f6662e/confused-about-constraints?forum=infer.net#9d461db3-d1e3-4e8c-9074-66b6182014f9Tom Minkahttps://social.microsoft.com:443/profile/tom%20minka/?type=forumConfused about ConstraintsA constraint is not applied to the prior or to the posterior, it is applied to the variable. A useful way to think about constraints is in terms of rejection sampling. Consider sampling all variables from their priors, then rejecting
samples that violate any constraint or observation. The distribution of the kept samples is the posterior.Wed, 14 Oct 2015 12:26:58 Z2015-10-14T12:26:58Z
- https://social.microsoft.com/Forums/en-US/94117a1e-722b-4884-9527-f85c22f6662e/confused-about-constraints?forum=infer.net#5d47e24c-2ee4-457a-bb1e-80f361162a5chttps://social.microsoft.com/Forums/en-US/94117a1e-722b-4884-9527-f85c22f6662e/confused-about-constraints?forum=infer.net#5d47e24c-2ee4-457a-bb1e-80f361162a5cYordan Zaykovhttps://social.microsoft.com:443/profile/yordan%20zaykov/?type=forumConfused about Constraints<p>Let me just add to what Tom has said. I personally find ConstrainEqualRandom very different from the other constraints, because it gives rise to a separate set of challenges, like obtaining the marginal divided by the prior and working with it. An example
of this is given in the post with the pictures <a href="https://social.microsoft.com/Forums/en-US/1b70c631-c2b3-4740-8641-f07444cf0499/iterative-observation-of-data-arrays-and-problem-with-jagged-array-type-specification-infernet?forum=infer.net" target="_blank">
<span style="text-decoration:underline">here</span></a>. Other than that, how to attach constraints to variables is explained
<a href="http://research.microsoft.com/en-us/um/cambridge/projects/infernet/docs/Attaching%20constraints%20to%20variables.aspx" target="_blank">
<span style="text-decoration:underline">here</span></a>, and a list of all constraints is given
<a href="http://research.microsoft.com/en-us/um/cambridge/projects/infernet/docs/Constraints.aspx" target="_blank">
<span style="text-decoration:underline">here</span></a>.</p>
<p>-Y-</p>Wed, 14 Oct 2015 14:38:45 Z2015-10-14T14:38:45Z