Using Variable.ConstrainPositive on a Gaussian to approximate a Gamma random variable<p>Hi guys,</p>
<p>In <a href="https://social.microsoft.com/Forums/en-US/859caf78-ee2f-4711-93f7-e47b5aad4db1/how-to-use-nonconjugate-prior-distribution-in-the-modeling?forum=infer.net">
this thread</a> Tom mentioned using a Gaussian distribution together with Variable.ConstrainPositive as a replacement for a Gamma distribution for data that was positively real-valued.</p>
<p>Would it be possible to do something like this:</p>
<pre class="prettyprint">Variable<double> x = Variable.GaussianFromMeanAndVariance(0, 1).Named("x");
Variable.ConstrainPositive(x);
InferenceEngine engine = new InferenceEngine();
Console.WriteLine("Dist over positive x =" + engine.Infer<TruncatedGaussian>(x));</pre>
Running this code right now produces the following error:<br/>
<p> <img alt="" src="https://social.microsoft.com/Forums/getfile/587447"></p>
<p>What's the right way to use Variable.ConstrainPositive (as described above), and is it ever possible to infer a TruncatedGaussian from a constrained Gaussian? Or should we just assume that the inferred Gaussian can be approximated by a truncated Gaussian
with a lower bound of 0, and moment-matched mean and variance?</p>
<p>In using a constrained Gaussian instead of a Gamma for modeling purposes, where would we insert the Variable.ConstrainPositive - when the priors are being trained, or when trying to making predictions? Do the priors represent the means/precisions of constrained
gaussian distributions, or unconstrained distributions that will then be constrained during inference?</p>
<p>Thanks,</p>
<p>Andrew</p>© 2009 Microsoft Corporation. Alle rettigheder forbeholdes.Wed, 14 Jan 2015 21:17:01 Zdc056cb4-b8cb-4c13-9895-f68eba83b49b- https://social.microsoft.com/Forums/da-DK/dc056cb4-b8cb-4c13-9895-f68eba83b49b/using-variableconstrainpositive-on-a-gaussian-to-approximate-a-gamma-random-variable?forum=infer.net#dc056cb4-b8cb-4c13-9895-f68eba83b49bhttps://social.microsoft.com/Forums/da-DK/dc056cb4-b8cb-4c13-9895-f68eba83b49b/using-variableconstrainpositive-on-a-gaussian-to-approximate-a-gamma-random-variable?forum=infer.net#dc056cb4-b8cb-4c13-9895-f68eba83b49bAndrew Maohttps://social.microsoft.com:443/profile/andrew%20%20mao/?type=forumUsing Variable.ConstrainPositive on a Gaussian to approximate a Gamma random variable<p>Hi guys,</p>
<p>In <a href="https://social.microsoft.com/Forums/en-US/859caf78-ee2f-4711-93f7-e47b5aad4db1/how-to-use-nonconjugate-prior-distribution-in-the-modeling?forum=infer.net">
this thread</a> Tom mentioned using a Gaussian distribution together with Variable.ConstrainPositive as a replacement for a Gamma distribution for data that was positively real-valued.</p>
<p>Would it be possible to do something like this:</p>
<pre class="prettyprint">Variable<double> x = Variable.GaussianFromMeanAndVariance(0, 1).Named("x");
Variable.ConstrainPositive(x);
InferenceEngine engine = new InferenceEngine();
Console.WriteLine("Dist over positive x =" + engine.Infer<TruncatedGaussian>(x));</pre>
Running this code right now produces the following error:<br/>
<p> <img alt="" src="https://social.microsoft.com/Forums/getfile/587447"></p>
<p>What's the right way to use Variable.ConstrainPositive (as described above), and is it ever possible to infer a TruncatedGaussian from a constrained Gaussian? Or should we just assume that the inferred Gaussian can be approximated by a truncated Gaussian
with a lower bound of 0, and moment-matched mean and variance?</p>
<p>In using a constrained Gaussian instead of a Gamma for modeling purposes, where would we insert the Variable.ConstrainPositive - when the priors are being trained, or when trying to making predictions? Do the priors represent the means/precisions of constrained
gaussian distributions, or unconstrained distributions that will then be constrained during inference?</p>
<p>Thanks,</p>
<p>Andrew</p>Mon, 12 Jan 2015 05:38:28 Z2015-01-12T05:38:28Z
- https://social.microsoft.com/Forums/da-DK/dc056cb4-b8cb-4c13-9895-f68eba83b49b/using-variableconstrainpositive-on-a-gaussian-to-approximate-a-gamma-random-variable?forum=infer.net#17b62ecb-ffc4-4b36-98fd-802f3c743298https://social.microsoft.com/Forums/da-DK/dc056cb4-b8cb-4c13-9895-f68eba83b49b/using-variableconstrainpositive-on-a-gaussian-to-approximate-a-gamma-random-variable?forum=infer.net#17b62ecb-ffc4-4b36-98fd-802f3c743298Tom Minkahttps://social.microsoft.com:443/profile/tom%20minka/?type=forumUsing Variable.ConstrainPositive on a Gaussian to approximate a Gamma random variable<p>In this simple example, to infer a TruncatedGaussian you can either change the marginal prototype of x to be TruncatedGaussian:</p>
<pre class="prettyprint">x.AddAttribute(new MarginalPrototype(TruncatedGaussian.Uniform()));</pre>
<p>or you can change x to be drawn from a TruncatedGaussian (thus setting its marginal prototype implicitly):</p>
<pre class="prettyprint">var x = Variable.TruncatedGaussian(0, 1, double.NegativeInfinity, double.PositiveInfinity).Named("x");</pre>
<p>The approach that you suggest of converting the inferred Gaussian into a TruncatedGaussian would work, but Infer.NET doesn't provide any routines to do this.</p>
<p>The ConstrainPositive constraint should be applied only once, during training.</p>Mon, 12 Jan 2015 14:12:55 Z2015-01-12T14:12:55Z
- https://social.microsoft.com/Forums/da-DK/dc056cb4-b8cb-4c13-9895-f68eba83b49b/using-variableconstrainpositive-on-a-gaussian-to-approximate-a-gamma-random-variable?forum=infer.net#c58c81dd-4d4f-4112-be49-7a7136d798b7https://social.microsoft.com/Forums/da-DK/dc056cb4-b8cb-4c13-9895-f68eba83b49b/using-variableconstrainpositive-on-a-gaussian-to-approximate-a-gamma-random-variable?forum=infer.net#c58c81dd-4d4f-4112-be49-7a7136d798b7Andrew Maohttps://social.microsoft.com:443/profile/andrew%20%20mao/?type=forumUsing Variable.ConstrainPositive on a Gaussian to approximate a Gamma random variable<p>Hi Tom,</p>
<p>Thanks for the clarification. I'm still slightly confused about how to use ConstrainPositive in the way you described. I'll try and explain:</p>
<p>Suppose we are learning a distribution over Gaussians in the canonical way, with a Gaussian prior on the mean and a Gamma prior on the precision. In the model we apply Variable.ConstrainPositive on the Gaussian variable, set an array of observed data to
non-negative values, and infer a new Gaussian mean and Gamma precision posterior.</p>
<ul>
<li>Does the Variable.ConstrainPositive do anything during training, given that all of our data is non-negative anyway?</li><li>What do the posteriors that we learned correspond to? Are they the mean and precision of a Gaussian that would correspond to the right distribution after being truncated at 0, or are they the mean and precision of a Gaussian that approximates a truncated
Gaussian? (I know it wouldn't exactly be either, due to approximation, but is one closer than the other?)</li></ul>
<p>I'm guessing that your last comment above suggests that the former is true, not the latter, and that the non-truncated Gaussian inferred from the posterior would have the same mean as the truncated Gaussian that we're trying to learn.
</p>
<p>What would it mean to apply ConstrainPositive only during training? Does that mean we'll need to create a separate model for prediction that doesn't include the Variable.ConstrainPositive, and use the Gaussian and Gamma posterior distribution that we learned
during training?</p>
<p>Thanks again for your help.</p>Mon, 12 Jan 2015 15:52:26 Z2015-01-12T15:52:26Z
- https://social.microsoft.com/Forums/da-DK/dc056cb4-b8cb-4c13-9895-f68eba83b49b/using-variableconstrainpositive-on-a-gaussian-to-approximate-a-gamma-random-variable?forum=infer.net#17f22ad0-7185-4551-bd07-bd1448f62d0ahttps://social.microsoft.com/Forums/da-DK/dc056cb4-b8cb-4c13-9895-f68eba83b49b/using-variableconstrainpositive-on-a-gaussian-to-approximate-a-gamma-random-variable?forum=infer.net#17f22ad0-7185-4551-bd07-bd1448f62d0aTom Minkahttps://social.microsoft.com:443/profile/tom%20minka/?type=forumUsing Variable.ConstrainPositive on a Gaussian to approximate a Gamma random variableI assume in your example that ConstrainPositive is being applied to the mean. It does have an effect during training since it eliminates the possibility of the true mean being negative (which is possible when the data is non-negative).
The Gaussian posterior that you get approximates a truncated Gaussian. The prediction model does need to be different since it must exclude the constraint.Mon, 12 Jan 2015 17:59:46 Z2015-01-12T17:59:46Z
- https://social.microsoft.com/Forums/da-DK/dc056cb4-b8cb-4c13-9895-f68eba83b49b/using-variableconstrainpositive-on-a-gaussian-to-approximate-a-gamma-random-variable?forum=infer.net#e296a212-7be3-457b-a92a-274c0c15e7b7https://social.microsoft.com/Forums/da-DK/dc056cb4-b8cb-4c13-9895-f68eba83b49b/using-variableconstrainpositive-on-a-gaussian-to-approximate-a-gamma-random-variable?forum=infer.net#e296a212-7be3-457b-a92a-274c0c15e7b7Andrew Maohttps://social.microsoft.com:443/profile/andrew%20%20mao/?type=forumUsing Variable.ConstrainPositive on a Gaussian to approximate a Gamma random variable<p>Hi Tom,</p>
<p>I had actually thought that Variable.ConstrainPositive was being applied to the variable from Gaussian.FromMeanAndPrecision(meanPrior, precisionPrior). I didn't realize that you were applying it to the mean. Is this what you meant by the following?</p>
<p><span style="background-color:#d0d0d0">The prediction model does need to be different since it must exclude the constraint.</span></p>
<p>Applying it to the mean seems like it would only constrain the mean to be positive, not actual realizations of the variable.</p>
<p>In either case, I'd be predicting the mean of the posterior distribution. Are you saying that if we add the constraint again in prediction, we'd then be truncating a Gaussian that is already approximating a truncated Gaussian, which would further inflate
the mean prediction?</p>
<p>I'm sorry if I'm being unclear. I'll use some code to illustrate. Suppose I'm trying to train a simple model like the following, where I have some non-negative data and I'd like to learn a posterior distribution. I'd like to use the mean and precision distributions
I learned to make some other inferences. Are you suggesting to use Variable.ConstrainPositive in the first way, or the second way?</p>
<pre class="prettyprint">Variable<double> mean = Variable.GaussianFromMeanAndPrecision(0, 1);
Variable<double> prec = Variable.GammaFromShapeAndScale(1, 1);
Range k = new Range(100);
VariableArray<double> x = Variable.Array<double>(k).Named("x");
x[k] = Variable.GaussianFromMeanAndPrecision(mean, prec).ForEach(k);
Variable.ConstrainPositive(mean);
using (Variable.ForEach(k))
{
Variable.ConstrainPositive(x[k]);
}
// Observe some values
x.ObservedValue = Util.ArrayInit(k.SizeAsInt, i => 42d);
InferenceEngine engine = new InferenceEngine();
Console.WriteLine(engine.Infer<Gaussian>(mean));
Console.WriteLine(engine.Infer<Gamma>(prec));</pre>
<p>Thanks,</p>
<p>Andrew</p>
<div><span style="background-color:#d0d0d0"><br/>
</span></div>Mon, 12 Jan 2015 18:25:31 Z2015-01-12T18:25:31Z
- https://social.microsoft.com/Forums/da-DK/dc056cb4-b8cb-4c13-9895-f68eba83b49b/using-variableconstrainpositive-on-a-gaussian-to-approximate-a-gamma-random-variable?forum=infer.net#8bbc0f5a-5104-4a06-8c1d-e501a10bfecbhttps://social.microsoft.com/Forums/da-DK/dc056cb4-b8cb-4c13-9895-f68eba83b49b/using-variableconstrainpositive-on-a-gaussian-to-approximate-a-gamma-random-variable?forum=infer.net#8bbc0f5a-5104-4a06-8c1d-e501a10bfecbTom Minkahttps://social.microsoft.com:443/profile/tom%20minka/?type=forumUsing Variable.ConstrainPositive on a Gaussian to approximate a Gamma random variableIn the code you gave, applying ConstrainPositive to x[k] has no effect since x is observed to non-negative values. So it reduces to only constraining the mean as in my previous post.Tue, 13 Jan 2015 14:20:09 Z2015-01-13T14:20:09Z