Answered by:
ImproperDistributionException in a builtin factor
Question

Hi all,
I am new to Infer.NET. I am trying to implement my custom operator for Logistic factor. Operator attribute:
[FactorMethod(typeof(MMath), "Logistic", typeof(double))]
I create a binary logistic model with the following code.
Variable<int> dataCount = Variable.Observed(xObs.Length).Named("dataCount"); int D = xObs[0].Count; Range n = new Range(dataCount).Named("n"); Vector meanW = Vector.Zero(D); Variable<Vector> w = Variable.VectorGaussianFromMeanAndPrecision( meanW, PositiveDefiniteMatrix.Identity(D)).Named("W"); Variable<double> bias = Variable.GaussianFromMeanAndPrecision(0, 1).Named("bias"); VariableArray<Vector> x = Variable.Observed<Vector>(null, n).Named("X"); VariableArray<double> logisticArgs = Variable.Array<double>(n).Named("logist_arg"); logisticArgs[n] = Variable.InnerProduct(x[n], w).Named("inner") + bias ; VariableArray<bool> yData = Variable.Observed<bool>(null, n).Named("Y"); VariableArray<double> probs = Variable.Array<double>(n).Named("probs"); probs[n] = Variable.Logistic(logisticArgs[n]).Named("logistic"); yData[n] = Variable.Bernoulli(probs[n]); // x.ObservedValue and yData.ObservedValue are properly set
The goal is to infer meanW (VectorGaussian) and bias (Gaussian) with EP. Among others, I would need to implement
public static Gaussian XAverageConditional(Beta logistic, Gaussian x){....}
in the class of my custom operator for sending an EP message to X, where X here is the argument of logistic(X). As I understand it, an EP message to X in this case can be an improper Gaussian. In fact during the inference, most of my outgoing messages to X are improper. However, when an outgoing message from the plus factor needs to be computed, an exception is thrown. I tracked it down and believed this method threw it.
public static Gaussian AAverageConditional([SkipIfUniform] Gaussian Sum, [SkipIfUniform] Gaussian b) { if (Sum.IsPointMass) return AAverageConditional(Sum.Point, b); if (b.IsPointMass) return AAverageConditional(Sum, b.Point); if (Sum.Precision == 0) return Gaussian.FromNatural(Sum.MeanTimesPrecision, 0); double prec = Sum.Precision + b.Precision; if (prec <= 0) throw new ImproperDistributionException(Sum.IsProper() ? b : Sum); return Gaussian.FromNatural((Sum.MeanTimesPrecision*b.Precision  b.MeanTimesPrecision*Sum.Precision)/prec, Sum.Precision*b.Precision/prec); }
The method AAverageConditional(...) is from DoublePlusOp class in PlusDouble.cs
I *guess* that the argument Gaussian sum is what is output from my custom operator. The argument Gaussian b is supposed to be my bias variable. It seems the exception is thrown because sum is improper.
 How do we deal with this issue ?
 In general, if a custom implementation of message operator outputs an improper message, how can we be sure that the next operator in the chain (message operator for plus in this case) can accept it ?
 Or do outgoing messages from a factor have to always be proper ? If so, is there any heuristic to convert an improper message to its "closest" (in some sense) proper message ?
I would love to attach my factor graph but I could not. Got "Body text cannot contain images or links until we are able to verify your account.".
Thanks,
Wittawat
 Edited by wittawatj Sunday, September 14, 2014 7:33 PM
Sunday, September 14, 2014 6:53 PM
Answers

Under a correct implementation of EP, XAverageConditional cannot be improper because a Gaussian times a logistic likelihood always has smaller variance than the original Gaussian.
 Marked as answer by wittawatj Monday, September 15, 2014 5:44 PM
Monday, September 15, 2014 12:42 PMOwner
All replies

In the model you attached, the output of XAverageConditional cannot be improper. So I'm guessing it is a bug in your implementation of this method. Since Infer.NET already has a method (on LogisticOp) that does the same thing, you can check whether the results match.Monday, September 15, 2014 12:25 PMOwner

Thanks Tom. My operator is a distributiontodistribution regression map which is expected to produce inaccurate messages. Why cannot XAverageConditional output an improper message ? Is it simply because the plus operator expects proper input messages ? Any reference for me to understand more on this matter ? Thanks.
 Edited by wittawatj Monday, September 15, 2014 12:44 PM
Monday, September 15, 2014 12:38 PM 
Under a correct implementation of EP, XAverageConditional cannot be improper because a Gaussian times a logistic likelihood always has smaller variance than the original Gaussian.
 Marked as answer by wittawatj Monday, September 15, 2014 5:44 PM
Monday, September 15, 2014 12:42 PMOwner 
Thanks. I will think about it.
I have another question. Since Y is observed, shouldn't an outgoing message from Bernoulli factor produce a Beta point mass ? Say Y[0] = false. Intuitively I expect something like a Beta(true count=1, false count=large number). However the messages I got are Beta(1, 2) which looks as if you originally have uniform Beta(1, 1) and increment false count by 1. Why is that ? I think I may have missed something totally obvious.
Monday, September 15, 2014 1:06 PM 
In general, the message from EP is the ratio of the posterior over the prior. If you are trying to learn the bias of a coin and you see one flip which is false, the posterior increments the false count by 1. Taking the ratio gives the message Beta(1,2).Monday, September 15, 2014 1:15 PMOwner

Thanks Tom. I understand the part of updating Beta prior to a get a Beta posterior by incrementing false/true counts. What I don't see is where the prior Beta(1, 1) of Y comes from. It is not in my factor graph when I do ie.ShowFactorGraph = true; . Is it embedded somewhere by default ? If so how can I change the prior ?Monday, September 15, 2014 2:29 PM

Just to make it precise. The prior may not be on Y. We have the graph that looks like
(something more)... > Plus factor > logist_arg[n] > Logistic factor > probs[n] > Bernoulli > Y[n]
I observe Y[n]. Say Y[0] = false. Now the message "logistic" that gets into the logistic operator
public static Gaussian XAverageConditional(Beta logistic, Gaussian x){....}
is Beta(1, 2). So where is Beta(1, 1) defined ? How can I change this prior ? Thanks for your time.Monday, September 15, 2014 2:35 PM 
No one said the prior was Beta(1,1). The message is the ratio of the posterior over the prior, so the prior is not part of the message. Try the following with Infer.NET: (new Beta(a,b+1))/(new Beta(a,b)) for any choice of a and b.Monday, September 15, 2014 2:41 PMOwner

Got it after some derivations. You are right. The outgoing Beta message is the ratio which is either Beta(1, 2) or Beta(2, 1) depending on the observed Y. I am not sure why you said "message from EP is the ratio of the posterior over the prior" though. Perhaps what you refer to as the prior is in general the cavity ? But this is not important now. I will think about the product of Gaussian and logistic likelihood issue and ask again. Thanks.Monday, September 15, 2014 4:51 PM