locked
Strange errors and jumpy predictions with EP on a simple conditional model RRS feed

  • Question

  • Hi Infer.NET team,

    I'm trying to implement a model that captures the process of users doing tasks in a system. Each observation is the time since we last saw the user take an action. We'd like to model the idea that for each task, the user can be interrupted, and if they are interrupted, they may quit, which causes the observation of the time since we've last seen the user to take one of three distributions. Taking inspiration from the click through model, it looks something like this (please excuse the abuse of notation, code is below):

    Training the model on about 300,000 observations seems to give reasonable priors, i.e. a user has a 7% chance of being interrupted on any particular task, and a 4.7% chance of quitting if interrupted.

    Interruption Prob
    Beta(2.125e+04,2.786e+05)[mean=0.07086]
    Quit Prob
    Beta(1001,2.039e+04)[mean=0.0468]
    Inter-Task (Mins)
    Gaussian(0.1037, 3.326e-08)
    Inter-Session (Mins)
    Gaussian(399.3, 598)
    Quit Time (Mins)
    Gaussian(4.508e+04, 9993)
    

    I'm trying to use the model to make a prediction such as "if the user has been gone for X time", what's the probability that they have been interrupted, or quit? To make this prediction I'm constraining the value of the observed time and inferring the distribution on the Bernoulli variables using the priors. This is where things start to get a little wonky.

    For starters, the inference crashes with an error about a gaussian with infinite variance for some values of time. If I try another value close by, it can usually produce a result. The error looks like this:


    For reference, the relevant parts of code for the model is below.

    class TaskTimeModel {
        private Variable<bool> evidence;
    
        private Variable<Beta> interruptionPrior;
        private Variable<Beta> quittingPrior;
    
        private Variable<double> interruptionProb;
        private Variable<double> quittingProb;
    
        private Variable<Gaussian> interTaskMeanPrior;
        private Variable<Gamma> interTaskPrecPrior;
    
        private Variable<Gaussian> interSessionMeanPrior;
        private Variable<Gamma> interSessionPrecPrior;
    
        private Variable<Gaussian> quitTimeMeanPrior;
        private Variable<Gamma> quitTimePrecPrior;
    
        private Variable<double> interTaskMean;
        private Variable<double> interTaskPrec;
    
        private Variable<double> interSessionMean;
        private Variable<double> interSessionPrec;
    
        private Variable<double> quitTimeMean;
        private Variable<double> quitTimePrec;        
    
        // User variables
        private Variable<int> numUsers;
        private Range n;
    
        private VariableArray<int> numTasks; 
        private Variable<int> maxTasks;
        private Range nt;
        private Range mt;
    
        private VariableArray<VariableArray<bool>, bool[][]> interrupted;
        private VariableArray<VariableArray<bool>, bool[][]> quit;
        private VariableArray<VariableArray<double>, double[][]> dwellTime;
    
        // Special variable for prediction
        private Variable<bool> isPredicting;
        private VariableArray<double> dwellTimeConstraint;
    
        // TODO set this up in a parent class
        InferenceEngine engine = new InferenceEngine {
            Algorithm = new ExpectationPropagation() // EP is needed for both the AND factor as well as truncated inference
            // Algorithm = new VariationalMessagePassing()
        };
    
        public TaskTimeModel() {
            evidence = Variable.Bernoulli(0.5).Named("evidence");
    
            using (Variable.If(evidence)) {
                numUsers = Variable.New<int>().Named("numUsers");
                n = new Range(numUsers).Named("n");
    
                numTasks = Variable.Array<int>(n).Named("numTasks");
                nt = new Range(numTasks[n]).Named("nt");
    
                maxTasks = Variable.New<int>().Named("maxTasks");
                mt = new Range(maxTasks).Named("mt");
    
                interrupted = Variable.Array(Variable.Array<bool>(nt), n).Named("interrupted");
                interrupted.SetValueRange(mt);
    
                quit = Variable.Array(Variable.Array<bool>(nt), n).Named("quit");
                quit.SetValueRange(mt);
    
                dwellTime = Variable.Array(Variable.Array<double>(nt), n).Named("dwellTime");
                dwellTime.SetValueRange(mt);
    
                // Priors
                interruptionPrior = Variable.New<Beta>();
                quittingPrior = Variable.New<Beta>();
    
                interTaskMeanPrior = Variable.New<Gaussian>();                                
                interTaskPrecPrior = Variable.New<Gamma>();
                    
                interSessionMeanPrior = Variable.New<Gaussian>();                
                interSessionPrecPrior = Variable.New<Gamma>();
    
                quitTimeMeanPrior = Variable.New<Gaussian>();
                quitTimePrecPrior = Variable.New<Gamma>();
    
                // Model
                interruptionProb = Variable<double>.Random(interruptionPrior);
                quittingProb = Variable<double>.Random(quittingPrior);
    
                interTaskMean = Variable<double>.Random(interTaskMeanPrior);
                interTaskPrec = Variable<double>.Random(interTaskPrecPrior);
    
                interSessionMean = Variable<double>.Random(interSessionMeanPrior);
                interSessionPrec = Variable<double>.Random(interSessionPrecPrior);
    
                quitTimeMean = Variable<double>.Random(quitTimeMeanPrior);
                quitTimePrec = Variable<double>.Random(quitTimePrecPrior);
    
                isPredicting = Variable.New<bool>();
                dwellTimeConstraint = Variable.Array<double>(n);
    
                using (Variable.ForEach(n))
                {
                    using (Variable.ForEach(nt)) {
                        interrupted[n][nt] = Variable.Bernoulli(interruptionProb);
                        // The following AND factor requires EP for inference (quite a bit slower)
                        quit[n][nt] = interrupted[n][nt] & Variable.Bernoulli(quittingProb);
                            
                        using (Variable.IfNot(interrupted[n][nt])) {                            
                            dwellTime[n][nt] = Variable.GaussianFromMeanAndPrecision(interTaskMean, interTaskPrec);
                        }
                        using (Variable.If(interrupted[n][nt]))
                        {
                            // Must define this nested way to avoid gate transform error (variable not defined in all cases)
                            using (Variable.IfNot(quit[n][nt]))
                            {
                                dwellTime[n][nt] = Variable.GaussianFromMeanAndPrecision(interSessionMean, interSessionPrec);
                            }
                            using (Variable.If(quit[n][nt])) {
                                dwellTime[n][nt] = Variable.GaussianFromMeanAndPrecision(quitTimeMean, quitTimePrec);    
                            }                                                        
                        }                        
                    }
                }
    
                // Constraint if predicting
                using (Variable.If(isPredicting))
                {
                    Variable.ConstrainPositive(dwellTime[n][nt] - dwellTimeConstraint[n]);
                }
    
            }
        }        
    
        public void SetPriors(TaskTimePriors p) {            
            interruptionPrior.ObservedValue = p.InterruptionProb;
            quittingPrior.ObservedValue = p.QuittingProb;
    
            interTaskMeanPrior.ObservedValue = p.InterTaskMean;
            interTaskPrecPrior.ObservedValue = p.InterTaskPrec;
    
            interSessionMeanPrior.ObservedValue = p.InterSessionMean;
            interSessionPrecPrior.ObservedValue = p.InterSessionPrec;
    
            quitTimeMeanPrior.ObservedValue = p.QuitTimeMean;
            quitTimePrecPrior.ObservedValue = p.QuitTimePrec;                         
        }
    
        public TaskTimePriors Train(IList<UserTaskHistory> data) {
            isPredicting.ObservedValue = false;
            // dwellTimeConstraint.ObservedValue = 0;
    
            numUsers.ObservedValue = data.Count;
            numTasks.ObservedValue = data.TaskCounts();
    
            maxTasks.ObservedValue = numTasks.ObservedValue.Max();
    
            quit.ObservedValue = data.QuitBools();
            // TODO: may need to label interruption values as well
            dwellTime.ObservedValue = data.LastSeenMinutes();
    
            return new TaskTimePriors {
                Evidence = engine.Infer<Bernoulli>(evidence),
                InterruptionProb = engine.Infer<Beta>(interruptionProb),
                QuittingProb = engine.Infer<Beta>(quittingProb),
                InterTaskMean = engine.Infer<Gaussian>(interTaskMean),
                InterTaskPrec = engine.Infer<Gamma>(interTaskPrec),
                InterSessionMean = engine.Infer<Gaussian>(interSessionMean),
                InterSessionPrec = engine.Infer<Gamma>(interSessionPrec),
                QuitTimeMean = engine.Infer<Gaussian>(quitTimeMean),
                QuitTimePrec = engine.Infer<Gamma>(quitTimePrec)
            };
        }
    
        public IList<TimePredictionResult> Predict(IList<UserTask> obs) {
            isPredicting.ObservedValue = true;
            dwellTimeConstraint.ObservedValue = obs.Select(o => o.MinutesSinceLastSeen).ToArray();
    
            int num = obs.Count;
            numUsers.ObservedValue = num;
            numTasks.ObservedValue = Util.ArrayInit(num, t => 1);
            maxTasks.ObservedValue = 1;
    
            engine.ShowProgress = false;
                
            var intProb = engine.Infer<Bernoulli[][]>(interrupted);
            var quitProb = engine.Infer<Bernoulli[][]>(quit);
    
            var result = new List<TimePredictionResult>(num);
    
            for (int i = 0; i < intProb.Length; i++) {
                result.Add(new TimePredictionResult {
                    CondIntProb = intProb[i][0].GetProbTrue(),
                    CondQuitProb = quitProb[i][0].GetProbTrue()
                });
            }
    
            return result;
        }
    
        // other stuff
    }
    There's a limit of two images per message, so I'll continue the question in another post.


    Wednesday, February 4, 2015 4:34 PM

All replies

  • If I ignore the values for which the model fails on and plot the probabilities as predicted for different values of the truncated observation, I get something like the following. It looks like the predicted probabilities aren't smooth across parts of the range.

    It looks like the graph wants to look something like this (which we'd expect from the model), but something is going wrong in the inference process.

    I'd be grateful if you could help with any of the following questions:

    • What could be causing the prediction inference to fail on certain values of the observed time constraint?
    • Why are the predicted values that are close to each other (on log scale) so nonlinear? Could it be a problem with the approximating distribution, or perhaps a convergence issue?

    If you have any suggestions for adjustments to the model that could make it perform better, I'd be happy to try them.

    Thanks!

    Wednesday, February 4, 2015 4:42 PM
  • The way that you have written the model makes it difficult for EP to handle.  Consider the marginal distribution of dwellTime just as it reaches the ConstrainPositive statement.  It will be a mixture of 3 Gaussians.  You can remove this complexity by instead copying the ConstrainPositive statement 3 times, putting each copy directly after an assignment to dwellTime.  That way dwellTime has a simple distribution in each ConstrainPositive statement.  For example:

    using (Variable.IfNot(interrupted[n][nt])) {                            
        dwellTime[n][nt] = Variable.GaussianFromMeanAndPrecision(interTaskMean, interTaskPrec);
        // Constraint if predicting
        using (Variable.If(isPredicting))
        {
           Variable.ConstrainPositive(dwellTime[n][nt] - dwellTimeConstraint[n]);
        }
    }
    

    Wednesday, February 4, 2015 8:07 PM
    Owner
  • Hi Tom, that was very helpful and completely makes sense with regard to how EP works. Making that change gets rid of all the inference errors. However, there are still a few values that result in strange predictions - on the graph below, 60, 75, 90, and 30240 (minutes).

    The model is predicting both probabilities as 0, which seems like it should be impossible for any positive value of the constraint. Any idea what might be causing this behavior? Could it possibly be a symmetry problem that could be helped with initialization?

    Note also the blip at the constraint of 30240.

    Trying to make a bunch of predictions with the same inference run also seems to result in this behavior, although at different values - the probabilities pop out as 0 or 1 instead of being smooth.

    Thanks!

    Thursday, February 5, 2015 3:20 AM
  • You can try the same trick with the definition of quit.  In the not interrupted branch, you know that quit is false.  In the interrupted branch, you know that quit = Variable.Bernoulli(quittingProb);
    Thursday, February 5, 2015 12:23 PM
    Owner
  • Hi Tom,

    I'm a little confused by what you're suggesting. Is it the following:

    using (Variable.ForEach(n))
    {
        using (Variable.ForEach(nt)) {
            interrupted[n][nt] = Variable.Bernoulli(interruptionProb);                        
    
            using (Variable.IfNot(interrupted[n][nt])) {
                quit[n][nt].SetTo(false);
                dwellTime[n][nt] = Variable.GaussianFromMeanAndPrecision(interTaskMean, interTaskPrec);
    
                using (Variable.If(isPredicting))
                    Variable.ConstrainPositive(dwellTime[n][nt] - dwellTimeConstraint[n]);
            }
            using (Variable.If(interrupted[n][nt])) {
                quit[n][nt].SetTo(Variable.Bernoulli(quittingProb));
    
                using (Variable.IfNot(quit[n][nt]))
                {
                    dwellTime[n][nt] = Variable.GaussianFromMeanAndPrecision(interSessionMean, interSessionPrec);
    
                    using (Variable.If(isPredicting))
                        Variable.ConstrainPositive(dwellTime[n][nt] - dwellTimeConstraint[n]);
                }
                using (Variable.If(quit[n][nt])) {
                    dwellTime[n][nt] = Variable.GaussianFromMeanAndPrecision(quitTimeMean, quitTimePrec);
    
                    using (Variable.If(isPredicting))
                        Variable.ConstrainPositive(dwellTime[n][nt] - dwellTimeConstraint[n]);
                }                                                        
            }                        
        }
    }   

    When I tried this before, I had errors training the model - the distributions weren't learned properly (only the form with the AND factor worked). Anyway, it works fine for inference, but still results in a similar result - although now, note that the last few predictions are different than before. Any idea what is going on?

    On a related note, there is this page on the importance of using the SetTo function. However, things seem to work either when when using `=` or `SetTo` in the conditional branches above. Is this just because the assignments are to VariableArrays indexed by Ranges, and not vanilla variables?

    Thanks,

    Andrew

    Friday, February 6, 2015 5:38 AM
  • Have you checked that inference is converging?  The section at the end of the FAQ has some more tips.
    Friday, February 6, 2015 1:19 PM
    Owner