locked
Trouble scaling to a very large graph with SharedVariables. RRS feed

  • Question

  • Dear Infer.NET community:

    I am having issues scaling a large model, probably due to the lack of understanding how multiple models should be created. I have read all the resources I could find on it, but still having trouble. My problem involves loading a very large graph (many hundreds of thousands of variables and about a million factors). I want to split the graph into pieces and perform inference with separate models, using shared variables. I create the variables and factors in the following way:

    for (int i = 0; i < this.scores.Count; i++)
                {
                    int user_A_id     = this.scores[i].user_A_id;
                    int user_B_id     = this.scores[i].user_B_id;
                    int user_A_module = this.scores[i].user_A_module;
                    int user_B_module = this.scores[i].user_B_module;
                    double score_diff = this.scores[i].user_A_score - this.scores[i].user_B_score;
    
                    this.ScoreVariable[i] = Variable.BernoulliFromLogOdds( score_diff
                                                                    * (this.UserById[user_A_id].GetCopyFor(this.modelByModule[user_A_module]) -
                                                                    this.UserById[user_B_id].GetCopyFor(this.modelByModule[user_B_module])));

    the UserById[user_id] simply returns the appropriate random variable from an array of Gaussian random variables. And modelBymodule function looks up the model that this variable belongs to. This is determined and pre-computed entirely separately, in a way that each of the models is sufficiently small to fit in memory.

    Finally, inference is performed as follows:

     public void performInferencePass()
            {
                for (int i = 0; i < this.models.Length; i++)
                {
                  
                    this.models[i].InferShared(this.engine, 0);
                }
            }

    The problem that I run into is that the loop never completes the first iteration --- at least it hasn't in about an hour. I don't get out of memory exceptions (like I did when the model wasn't split up as is now), but it just seems to take forever. It seems then, unless I completely did this wrong, that the entire graph is processed first, and regardless of how the models are split, this step might still take a very very long time?

    Do you have any insight into how such models can be inferred tractably? I looked at the tutorial on irregular graphs, which suggests to encode all of the users into a single VariableArray, but then I can't seem to figure out how to define the likelihood such that different pairs are allowed to have different multiplicative constants in the factor definition? (see the way I defined the BernoulliFromLogOdds, where score_diff is pre-computed from the data and depends on the items).

    Finally, I was wondering if the following trick could work: Can I load each of the modules (a subset of the graph) sequentially, perform inference, and simply assign each variable's old prior as the new posterior. When the next section of the graph is loaded, those variables that were part of the first module would retain their posterior in the form of the prior. Then multiple passes are performed over the entire graph until convergence. I can't quite see if this will be equivalent to saving outgoing and incoming messages as done automatically by SharedVariables?

    Thank you so much for your help!

    Thursday, April 17, 2014 5:47 AM

Answers

  • Mostly likely the graph is too big.  Try putting the users into a single VariableArray, with another VariableArray for the scores.
    • Marked as answer by IgorNeletov Monday, April 21, 2014 11:40 PM
    Thursday, April 17, 2014 9:43 AM
    Owner
  • Hi Igor

    It is likely that it is the model compilation that is taking all the time. Are you seeing:

    Compiling model...done.?

    Any .NET arrays will be unrolled in the compiled model, so it is essential that any array variables use the VariableArray type as Tom suggests. If you cannot see how to do the likelihood, the easiest way for us to provide help is for you to provide a small running test program of your original model (i.e. not using shared variables) along with a dummy data set (for example just sample from your model). Send these to infersup at Microsoft dot com.

    The approach that you suggest of looping several times over the modules, just using posteriors as priors, is not valid and is not equivalent to shared variables (see the first paragraph of http://research.microsoft.com/en-us/um/cambridge/projects/infernet/docs/Sharing%20variables%20between%20models.aspx). If you do just a single pass over the modules, this will be a batched online version of inference (i.e. batched in that you are presenting batches of data, and online because you are never revisiting the data); this is valid, but less accurate than full inference. But I think you should concentrate in the first instance on getting things into VariableArrays.

    John

    • Marked as answer by IgorNeletov Monday, April 21, 2014 11:40 PM
    Thursday, April 17, 2014 10:02 AM
    Owner

All replies

  • Mostly likely the graph is too big.  Try putting the users into a single VariableArray, with another VariableArray for the scores.
    • Marked as answer by IgorNeletov Monday, April 21, 2014 11:40 PM
    Thursday, April 17, 2014 9:43 AM
    Owner
  • Hi Igor

    It is likely that it is the model compilation that is taking all the time. Are you seeing:

    Compiling model...done.?

    Any .NET arrays will be unrolled in the compiled model, so it is essential that any array variables use the VariableArray type as Tom suggests. If you cannot see how to do the likelihood, the easiest way for us to provide help is for you to provide a small running test program of your original model (i.e. not using shared variables) along with a dummy data set (for example just sample from your model). Send these to infersup at Microsoft dot com.

    The approach that you suggest of looping several times over the modules, just using posteriors as priors, is not valid and is not equivalent to shared variables (see the first paragraph of http://research.microsoft.com/en-us/um/cambridge/projects/infernet/docs/Sharing%20variables%20between%20models.aspx). If you do just a single pass over the modules, this will be a batched online version of inference (i.e. batched in that you are presenting batches of data, and online because you are never revisiting the data); this is valid, but less accurate than full inference. But I think you should concentrate in the first instance on getting things into VariableArrays.

    John

    • Marked as answer by IgorNeletov Monday, April 21, 2014 11:40 PM
    Thursday, April 17, 2014 10:02 AM
    Owner
  • Thank you Tom and John, very much! I have rewritten the model with VariableArrays, and there is no need for shared variables anymore. I will post the code here in a bit, in case someone else comes across a similar problem.


    Thanks again!

    Friday, April 18, 2014 5:28 AM