Strange Results Recommender System Example RRS feed

  • Question

  • Hi Everyone

    I'm working with the Recommender System Example in order to take confidence with Infer.NET. I have changed a little bit the code in order to retrieve the real value of movielens dataset from a file. The code i'm currently use is the following:

    using System;
    using System.Linq;
    using System.IO;
    using MicrosoftResearch;
    using MicrosoftResearch.Infer;
    using MicrosoftResearch.Infer.Models;
    using MicrosoftResearch.Infer.Utils;
    using MicrosoftResearch.Infer.Distributions;
    using MicrosoftResearch.Infer.Maths;
    using MicrosoftResearch.Infer.Factors;
    using MicrosoftResearch.Infer.Collections;
    namespace ColdStart
        public class ModelRS
            // Define counts
            private int numUsers;
            private int numItems;
            private int numTraits;
            private int numLevels;
            private Variable<int> numObservations;
            // Define ranges
            private Range user;
            private Range item;
            private Range trait;
            private Range observation;
            private Range level;
            // Define latent variables
            private VariableArray<VariableArray<double>, double[][]> userTraits;
            private VariableArray<VariableArray<double>, double[][]> itemTraits;
            private VariableArray<double> userBias;
            private VariableArray<double> itemBias;
            private VariableArray<VariableArray<double>, double[][]> userThresholds;
            // Define priors
            private VariableArray<VariableArray<Gaussian>, Gaussian[][]> userTraitsPrior;
            private VariableArray<VariableArray<Gaussian>, Gaussian[][]> itemTraitsPrior;
            private VariableArray<Gaussian> userBiasPrior;
            private VariableArray<Gaussian> itemBiasPrior;
            private VariableArray<VariableArray<Gaussian>, Gaussian[][]> userThresholdsPrior;
            private Gaussian[][] userTraitsPosterior;
            private Gaussian[][] itemTraitsPosterior;
            private Gaussian[] userBiasPosterior;
            private Gaussian[] itemBiasPosterior;
            private Gaussian[][] userThresholdsPosterior;
            // Model noises
            Variable<double> affinityNoiseVariance;
            Variable<double> thresholdsNoiseVariance;
            private Gaussian traitPrior;
            private Gaussian biasPrior;
            InferenceEngine engine;
            public ModelRS(int iteration,double noiseVar, int numTrts)
                //numUsers = 943;
                //numItems = 1682;
                numUsers = 6040;
                numItems = 3982;
                numTraits = numTrts;
                numLevels = 4;
                user = new Range(numUsers).Named("user");
                item = new Range(numItems).Named("item");
                trait = new Range(numTraits).Named("trait");
                level = new Range(numLevels).Named("level");
                numObservations = Variable.Observed(1).Named("numObservations");
                observation = new Range(numObservations).Named("observation");
                //testObservation = new Range(numObservations).Named("testObservation");
                // Define latent variables
                userTraits = Variable.Array(Variable.Array<double>(trait), user).Named("userTraits");
                itemTraits = Variable.Array(Variable.Array<double>(trait), item).Named("itemTraits");
                userBias = Variable.Array<double>(user).Named("userBias");
                itemBias = Variable.Array<double>(item).Named("itemBias");
                userThresholds = Variable.Array(Variable.Array<double>(level), user).Named("userThresholds");
                // Define priors
                userTraitsPrior = Variable.Array(Variable.Array<Gaussian>(trait), user).Named("userTraitsPrior");
                itemTraitsPrior = Variable.Array(Variable.Array<Gaussian>(trait), item).Named("itemTraitsPrior");
                userBiasPrior = Variable.Array<Gaussian>(user).Named("userBiasPrior");
                itemBiasPrior = Variable.Array<Gaussian>(item).Named("itemBiasPrior");
                userThresholdsPrior = Variable.Array(Variable.Array<Gaussian>(level), user).Named("userThresholdsPrior");
                // Define latent variables statistically
                userTraits[user][trait] = Variable<double>.Random(userTraitsPrior[user][trait]);
                itemTraits[item][trait] = Variable<double>.Random(itemTraitsPrior[item][trait]);
                userBias[user] = Variable<double>.Random(userBiasPrior[user]);
                itemBias[item] = Variable<double>.Random(itemBiasPrior[item]);
                userThresholds[user][level] = Variable<double>.Random(userThresholdsPrior[user][level]);
                // This example requires EP
                engine = new InferenceEngine();
                engine.NumberOfIterations = iteration;
                // Set model noises explicitly
                affinityNoiseVariance = Variable.Observed(noiseVar).Named("affinityNoiseVariance");
                thresholdsNoiseVariance = Variable.Observed(noiseVar).Named("thresholdsNoiseVariance");
            public void Intialize(double priorVar)
                traitPrior = Gaussian.FromMeanAndVariance(0.0, priorVar);
                biasPrior = Gaussian.FromMeanAndVariance(0.0, priorVar);
                userTraitsPrior.ObservedValue = Util.ArrayInit(numUsers, u => Util.ArrayInit(numTraits, t => traitPrior));
                itemTraitsPrior.ObservedValue = Util.ArrayInit(numItems, i => Util.ArrayInit(numTraits, t => traitPrior));
                userBiasPrior.ObservedValue = Util.ArrayInit(numUsers, u => biasPrior);
                itemBiasPrior.ObservedValue = Util.ArrayInit(numItems, i => biasPrior);
                userThresholdsPrior.ObservedValue = Util.ArrayInit(numUsers, u =>
                        Util.ArrayInit(numLevels, l => Gaussian.FromMeanAndVariance(l - numLevels / 2.0 + 0.5, 1.0)));
                // Break symmetry and remove ambiguity in the traits
                for (int i = 0; i < numTraits; i++)
                    // Assume that numTraits < numItems
                    for (int j = 0; j < numTraits; j++)
                        itemTraitsPrior.ObservedValue[i][j] = Gaussian.PointMass(0);
                    itemTraitsPrior.ObservedValue[i][i] = Gaussian.PointMass(1);
            public void Test(double priorVar, string trainFilename, string testFilename, out float MAE, out double RMSE)
                //*********************** File Reading Start ***********************
                int[] tmpUser;
                int[] tmpItem;
                int[] tmpRating;
                LoadData(trainFilename, out tmpUser, out tmpItem, out tmpRating, "train");
                //*********************** File Reading End ************************
                Variable<int> testNumObservations = Variable.Observed(tmpItem.Length);
                Range testObservation = new Range(testNumObservations);
                var trainUserData = Variable.Array<int>(testObservation);
                var trainItemData = Variable.Array<int>(testObservation);
                var trainRatingData = Variable.Array(Variable.Array<bool>(level), testObservation);
                // Model
                using (Variable.ForEach(testObservation))
                    VariableArray<double> products = Variable.Array<double>(trait);//.Named("products");
                    products[trait] = userTraits[trainUserData[testObservation]][trait] * itemTraits[trainItemData[testObservation]][trait];
                    Variable<double> bias = (userBias[trainUserData[testObservation]] + itemBias[trainItemData[testObservation]]);//.Named("bias");
                    Variable<double> affinity = (bias + Variable.Sum(products));//.Named("productSum")).Named("affinity");
                    Variable<double> noisyAffinity = Variable.GaussianFromMeanAndVariance(affinity, affinityNoiseVariance);//.Named("noisyAffinity");
                    VariableArray<double> noisyThresholds = Variable.Array<double>(level);//.Named("noisyThresholds");
                    noisyThresholds[level] = Variable.GaussianFromMeanAndVariance(userThresholds[trainUserData[testObservation]][level], thresholdsNoiseVariance);
                    trainRatingData[testObservation][level] = noisyAffinity > noisyThresholds[level];
                // Observe training data
                GenerateData(numUsers, numItems, numTraits, testNumObservations.ObservedValue, numLevels,
                                         trainUserData, trainItemData, trainRatingData,
                                         userTraitsPrior.ObservedValue, itemTraitsPrior.ObservedValue,
                                         userBiasPrior.ObservedValue, itemBiasPrior.ObservedValue, userThresholdsPrior.ObservedValue,
                                         affinityNoiseVariance.ObservedValue, thresholdsNoiseVariance.ObservedValue, tmpUser, tmpItem,
                // Allow EP to process the product factor as if running VMP
                // as in Stern, Herbrich, Graepel paper.
                engine.Compiler.ShowWarnings = true;
                // Run inference
                userTraitsPosterior = engine.Infer<Gaussian[][]>(userTraits);
                itemTraitsPosterior = engine.Infer<Gaussian[][]>(itemTraits);
                userBiasPosterior = engine.Infer<Gaussian[]>(userBias);
                itemBiasPosterior = engine.Infer<Gaussian[]>(itemBias);
                userThresholdsPosterior = engine.Infer<Gaussian[][]>(userThresholds);
                // Feed in the inferred posteriors as the new priors
                userTraitsPrior.ObservedValue = userTraitsPosterior;
                itemTraitsPrior.ObservedValue = itemTraitsPosterior;
                userBiasPrior.ObservedValue = userBiasPosterior;
                itemBiasPrior.ObservedValue = itemBiasPosterior;
                userThresholdsPrior.ObservedValue = userThresholdsPosterior;
                // Make a prediction
                int[] usersTest;
                int[] itemsTest;
                int[] testRatings;
                LoadData(testFilename, out usersTest, out itemsTest, out testRatings, "test");
                testNumObservations.ObservedValue = usersTest.Length;
                trainUserData.ObservedValue = usersTest;
                trainItemData.ObservedValue = itemsTest;
                Bernoulli[][] predictedRatings = engine.Infer<Bernoulli[][]>(trainRatingData);
                computeMAE_RMSE(testNumObservations.ObservedValue, predictedRatings, testRatings, out MAE, out RMSE);
            private void computeMAE_RMSE(int numObservations, Bernoulli[][] predictedRating, int[] testRating, out float MAE, out double RMSE)
                float tmpMAE = 0;
                double tmpRMSE = 0;
                for (int i = 0; i < numObservations; i++)
                    double value;
                    int prediction = 1;
                    foreach (var rating in predictedRating[i])
                        value = rating.GetMean();
                        if ((float)value > 0.5)
                            prediction += 1;
                    tmpMAE += Math.Abs(prediction - testRating[i]);
                    tmpRMSE += Math.Pow(prediction - testRating[i], 2);
                tmpMAE = tmpMAE / numObservations;
                tmpRMSE = Math.Sqrt(tmpRMSE / numObservations);
                MAE = tmpMAE;
                RMSE = tmpRMSE;
            // Generates data from the model
            void GenerateData(int numUsers, int numItems, int numTraits, int numObservations, int numLevels,
                                                VariableArray<int> userData, VariableArray<int> itemData,
                                                VariableArray<VariableArray<bool>, bool[][]> ratingData,
                                                Gaussian[][] userTraitsPrior, Gaussian[][] itemTraitsPrior,
                                                Gaussian[] userBiasPrior, Gaussian[] itemBiasPrior, Gaussian[][] userThresholdsPrior,
                                                double affinityNoiseVariance, double thresholdsNoiseVariance,
                                                int[] users, int[] items, int[] ratings)
                int[] generatedUserData = users;
                int[] generatedItemData = items;
                bool[][] generatedRatingData = new bool[numObservations][];
                // Repeat the model with fixed parameters
                for (int observation = 0; observation < numObservations; observation++)
                    switch (ratings[observation])
                        case 1:
                            generatedRatingData[observation] = new bool[] { false, false, false, false };
                        case 2:
                            generatedRatingData[observation] = new bool[] { true, false, false, false };
                        case 3:
                            generatedRatingData[observation] = new bool[] { true, true, false, false };
                        case 4:
                            generatedRatingData[observation] = new bool[] { true, true, true, false };
                        case 5:
                            generatedRatingData[observation] = new bool[] { true, true, true, true };
                userData.ObservedValue = generatedUserData;
                itemData.ObservedValue = generatedItemData;
                ratingData.ObservedValue = generatedRatingData;
            static private void LoadData(
                string ifn,         // The file name
                out int[] tmpUser,   // users
                out int[] tmpItem,   // movies
                out int[] tmpRating,   // ratings
                string t)
                // File is assumed to have tab or comma separated label, clicks, exams
                tmpUser = null;
                tmpItem = null;
                tmpRating = null;
                int totalDocs = 0;
                string myStr;
                StreamReader mySR;
                char[] sep = { '\t', ',', ' ' };
                for (int pass = 0; pass < 2; pass++)
                    if (1 == pass)
                        tmpUser = new int[totalDocs];
                        tmpItem = new int[totalDocs];
                        tmpRating = new int[totalDocs];
                        totalDocs = 0;
                    mySR = new StreamReader(ifn);
                    while ((myStr = mySR.ReadLine()) != null)
                        if (1 == pass)
                            string[] mySplitStr = myStr.Split(sep);
                            int rat = int.Parse(mySplitStr[2]);
                            int urs = int.Parse(mySplitStr[0]);
                            int itm = int.Parse(mySplitStr[1]);
                            tmpUser[totalDocs] = urs - 1;
                            tmpItem[totalDocs] = itm - 1;
                            tmpRating[totalDocs] = rat;

    I call this code that retrieve the data from different files for train and test and convert these into the correct format. 

    I've also made some tuning parameter and i reach results similar to the ones of the paper.I cannot post the image, but  the trend of the MAE index variing the number of iterations of the Engine system(like what is described in the paper) is the ollowing:

    [0.674858, 0.662203, 0.6413142, 0.6559569, 0.9494163, 1.583454, 1.771601, 1.7029443, 1.685406, 1.6210293333333332]

    As you can see, the minumum value is comparable with the paper results(i'm not using metadata). However the system have a strange behavior increasing the number of iteration. Differently from the paper case, The MAE index doesn't stabilize around the minimum, but instead it increase with the number of iteration untill it stabilize around a value very far away from the optimum. I really don't know how to explain this strange behaviour, and I suspecting some kind of bug. 

    Have someone of you got the same strange behavior in this or different problem? Does anyone know the reason of this trend??

    Thank you very much

    Best Regards


    Monday, May 27, 2013 3:04 PM


All replies

  • Try following the advice given in the earlier thread.
    • Marked as answer by MarcoASDF Monday, June 3, 2013 9:03 AM
    Tuesday, May 28, 2013 10:54 AM
  • Hi 

    Thanks for your reply. I can consider this topic solved, because thanks to your advice, i dont have the previous problem anymore. However I have a question about this problem. 

    I have solved the problem not only using the "trick" suggested in the earlier thread, but also changing a lot my code. As you can se in my first message, i had changed the problem creating a class with constructor and everithig required. Adding the sequential arrtribute to my code nothing change and the problem remain. However i have tried to add this two new line code to a version of the example more close to the original version where i not use a common class declaration and object instance, but i use the code

    Type tutorialClass = null;
    tutorialClass = typeof(ColdStart.ModelRS);
    object obj = Activator.CreateInstance(tutorialClass);
                MethodInfo mi = tutorialClass.GetMethod("Test");
                if (mi == null) return;
                mi.Invoke(obj, new object[] { numItr, numTraits, noiseVar, priorVar, trainPath, testPath, resDir });

    to call the train & test procedure. Have you got any ideas about the reason of this thing?? Why this work for the original formattation while it doesn't work in a classical formattation??

    Thank you in advance


    Tuesday, May 28, 2013 3:30 PM