locked
Problem with MatrixMultiply and inference RRS feed

  • Question

  • Hi,

    I'm trying to make a Bayesian Group Factor Analysis model(very similar to CCA) based on the Bayesian PCA example. I've made a simple console app to load data, infer and write out the result and I can get the code running in debug mode from Visual Studio, but when running the program from windows, it doesn't infer. 

    I think the problem is the MatrixMultiply method:

    for (int i = 0; i < views; i++)
        vT[i] = Variable.MatrixMultiply(vZ, vW[i]);

    When I change the matrix multiplication to an explicit way, as suggested in this forum, I get an output, but the weights in vW explodes:

    using (Variable.ForEach(rM))
    {
         using (Variable.ForEach(rN))
         {
             using (Variable.ForEach(rD))
             {
                 var vProds = Variable.Array<double>(rK);
                 vProds[rK] = vZ[rN, rK] * vW[rM][rK, rD];
                 vT[rM][rN, rD] = Variable.Sum(vProds);
             }
         }
    }

    So, any reason why MatrixMultiply won't work in a compiled app?

    The rest of the model:

    // Inference engine
            public InferenceEngine engine = null;
    
            // Model variables
            public Variable<int> vN; // Number of data points
            public Variable<int> vD; // Dimensionality of data
            public Variable<int> vK; // Dimensionality of latent variables
            public Variable<int> vM; // Number of datasets
            public VariableArray<VariableArray2D<double>, double[][,]> vData;
    
            public VariableArray2D<double> vZ;
            public VariableArray<VariableArray2D<double>, double[][,]> vW;
            public VariableArray<VariableArray2D<double>, double[][,]> vT;
    
            public VariableArray<double> vPi;
            public VariableArray<VariableArray<double>, double[][]> vAlpha;
    
            // Priors - these are declared as distribution variables
            // so that we can set them at run-time.
            public Variable<Gamma> priorAlpha;
            public Variable<Gamma> priorPi;
    
            // Ranges of vN, vD, vK, vM
            public Range rN;
            public Range rD;
            public Range rK;
            public Range rM;
    
            /// <summary>
            /// Model constructor
            /// </summary>
            public BIBFA_Model(int views)
            {
                // The various dimensions will be set externally...
                vN = Variable.New<int>().Named("NumObs");
                vD = Variable.New<int>().Named("NumFeats");
                vK = Variable.New<int>().Named("MaxComponents");
                vM = Variable.New<int>().Named("NumViews");
    
                rN = new Range(vN).Named("N");
                rD = new Range(vD).Named("D");
                rK = new Range(vK).Named("K");
                rM = new Range(vM).Named("M");
    
                // ... as will the data
                vData = Variable.Array(Variable.Array<double>(rN, rD), rM).Named("data");
    
                // ... and the priors
                priorAlpha = Variable.New<Gamma>().Named("PriorAlpha");
                priorPi = Variable.New<Gamma>().Named("PriorPi");
    
                // Mixing matrix and ARD prior
                vAlpha = Variable.Array(Variable.Array<double>(rK), rM).Named("Alpha");
                vW = Variable.Array(Variable.Array<double>(rK, rD), rM).Named("W");
                vAlpha[rM][rK] = Variable.Random<double, Gamma>(priorAlpha).ForEach(rM, rK);
                vW[rM][rK, rD] = Variable.GaussianFromMeanAndPrecision(0, vAlpha[rM][rK]).ForEach(rD);
    
                // Data mean
                vT = Variable.Array(Variable.Array<double>(rN, rD), rM).Named("T");
    
                // Latent variables are drawn from a standard Gaussian
                vZ = Variable.Array<double>(rN, rK).Named("Z");
                vZ[rN, rK] = Variable.GaussianFromMeanAndPrecision(0.0, 1.0).ForEach(rN, rK);
    
                // Matrix multiplication
                //for (int i = 0; i < views; i++)
                //    vT[i] = Variable.MatrixMultiply(vZ, vW[i]);
    
                 //Explicit Matrix multiply
                using (Variable.ForEach(rM))
                {
                    using (Variable.ForEach(rN))
                    {
                        using (Variable.ForEach(rD))
                        {
                            var vProds = Variable.Array<double>(rK);
                            vProds[rK] = vZ[rN, rK] * vW[rM][rK, rD];
                            vT[rM][rN, rD] = Variable.Sum(vProds);
                        }
                    }
                }
    
                // Add observation noise. Diagonal for each dataset
                vPi = Variable.Array<double>(rM).Named("pi");
                vPi[rM] = Variable.Random<double, Gamma>(priorPi).ForEach(rM);
    
                // Data
                vData[rM][rN, rD] = Variable.GaussianFromMeanAndPrecision(vT[rM][rN, rD], vPi[rM]);
    
                // Inference engine
                engine = new InferenceEngine(new VariationalMessagePassing());
                //engine = new InferenceEngine(new ExpectationPropagation());
                return;
            }

    Thursday, August 1, 2013 6:28 PM

Answers

  • This is caused by a known bug where InitialiseTo is sometimes ignored.  This will be fixed in the next version.  Meanwhile you can workaround by also initializing vZ, i.e.

                model.vZ.InitialiseTo(DataUtils.randomGaussianMatrix(model.vN.ObservedValue, model.vK.ObservedValue));

    This type of model is very sensitive to initialization, so it is not surprising that you can get different results just by changing the model definition slightly or the random seed.

    • Marked as answer by Simon Kamronn Thursday, August 8, 2013 4:09 PM
    Monday, August 5, 2013 1:56 PM
    Owner

All replies

  • As regards MatrixMultiply, you are indexing by I rather than by the range. Just have the line:

    vT[rM] = Variable.MatrixMultiply(vZ, vW[rM]);

    You can observe vM before running (vm.ObservedValue = ...). You don't need to pass 'views' to the constructor.

    However, Matrix multiply, and the explicit version should be equivalent. When you say 'explodes', can you be more explicit? Can you send a small data set and program that calls into the model that shows this? (If you cannot post here, can you send to infersup@microsoft.com.

    Thanks

    John G.

    Friday, August 2, 2013 10:43 AM
    Owner
  • I did try using the range at first, by got an error:

    Error 0: This model is not supported with VariationalMessagePassing due to Factor.MatrixMultiply(double[,] matrixMultiply, double[,] A, double[,] B). Try using a different algorithm or expressing the model differently in

    Factor.MatrixMultiply(Z_rep[M], W_use[M])

    I thought I could get around that with the loop, but apparently only when executing the program from Visual Studio. Regarding the exploding weights that was due to a bad prior on the data precision

    priorPi.ObservedValue = Gamma.FromShapeAndScale(1e-3, 1e3);

    By using 2 and 2 as inputs, it behaves. However, it still doesn't work as well as when using MatrixMultiply, which is strange since the result should be the same. I've emailed the code if you want to have a look. 

    Simon

    Friday, August 2, 2013 12:59 PM
  • Thanks for reporting the issue with MatrixMultiply without a loop.  We'll work on that.  Your workaround of using a loop should work.  I can run your original version with MatrixMultiply in a loop, both from Visual Studio and the command line.  What error do you get when running from the command line?

    Monday, August 5, 2013 12:22 PM
    Owner
  • I don't get an error, there's just no result from the inference. Do you have any idea why the explicit multiplication results in a different inference compared to MatrixMultiply? If you use the priorPi mentioned in my previous post you'll get quite different results.

    I tried posting a screenshot but wasn't allowed so I'll email it instead. 

    Monday, August 5, 2013 1:21 PM
  • This is caused by a known bug where InitialiseTo is sometimes ignored.  This will be fixed in the next version.  Meanwhile you can workaround by also initializing vZ, i.e.

                model.vZ.InitialiseTo(DataUtils.randomGaussianMatrix(model.vN.ObservedValue, model.vK.ObservedValue));

    This type of model is very sensitive to initialization, so it is not surprising that you can get different results just by changing the model definition slightly or the random seed.

    • Marked as answer by Simon Kamronn Thursday, August 8, 2013 4:09 PM
    Monday, August 5, 2013 1:56 PM
    Owner
  • I wasn't aware of the bug, but initializing worked well. Thanks a lot.
    Monday, August 5, 2013 3:10 PM