locked
Not convergence RRS feed

  • Question

  • Hi

    My question is in what condition an algorithm may not converge?

    Is it about the data? Or selection of priors?

    If I don't get convergence what should I do?

    Thanks

    Monday, April 11, 2016 9:31 PM

Answers

  • No, it doesn't fix the noise variable.  I meant the set of solutions is preserved.  If you think there is another solution it should be choosing, try initialization.
    • Marked as answer by RazinR Sunday, April 17, 2016 3:41 PM
    Sunday, April 17, 2016 6:28 AM
    Owner

All replies

  • The answer depends a lot on the model.  If you post runnable code for the model (data not required) then we can give advice on how to fix it.
    Tuesday, April 12, 2016 10:05 AM
    Owner
  • Hi 

    My code is:

    * I also added engine.Compiler.GivePriorityTo(typeof(GaussianProductOp_SHG09)) in my inference part, which did not help.

    Variable<double> noise; VariableArray<VariableArray<double>, double[][]> dataRows; VariableArray<double> outputData;

    // I try to find features (sparse) which are related in generating the output (a kind of regression) // indicesOfRelatedFeatures is a kind of prior knowledge

    public void CreateFeatureSelectionModel( int numberOfFeatures, int numberOfData, int[] indicesOfRelatedFeatures, double[] selectProbPrior ) { Variable<int> numNodes = Variable.Observed(numberOfFeatures); Variable<int> numberOfDataVar = Variable.Observed(numberOfData); Range featureRange = new Range(numberOfFeatures).Named("nodeRange"); Range dataRange = new Range(numberOfDataVar).Named("experimentRange"); Range wRange = new Range(indicesOfRelatedFeatures.Length).Named("wRange"); // VariableArray<VariableArray<double>, double[][]> dataRows;// this is defined outside of the function in main code dataRows = Variable.Array(Variable.Array<double>(featureRange), dataRange).Named("dataRows"); Gaussian w1MeanPrior = Gaussian.FromMeanAndPrecision(0, 1);//(1, 0.001); Gamma w1PrecisionPrior = Gamma.FromShapeAndScale(1, 99999); var rho = Variable.Array<double>(wRange).Named("rho"); rho.ObservedValue = selectProbPrior; selector = Variable.Array<bool>(wRange).Named("selector"); selector[wRange] = Variable.Bernoulli(rho[wRange]); w = Variable.Array<double>(wRange).Named("W"); using (Variable.ForEach(wRange)) { using (Variable.If(selector[wRange])) { w[wRange].SetTo(Variable.GaussianFromMeanAndPrecision(Variable.Random(w1MeanPrior), Variable.Random(w1PrecisionPrior))); } using (Variable.IfNot(selector[wRange])) { w[wRange].SetTo(Variable.GaussianFromMeanAndVariance(0, 0.0000001)); } } var wIndices = Variable.Array<int>(wRange).Named("wIndices"); wIndices.ObservedValue = indicesOfRelatedFeatures; // VariableArray<double> outputData; // this is defined outside of the function in main code outputData = Variable.Array<double>(dataRange).Named("outputData"); Gamma noisePrior = Gamma.FromShapeAndScale(1, 99999); noise = Variable.Random(noisePrior).Named("noise"); // make an array to seperate related features VariableArray<double> relatedFeatures = Variable.Array<double>(wRange).Named("ndoeSubarray"); using (ForEachBlock nodeBlock = Variable.ForEach(dataRange)) { VariableArray<double> weightSumArray = Variable.Array<double>(wRange); relatedFeatures = Variable.Subarray(dataRows[dataRange], wIndices); // dataRows[t - 1][experimentRange]; weightSumArray[wRange] = (w[wRange] * relatedFeatures[wRange]).Named("weightSumArray"); Variable<double> linearEffect = Variable.Sum(weightSumArray); outputData[dataRange] = Variable<double>.GaussianFromMeanAndPrecision(linearEffect, noise); }// dataRange.AddAttribute(new Sequential()); }


    Friday, April 15, 2016 8:22 AM
  • The oscillation seems caused by learning the noise at the same time as w.  You can fix it by adding a small amount of damping to the noise variable, like so:

    using (ForEachBlock nodeBlock = Variable.ForEach(dataRange)) {
    ...
    var noiseDamped = Variable<double>.Factor(Damp.Backward<double>, noise, 0.5); outputData[dataRange] = Variable<double>.GaussianFromMeanAndPrecision(linearEffect, noiseDamped);
    }


    Friday, April 15, 2016 12:37 PM
    Owner
  • Thank You

    That solved the convergence problem.

    But does this change reduce the functionality of the code?

    Since on the datasets it was working before, now I got poorer result.

    Friday, April 15, 2016 2:08 PM
  • This change should preserve the fixed points of the algorithm, however if there are multiple solutions it could change the one chosen.  To bias it toward one of the solutions, you can initialize some of the variables.
    Friday, April 15, 2016 2:11 PM
    Owner
  • I didn't get what you mean by fixed points. You mean it fixes noise variable (because the change is made to noise)? So I can fix it manually to the final value of noise precision and run the algorithm again?

    By the way, I checked it on several datasets with different options for rho (ranging from 0.05 to 0.95).

    On some it performs worst than the main code. Also for some the algorithm sets all the feature weights too high (100% overfitting). Should I increase the number of iterations? or should I change initialization of variables? 

    Friday, April 15, 2016 8:47 PM
  • No, it doesn't fix the noise variable.  I meant the set of solutions is preserved.  If you think there is another solution it should be choosing, try initialization.
    • Marked as answer by RazinR Sunday, April 17, 2016 3:41 PM
    Sunday, April 17, 2016 6:28 AM
    Owner