Answered by:
convergence condition of LBFGS/Quasi Newton
Question

I found the LBFGS optimizer (MicrosoftResearch.Infer.Maths.LBFGS) is a great tool. In practice, however, it seems the convergence condition/eps of this class is rather rigorous. While I do not need so rigorous convergence condition and I want the optimizer to stop quickly, I am not sureif I can set the stopping/convergence condition, or eps by myself?
Thanks!
Friday, August 8, 2014 6:52 AM
Answers

There are several options/considerations for convergence. You can both set an upper limit for the number of iterations, and set an epsilon, and it will stop with whichever condition gets met first. As regards the epsilon, you can choose whether to use either the objective (F(k+1)F(k) <= epsilon*max{F(k),F(k+1),1}) or the gradient (grad F/sqrt(dimension) <= epsilon) to determine convergence.
Here is what the code looks like for these options:
LBFGS optimizer = new LBFGS(5);
optimizer.Epsilon = 0.001;
optimizer.convergenceCriteria = BFGS.ConvergenceCriteria.Gradient;
// optimizer.convergenceCriteria = BFGS.ConvergenceCriteria.Objective;
optimizer.MaximumIterations = 50;John
 Proposed as answer by John GuiverMicrosoft employee, Owner Friday, August 8, 2014 4:14 PM
 Marked as answer by John GuiverMicrosoft employee, Owner Monday, August 11, 2014 7:51 AM
Friday, August 8, 2014 7:16 AMOwner
All replies

There are several options/considerations for convergence. You can both set an upper limit for the number of iterations, and set an epsilon, and it will stop with whichever condition gets met first. As regards the epsilon, you can choose whether to use either the objective (F(k+1)F(k) <= epsilon*max{F(k),F(k+1),1}) or the gradient (grad F/sqrt(dimension) <= epsilon) to determine convergence.
Here is what the code looks like for these options:
LBFGS optimizer = new LBFGS(5);
optimizer.Epsilon = 0.001;
optimizer.convergenceCriteria = BFGS.ConvergenceCriteria.Gradient;
// optimizer.convergenceCriteria = BFGS.ConvergenceCriteria.Objective;
optimizer.MaximumIterations = 50;John
 Proposed as answer by John GuiverMicrosoft employee, Owner Friday, August 8, 2014 4:14 PM
 Marked as answer by John GuiverMicrosoft employee, Owner Monday, August 11, 2014 7:51 AM
Friday, August 8, 2014 7:16 AMOwner 
Thanks a lot, John!
 Marked as answer by John GuiverMicrosoft employee, Owner Monday, August 11, 2014 7:51 AM
 Unmarked as answer by John GuiverMicrosoft employee, Owner Monday, August 11, 2014 7:51 AM
Friday, August 8, 2014 8:22 PM