Question

I’m trying to minimize a function with lots of parameters (a little over 7000) using fmin_bfgs() or fmin_l_bfgs_b(). When I enter the command

 opt_pars = fmin_l_bfgs_b(obj_f, pars, approx_grad=1)

(where obj_f is the function I’m trying to minimize and pars is the vector of initial parameters) the function just runs forever until python tells me that it has to terminate the program. There is never any output. I tried adding the argument maxfunc = 2 to see if it was getting anywhere at all and the same thing happened (ran forever then python terminated the program).

I’m just trying to figure out what could be going wrong with the function. It seems like maybe it’s getting caught in a while loop or something. Has anyone encountered this problem? If not, I could also use some general debugging help here (as I’m relatively new to Python) on how to monitor what the function is doing.

And finally, maybe someone can recommend a different function or package for the task I’m attempting. I’m trying to fit a lasso regularized Poisson regression to sparse data with about 12 million observations of 7000 variables.

PS Sorry for not including the -log likelihood function I’m trying to minimize, but it would be completely uninterpretable.

Thanks a lot for any help!

Zach

Answer 1

Since you don’t provide gradients to fmin_bfgs and fmin_l_bfgs_b, your objective function is evaluated len(x) > 7000 times each time the gradient is needed. If the objective function is slow to evaluate, that will add up.

The maxfun option doesn’t apparently count the gradient estimation, so it’s possible that it’s actually not an infinite loop, just that it takes a very long time.

What do you mean by “python tells me that it has to terminate the program”?

Please in any case try to provide a reproducible test case here. It doesn’t matter if the objective function is incomprehensible — what is important is that people interested can reproduce the condition you encounter.

I don’t see infinite loops problem on my system even for 7000 parameters. However, the function evaluation count was about 200000 for a simple 7000-parameter problem with l_bfgs_b and no gradient provided. Profile your code to see what such evaluation counts would mean for you. With gradient provided, it was 35 (+ 35 times the gradient). Providing a gradient may then help. (If the function is complicated, automatic differentiation may still work — there are libraries for that in Python.)

Other optimization libraries for Python, see: http://scipy.org/Topical_Software (can’t say which are the best ones, though — ipopt or coin-or could be worth a try)

For reference: the L-BFGS-B implementation in Scipy is this one (and is written by guys who are supposed to know what they are doing):
http://users.eecs.northwestern.edu/~nocedal/lbfgsb.html

***

You can debug what is going on e.g. by using Python debugger pdb, python -m pdb your_script.py. Or just by inserting print statements inside it.

Also try to Google “debug python” and “profile python” 😉