Question

I use logistic regression with scipy.optimize.fmin_bfgs for minimizing the cost function. The cost function stays constant for my particular data set and BFGS does not converge, so I want to apply lasso+ridge regularization.

Now, I want to try out optimizing the cost for various values of the regularization parameters lambda1/2 in order to find the best combination:

for lambda1 in range(...):
    for lambda2 in range(..):
        scipy.optimize.fmin_bfgs(...) # Optimize cost with lambda1 and lambda2

The problem is that, because BFGS is not converging, it stays “forever” in the call for the first values of lambda1/2.

Is there a way to automatically stop fmin_bfgs after a while? The maxiter parameter does not help me, because I have 1000s of samples and a large number of features/sample, so it doesn’t even finish one such iteration in acceptable time.

In scipy 0.11, fmin_bfgs has a maxfun parameter — can one emulate this somehow in scipy 0.10?

EDIT: By popular demand, here are some relevant snippets of code:

The function computing the cost (the usual notations apply):

def computeCost(theta, X, y):
    h = sigmoid(X.dot(theta.T))
    J = y.T.dot(log(h)) + (1.0 - y.T).dot(log(1.0 - h))
    J_reg2 = theta[1:]**2
    J_reg1 = theta[1:]
    cost = (-1.0 / m) * (J.sum() + LAMBDA2 * J_reg2.sum() + LAMBDA1 * J_reg1.sum())
    return cost

Invoking the fmin_bfgs function:

initial_thetas = numpy.zeros((len(train_X[0]), 1))
myargs = (train_X, train_y)
theta = scipy.optimize.fmin_bfgs(computeCost, x0=initial_thetas, args=myargs)

Answer 1

Your problem isn’t the number of iterations. The reason why the lambda numbers aren’t changing is that the optimisation isn’t working.
Scipy should be working out those numbers for you rather than you supplying them through for loops.

Maybe if you could include more code it would be easier to see how to fix it.