Question

In most of classifications (e.g. logistic / linear regression) the bias term is ignored while regularizing. Will we get better classification if we don’t regularize the bias term?

Answer 1

Example:

Y = aX + b

Regularization is based on the idea that overfitting on Y is caused by a being "overly specific", so to speak, which usually manifests itself by large values of a‘s elements.

b merely offsets the relationship and its scale therefore is far less important to this problem. Moreover, in case a large offset is needed for whatever reason, regularizing it will prevent finding the correct relationship.

So the answer lies in this: in Y = aX + b, a is multiplied with the explanatory/independent variable, b is added to it.