Question

In linear or logistic regression if we find a hypothesis function which fits the training set perfectly then it should be a good thing because in that case we have used 100 % of the information given to predict new information.
While it is called to be overfitting and said to be bad thing.
By making the hypothesis function simpler we may be actually increasing the noise instead of decreasing it.
Why is it so?

Answer 1

Overfitting occurs when you try “too hard” to make the examples in the training set fit the classification rule.

It is considered bad thing for 2 reasons main reasons:

1. The data might have noise. Trying too hard to classify 100% of the examples correctly, will make the noise count, and give you a bad rule while ignoring this noise – would usually be much better.
2. Remember that the classified training set is just a sample of the real data. This solution is usually more complex than what you would have got if you tolerated a few wrongly classified samples. According to Occam’s Razor, you should prefer the simpler solution, so ignoring some of the samples, will be better,

Example:

According to Occam’s razor, you should tolerate the misclassified sample, and assume it is noise or insignificant, and adopt the simple solution (green line) in this data set: