Question

I am teaching myself machine learning now.

One simple question regarding information gain.
How to calculate information gain from the pic’s data?
I do not understand it.

Could anyone explain how to get 0.992385 from the first row?

Thanks a lot!

Answer 1

Let g(x) = -x * log(x) /log(2)

The total number of people is 48842.

H(wealth, relation) = g[52/48842] + g[111/48842] + g[309/48842] + g[1093/48842]
+ g[1238/48842] + g[1276/48842] + g[1454/48842] + g[4816/48842] + g[7470/48842]
+ g[8846/48842] + g[10870/48842] + g[11307/48842] = 2.7835

H(wealth) = g(poor/total) + g(rich/total) = g[0.239282] + g[0.760718] = 0.793844

H(relation) = g(Husband/total) + g(Not_in_familly/total) + …
= g[0.0308341] + g[0.0477253] + g[0.10493] + g[0.155215] + g[0.257627] +
g[0.403669] = 2.15508

H(wealth | relation) = H(wealth, relation) – H(relation) = 2.7835 – 2.15508 = 0.628421

IG = H(wealth) – H(wealth|relation) = H(wealth) + H(relation) – H(wealth, relation)
= 0.165423

Here is source code written in Mathematica. If you feel you need to see the source code in another language, post a comment below with your preferred language. If I have time, I will type it up. – Cheers, Hans

Source Code in Mathematica

(* =================================================== *)

m = {{10870, 8846}, {11307, 1276}, {1454, 52}, {7470, 111}, 
     {4816, 309}, {1238, 1093}};

iTot = Total[ Flatten[m]];

h[x_] := -x * Log[2, x];

fHAll = Sum[  h[ m[[i, j]]/ iTot ], {i, 6}, {j, 2}] // N;

fHWealth = h[ Total[ m[[All, 1]]]/iTot] + h[ Total[ m[[All, 2]]]/iTot] // N ;

fHRelation = Sum[
     h[ Total[ m[[i ]]]/iTot] , {i, Length[m]}] // N;

fWealthGivenRelation = fHAll - fHRelation;

Print[" H(relation, wealth) = ", fHAll];
Print[" H(relation) = ", fHRelation];
Print[" H(wealth) = ", fHWealth];
Print[" H(wealth | relation) = ", fWealthGivenRelation];
Print[" IG = MI = ", fHWealth - fWealthGivenRelation, " = ", 
  fHWealth + fHRelation - fHAll];

(* ===================output ====================*)

 H(relation, wealth) = 2.7835
 H(relation) = 2.15508
 H(wealth) = 0.793844
 H(wealth | relation) = 0.628421
 IG = MI = 0.165423 = 0.165423

Oops, I didn’t answer you’re main question. Here’s the answer.

H(wealth | relation = husband)
= g(10870/(10870 + 8846)) + g(8846/(10870 + 8846))
= 0.992385