I am using Google Webfont’s with my website. Should i let the fonts stay hosted on Google’s Server and reference the required stylesheet(s) from my <head> or should I download the font and upload them on my server and use @font-face.
Which of the methods will out more load on my server and make my site slower.
The Size of the font is 40KB. But, when I use a Pingdom’s Pagespeed tool, the downloaded font-size is shown in a few bytes.
But, When the font is hosted on my server, I think the full 40KB file is downloaded by the client.
Why does this happen ?

Keep them on Google’s servers.
The reason why you’re seeing the small file size from Google is because your browser has already downloaded and cached the file. When you uploaded to your own server, your browser had to download it once again as it considers it a new source.
This fact is also one of the reasons you should reference the fonts from Google, as visitors to your site may have already downloaded them from another site that uses the same font from Google.
Hosting them yourself would naturally put additional load on your servers. Having Google host them lets them deal with that load.
Moreover, by referencing from Google you are letting Google handle the maintenance, such as updates to the fonts as well as the syntax that serves the fonts themselves.