Question

I am creating a web robot. Usually the http tools returns quite a few information and some of these are readonly (e.g. Connect: keep-alive). How to know which ones are required?

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: ISO-8859-9,utf-8;q=0.7,*;q=0.3
Accept-Encoding: gzip,deflate,sdch
Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control: max-age=0
Content-Length: 269
Content-Type: application/x-www-form-urlencoded
Host: closure-compiler.appspot.com
Origin: null
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko)       Chrome/14.0.794.0 Safari/535.1

Usually the code looks like the following. Someone pointed out that the following code missed to set Content-Type?

HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://closure-compiler.appspot.com/compile");
req.Connection = "keep-alive";
req.Headers.Add("Cache-Control", "max-age=0");
req.Headers.Add("Origin","null");
req.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.794.0 Safari/535.1";
req.ContentType = "application/x-www-form-urlencoded";
req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
req.Headers.Add("Accept-Encoding", "gzip,deflate,sdch");
req.Headers.Add("Accept-Language", "tr-TR,tr;q=0.8,en-US;q=0.6,en;q=0.4");
req.Headers.Add("Accept-Charset", " ISO-8859-9,utf-8;q=0.7,*;q=0.3");
req.Method = "POST";

Stream reqStr = req.GetRequestStream(); 

Answer 1

No headers are required for general requests. Particular resources may require different headers. The right way is to ask owner of the resource what headers are needed. But if you want to cheat in some sort of game/forum you will have to figure out headers and other values yourself.