Question

I need your help in finding better way in downloading a url using HttpWebResponse

I used next code

    HttpWebResponse myboot = HttpWebRequest.Create("http://www.wwenews.us/m1.php?id=441229").GetResponse() as HttpWebResponse;
    StreamReader myboot_content = new StreamReader(myboot.GetResponseStream(), Encoding.GetEncoding("windows-1256"));
    string temp_data = myboot_content.ReadToEnd();

but a problem says

The server committed a protocol violation. Section=ResponseHeader Detail=CR must be followed by LF

appears to me when trying parsing http://www.wwenews.us/m1.php?id=441229

please help me to download string of this site

note: test your solution code before present it as I had tested several solutions and no one solve the problem

Answer 1

Add a reference to System.Configuration to your project and add the following method.

public static bool SetUnsafeHeaderParsing()
    {
        Assembly oAssembly = Assembly.GetAssembly(typeof(System.Net.Configuration.SettingsSection));
        if (oAssembly != null)
        {
            Type oAssemblyType = oAssembly.GetType("System.Net.Configuration.SettingsSectionInternal");
            if (oAssemblyType != null)
            {

                object oInstance = oAssemblyType.InvokeMember("Section",
                  BindingFlags.Static | BindingFlags.GetProperty | BindingFlags.NonPublic, null, null, new object[] { });

                if (oInstance != null)
                {
                    FieldInfo objFeild = oAssemblyType.GetField("useUnsafeHeaderParsing", BindingFlags.NonPublic | BindingFlags.Instance);
                    if (objFeild != null)
                    {
                        objFeild.SetValue(oInstance, true);
                        return true;
                    }
                }
            }
        }
        return false;
    }

Call the method SetUnsafeHeaderParsing() before you use the HttpWebRequest.Create method call.

Its actually a problem with the server. The server is not following the HTTP specifications. However your .NET client by default would adhere to the specs and flags it as a potential problem.