This is a somewhat subjective question, but I’d like to hear the pros/cons for doing this. I manage an open source project called Quick and Dirty Feed Parser and the objective of the project is to make it as seamless as possible to consume RSS and Atom feeds in .NET.
One of the issues I ran into fairly early on in the development of the project was that some of the feeds I was using as test cases (namely the Hacker News RSS feed) used improperly formatted HTTP headers, and the HttpWebRequest class in .NET 1.1 and up promptly throws an “unsafe header” exception whenever you receive one of these headers in a GET request.
My issue is thus – I can enable the “useUnsafeHeader” configuration option programmatically, but it does it across ALL HttpWebRequests in that application’s context. I have users who’ve complained about QD Feed Parser being unable to consume valid feeds, and this header issue is why.
Right now I have my library set up in such a way that developers who use it have to enable unsafe header parsing themselves, although most of them aren’t aware that this is the problem and it creates a support overhead for me.
I can simply have Quick and Dirty Feed Parser enable unsafe header parsing by default and force security-concious users to disable it, but I don’t want to open up users who don’t know any better to security attacks either. What’s the best option here?
“Unsafe” is a bit extreme here; I would have named this setting differently. The problem comes up when ill-behaved servers emit headers which don’t follow the HTTP RFC exactly. For example the RFC says that CR characters must be followed by an LF character, so if there’s no LF you’ll get an execption unless you allow “unsafe” headers.
In practice, many HTTP clients ignore these minor violations in order to talk to as many servers as possible. That’s why your browser or RSS reader never complains about “unsafe” headers. Even if headers are bogus, the .NET client libraries are robust enough that you won’t, for example, crash your server if a malious attacker omits a linefeed. 🙂 So there’s not really a big safety issue here, unless (for example) you do dumb things with HTTP header names like emit them directly into your HTML (which might allow an attacker to inject an XSS attack into your HTML).
So, as long as you treat HTTP headers as if they’re just as untrustworthy as any other user-submitted data that comes into your application (like query strings, POST data, etc.), then you should be OK allowing “unsafe” headers in your app.