Question

I have a completely blank “.aspx” file residing on an IIS6 web server. The server has .net 3.5 installed.

When executed, this file generates an “Illegal characters in path” error. A full stack trace of this error is available here.

The problem can be temporarily cured by IISReset. However, once any site on this server starts exhibiting this problem, every hit to every .aspx page on that site will cause the same issue (including my completely blank example file).

What might be causing this, and how do I fix it?

Edit: I’ve added a bounty in the hopes of getting some more help with this before we open a support case with Microsoft. I have seen the knowledge base article and blog post that Adam Hughes pointed out, but that information by itself doesn’t offer a complete solution. The blog is written for people who have knowledge and skill that I lack, but apparently I need to pick up. I need help understanding some of the connections and reasoning going on here.

I am specifically looking for help with any of these questions, whichever ones will lead me to a functioning system:

1. The knowledge base article’s workaround has me selecting a CGI program executable in step six. What am I supposed to select here? Is the debugging process described in the blog supposed to lead me to discovering which program to choose for this?

2. The blog post talks about a means of debugging this, but it assumes I have knowledge that I don’t have (yet). Although I have deduced that I will need ADPlus, I’ve never used this tool, or otherwise needed to inspect a memory dump before. I am not sure what to do when the author says this:

   With time pressing on a not a lot to
   lose I went for the live debug
   approach. In this way, thanks to the
   problem being readily reproducible at
   this time, I soon got the needed
   memory dump.

   How do I use ADPlus (or any other tool) to generate the memory dump, and how do I make sure the memory dump contains the information I need to debug this? Will “adplus -crash -iis” do what I want?

3. The issue is happening on a production server. While I can reboot this server if I really have to, I need to avoid being disruptive if possible. Is it a safe and reasonable idea to generate a memory dump on a live server? Will the server’s other functions be affected by this, and if so, what should I look out for?

4. Does it sound like I am on the right track? What other avenues should I be looking at to solve this?

Edit: Vinay, here is the information you requested involving ISAPI Filters and wildcard mappings:

_{(source: photobooks.com)}

_{(source: photobooks.com)}

The full path for the file associated with .aspx files is C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll.

We have a mirror of this server on our local network. I was able to use that to search through all instances of web.config, and I did not find anything unusual. In fact, it looks like we don’t ever use this section of web.config, as all of the entries were identical to this:

<httpHandlers>
    <remove verb="*" path="*.asmx"/>
    <add verb="*" path="*.asmx" validate="false" type="..." />
    <add verb="*" path="*_AppService.axd" validate="false" type="..."/>
    <add verb="GET,HEAD" path="ScriptResource.axd" validate="false" type="..." />
</httpHandlers>

Looking at this, it appears that I should be suspicious of ISAPI_Rewrite3, which we are currently using to support “friendly” URLs on some ASP Classic sites. Is this your diagnosis as well? Am I right in thinking that “fixing” this problem will mean replacing ISAPI_Rewrite3?

Edit: I have removed the ASP.NET entry from the ISAPI filter list. The full path for this was C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\aspnet_filter.dll. There are no obvious ill-effects from having removed this. MSDN states that it is used to help maintain a cookieless session state, which is not vital for this server anyway.

It has been several hours now and the problem has not recurred since removing the ASP.NET filter. Unfortunately, I do not have a means to reproduce the problem, other than waiting for it to happen again. I’m going to let it sit until tomorrow evening. If the problem recurs, I will remove the ISAPI_Rewrite3 filter as well and try again (thankfully, this filter’s functionality is not widely used by our clients). Either way, I will come back and report my results.

I have checked the <httpModules> section as requested. All such entries across all web.config files on this server are identical to this example:

<httpModules>
    <add name="ScriptModule"
         type="System.Web.Handlers.ScriptModule,
               System.Web.Extensions,
               Version=3.5.0.0,
               Culture=neutral,
               PublicKeyToken=31BF3856AD364E35"/>
</httpModules>

Thank you for your assistance!

Answer 1

I’m not sure you need to reach for ADPlus just yet – what he found using ADPlus, you already know – namely that the most likely culprit is something which is changing the current directory.

A key point seems to be that IISReset cures the problem for a time, but the problem comes back after some point, as soon as something happens on one of the sites on the server. By “sites” here, I assume you mean ASP.NET application – please confirm if this isn’t the case.

The act that multiple sites are exhibiting the problem indicates that the problem could well be an ISAPI filter or extension, or an .EXE file which is mapped to a wildcard mapping on one or more specific ASP.NET applications.

So, fire up IIS Manager, and update your question with information on ISAPI filters and wildcard mappings. The ISAPI filter information will be in the “ISAPI Filters” tab on the Properties dialog box for the web site (under “Web Sites” in the tree view in IIS Manager). The wildcard mappings will be in the “Application Configuration” dialog which you invoke from the “Configuration…” button on the Properties dialog for your specific ASP.NET application.

As well as wildcard mappings, see exactly what mappings are registered against .aspx extensions, and report those in your update.

The problem could also be in HttpHandlers installed in individual ASP.NET web sites – so look in the <httphandler> section in Web.config to see if a handler is defined with something like

<add verb="*" path="*.aspx" type="..." />

This may be on an application which is other than the one you’re working with. To see if you can track down which one, check the server logs after an IISReset to see which is the first entry to return a 5xx HTTP status code (indicating an internal server error) and look at the request which triggered it. The application which handled it may be the one to focus on.

All the things I’ve suggested should be safe to do on a production server 🙂 I’m not suggesting you actually do an IISReset, just check the logs immediately after the last one you did.

Update: I’ve seen the updated information you posted. It might be ISAPI_Rewrite3 or not; I’ve certainly used it before without any problems. However, I’m surprised to see the ASP.NET_2.0.xxx in the ISAPI filters list – I don’t have it in mine (I’m running Windows Server 2003). So, I would (when time permits – you may have to schedule an outage) temporarily remove those two ISAPI filters from the list (in such a way that you can easily restore them again) and see if that makes a difference. If it seems to have gotten worse, add the ASP.NET filter back in (what’s the exact path of that entry?) and leave out ISAPI_Rewrite3, then try again.

The reason I’m surprised to see an ASP.NET DLL in the ISAPI filters list is that it’s normally configured using the mappings on a per-application basis (where the applications inherit a default configuration). It normally works as an ISAPI extension, not an ISAPI filter.

Can you also please check the <httpmodules> sections in the same way as for <httphandlers>?

It may be necessary to run aspnet_regiis again, which initialises the ASP.NET system for IIS; This is, however a big step that could break lots of things. If possible, copy the entire server to a spare machine on a separate test or dev network and do it there. Can you use your local mirror for this kind of testing?