So here is the solution to an annoying one. Have you ever made an asp.net website and tried to make custom error pages? So here is the issue. I could not, for the longest time, figure out how to make my custom error pages actually show up for the user. Seriously, everything worked fantastic, until my HTTP response was anything other than 200. My solution was to return a "soft" 404 page. Which web crawlers really hate.
Soft Error - When you catch the error on the server, and return an error page with a success error code. For example a user browses to a link on your site which does not exist, causing a 404. You serve up a 404 page, but with a "200 OK" response code. From a user's perspective this is fine, but bots have no idea what is going on.
To be entirely honest, I made up most of that definition. Wikipedia refers to it as a "phony 404 error". Realistically you can do the same thing for any type of error on the server side. Actually, you can throw response codes out the window and return whatever crap you want all the time. Do it... Return a 500 for every single page. Users will never notice, but bots will lose their minds. (Actually you'll just never make it anywhere near the top on any search engine).
Ok So let's get back to the point here. Instead of a "Soft Error", you should always return the proper error code. IIS doesn't like to make this easy for you by default, so here is what you need to configure to make sure everything works properly.
<system.webServer> <httpErrors existingResponse="PassThrough"></httpErrors> </system.webServer>
Just stick this in the System.webServer section in your web.config. "existingResponse=PassThrough" pretty much means that you are going to take matters into your own hands when it comes to errors. If an error response comes down the pipe (aka: http code >= 400) then IIS will leave the message untouched if a response exists.
Hope this helps. My ones of users have been consistently complaining about poor error handling issues, so I dedicated my night (after this glorious Donald vs. Hillary debate) to fixing the issue. Thanks for your feedback Daniel.