Question

The website I work on hosts content that constantly gets scraped and posted elsewhere.

Is is possible to do URL rewrites so normal users and white-listed crawlers can view the website, but block access to unidentifiable browsers?

Answer 1

Yes, you can do that using URL Rewrite module (I’m using v2 .. but it should work with v1.x as well, although I have no v1.x around to test):

<system.webServer>
    <rewrite>
        <rules>
            <rule name="UserAgentRedirect" stopProcessing="true">
                <match url="^(.*)$" />
                <conditions>
                    <add input="{HTTP_USER_AGENT}" pattern="(iphone|ipod)" />
                </conditions>
                <action type="Rewrite" url="/special-page.aspx" />
            </rule>
        </rules>
    </rewrite>
</system.webServer>

With the above rule ALL requests from iPhone or iPad (or any other browser/app that has iphone or ipod in User Agent String) will be rewritten (internal redirect) to /special-page.aspx.