Question

I have a url that I like to restrict the search engines from going to:

Is the following acceptable:

    User-agent: *
    Disallow: https://mysite.com/

or do I need to put something more like:

    User-agent: *
    Disallow: https://mysite.com/index.aspx

or would I just put:

    User-agent: *
    Disallow: /index.aspx

Answer 1

The last one is enough. You don’t need the whole URL, if you transfer you robots.txt to another website you have to change this file aswell, and that’s not what you want. But you do need to start with a / which means it’s the root of your domain.

Or you can do this:

Disallow ALL

User-agent: *
Disallow: /

Disallow 1 page

User-agent: *
Disallow: /index.php

Disallow 1 directory

User-agent: *
Disallow: /dirname/

Disallow 2 pages and 2 directories

User-agent: *
Disallow: /index.php
Disallow: /subpage.php
Disallow: /dirname/
Disallow: /otherdirname/

Allow 1 page (only index.php)

User-agent: *
Disallow: /~index.php