Question

Somehow a virtual robots.txt is blocking the site http://www.domda.se from google search. I don’t want that to happen.

The site is made with WordPress and open for search engines in the Integrity settings (of course 😉

I have tried:

• Turning the privacy setting/search engine allowance off and on again.

• Turning all the plugins off.

• Adding a totally allowing robots.txt to the root directory.

• Using a robot plugin to write what should be in robots.txt

• Change theme to Twentyeleven and back again

• Resetting the plugin folder with ftp

Now I wonder if someone can help me because my php knowledge so low it makes me blush. The thing I think about now is trying to solve the problem by looking into functions.php.

The robot doing code in wp-includes/functions.php says:

function do_robots() {
    header( 'Content-Type: text/plain; charset=utf-8' );

    do_action( 'do_robotstxt' );

    $output = "User-agent: *\n";
    $public = get_option( 'blog_public' );
    if ( '0' == $public ) {
        $output .= "Disallow: /\n";
    } else {
        $site_url = parse_url( site_url() );
        $path = ( !empty( $site_url['path'] ) ) ? $site_url['path'] : '';
        $output .= "Disallow: $path/wp-admin/\n";
        $output .= "Disallow: $path/wp-includes/\n";
    }

    echo apply_filters('robots_txt', $output, $public);
}

Maybe that code is totally alright. And me adding a real robots.txt would’ve overruled the virtual one, but I really don’t know what to do.

I hope someone can help.

Answer 1

I think you need to look first at whether the opening statement is entirely true.

You can view your robots.txt directly in a web browser.

If you visit http://www.domda.se/robots.txt, you get

User-agent: *

Disallow: /wpsystem/wp-admin/

Disallow: /wpsystem/wp-includes/

Which is blocking all robots from wpsystem files.

It is not blocking Google from the main site.

However, a Google search for site:http://www.domda.se does pull up

A description for this result is not available because of this site’s robots.txt – learn more

As you have made many changes, it seems likely that you used to have a robots.txt that blocked Google and now you don’t.

The whole point of robots.txt is that it is information for webcrawlers. These check sites regularly, and check some sites more often than others. It may be some time before your site is revisited by Google’s robot, and a further period of time before pages crawled on that visit show up in Google search.

You should use Google Webmaster Tools (it is free) to shine more light on this subject and to give Google a nudge.