Question

I have a Java AppEngine web application. I noticed in the dashboard in admin console of appengine that there is a URI error encountered everyday for /robots.txt.

How to remove the error?

Answer 1

robots.txt is a magic URL used by search engine and other robots before processing your site. See wikipedia for more details.

The best way of dealing with this error on GAE is to put a robots.txt file, and define it as a static file in your app.yaml for gae/python:

- url: /(robots\.txt)
  static_files: \1
  upload: (robots\.txt)

And in appengine-web.xml of gae/java:

<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0"
                   xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'           xsi:schemaLocation='http://kenai.com/projects/nbappengine/downloads/download/schema/appengine-web.xsd appengine-web.xsd'>
    ....
    <static-files>
        <include path="/favicon.ico" />
        <include path="/robots.txt" />
        <include path="/img/**.png" />
        <include path="/img/**.gif" />
        <include path="/css/**.css" />
</static-files>

Of course, you may just as well ignore the errors, they don’t matter to anyone but yourself (no human is encountering the error).