Question

I have a website (Django) on a linux server, but Google isn’t finding the site at all. I know that i don’t have a robots.txt file on the server. Can someone tell me how to create one, what to write inside and where to place it? That would be a great help!

Answer 1

robot txt is not for google find your site. i think you must register your site to google and also add sitemap.xml

• Webmaster Tools – Crawl URL ->
  https://www.google.com/webmasters/tools/submit-url?continue=/addurl&pli=1

also see this for robot.txt

• Three ways to add a robots.txt to your Django project | fredericiana
  -> http://fredericiana.com/2010/06/09/three-ways-to-add-a-robots-txt-to-your-django-project/

what is robot.txt

It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page (one for viewing in the browser and one for printing), you’d rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty. Also, if you happen to have sensitive data on your site that you do not want the world to see, you will also prefer that search engines do not index these pages (although in this case the only sure way for not indexing sensitive data is to keep it offline on a separate machine). Additionally, if you want to save some bandwidth by excluding images, stylesheets and javascript from indexing, you also need a way to tell spiders to keep away from these items.

One way to tell search engines which files and folders on your Web site to avoid is with the use of the Robots metatag. But since not all search engines read metatags, the Robots matatag can simply go unnoticed. A better way to inform search engines about your will is to use a robots.txt file.

from What is Robots.txt -> http://www.webconfs.com/what-is-robots-txt-article-12.php