Sign Up

Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.

Have an account? Sign In

Have an account? Sign In Now

Sign In

Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.

Sign Up Here

Forgot Password?

Don't have account, Sign Up Here

Forgot Password

Lost your password? Please enter your email address. You will receive a link and will create a new password via email.

Have an account? Sign In Now

You must login to ask a question.

Forgot Password?

Need An Account, Sign Up Here

Please briefly explain why you feel this question should be reported.

Please briefly explain why you feel this answer should be reported.

Please briefly explain why you feel this user should be reported.

Sign InSign Up

The Archive Base

The Archive Base Logo The Archive Base Logo

The Archive Base Navigation

  • Home
  • SEARCH
  • About Us
  • Blog
  • Contact Us
Search
Ask A Question

Mobile menu

Close
Ask a Question
  • Home
  • Add group
  • Groups page
  • Feed
  • User Profile
  • Communities
  • Questions
    • New Questions
    • Trending Questions
    • Must read Questions
    • Hot Questions
  • Polls
  • Tags
  • Badges
  • Buy Points
  • Users
  • Help
  • Buy Theme
  • SEARCH
Home/ Questions/Q 809835
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Editorial Team
Asked: May 15, 20262026-05-15T00:47:28+00:00 2026-05-15T00:47:28+00:00

What does robots.txt file do in PHP project?

  • 0

What does robots.txt file do in PHP project?

  • 1 1 Answer
  • 0 Views
  • 0 Followers
  • 0
Share
  • Facebook
  • Report

Leave an answer
Cancel reply

You must login to add an answer.

Forgot Password?

Need An Account, Sign Up Here

1 Answer

  • Voted
  • Oldest
  • Recent
  • Random
  1. Editorial Team
    Editorial Team
    2026-05-15T00:47:29+00:00Added an answer on May 15, 2026 at 12:47 am

    Not really PHP, see:

    Introduction to “robots.txt”

    “Robots.txt” is a regular text file
    that through its name, has special
    meaning to the majority of “honorable”
    robots on the web. By defining a few
    rules in this text file, you can
    instruct robots to not crawl and index
    certain files, directories within your
    site, or at all. For example, you may
    not want Google to crawl the /images
    directory of your site, as it’s both
    meaningless to you and a waste of your
    site’s bandwidth. “Robots.txt” lets
    you tell Google just that.

    • 0
    • Reply
    • Share
      Share
      • Share on Facebook
      • Share on Twitter
      • Share on LinkedIn
      • Share on WhatsApp
      • Report

Sidebar

Related Questions

I'm creating a robots.txt file for my website, but looking through my project structure,
When using a Robots.txt file, does the user agent string have to be exactly
I'm testing it and Nokogiri does not seem to respect Robots.txt file. Is there
For some reason, I want to serve my robots.txt via a PHP script. I
I'm looking at a robots.txt file of a site I would like to do
Does Facebook's webcrawling bots respect the Crawl-delay: directive in robots.txt files?
How to distinguish robots from normal user? How does SO do this job? Currently
For instance, if I have robots.txt declared as a static resource, and assuming that
I want to rewrite my robots.txt for all domains but main one (I have
Just because software is automated doesn't mean it will abide by your robots.txt. What

Explore

  • Home
  • Add group
  • Groups page
  • Communities
  • Questions
    • New Questions
    • Trending Questions
    • Must read Questions
    • Hot Questions
  • Polls
  • Tags
  • Badges
  • Users
  • Help
  • SEARCH

Footer

© 2021 The Archive Base. All Rights Reserved
With Love by The Archive Base

Insert/edit link

Enter the destination URL

Or link to existing content

    No search term specified. Showing recent items. Search or use up and down arrow keys to select an item.