Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I have the following URL format:
example.com/page/m6aoeh/embed
I want to block any bots from indexing any page with the url /page/*/embed
I’m assuming there’s some way to do wildcard URL block, but just not sure how to do it in this particular case.
If you simply want to indicate to bots that everything under the
/page/folder should not be indexed by bots, use the robots.txt exclusion standard. Create a/robots.txtthat looks like this:If you want bots to index some pages under the
/pagefolder but not others, you’ll need to add a<meta>tag in the<head>to each page you want to exclude.