I am having trouble stopping google crawling a few urls which cause errors.
I want to stop
- /project/123984932842/download/pdf
- /project/123984932842/download/zip
but allow
- /project/123984932842
- /project/123984932842/flat
I tried project/*/download/pdf but it doesn’t seem to work. Does anyone know what would?
Do you have a
/at the beginning of theDisallow:line?