Home/ Questions/Q 6994181
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-27T19:51:34+00:00

I am trying to crawl a website using a java program. Until last night

  • 0

I am trying to crawl a website using a java program.
Until last night it was working perfect, but now the server returns error code 401.

HOWEVER, I can still see the pages that I want through my web browsers.
So, I don’t know what is wrong? If the server add my IP to black list, so why can I see the URLs through the web browsers? If not, what else can cause 401 error?

Two more points:
There is no username and password for this site and authentication is based on my IP.

Also, I tried to change my user agent, and now I get Error 503.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Likely it could be blocking based on the user agent or request frequency based on your ip address.

    • 0