Home/ Questions/Q 9194311
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-17T21:22:53+00:00

When doing a scrape of a site, which would be preferable: using curl, or

  • 0

When doing a scrape of a site, which would be preferable: using curl, or using Python’s requests library?

I originally planned to use requests and explicitly specify a user agent. However, when I use this I often get an “HTTP 429 too many requests” error, whereas with curl, it seems to avoid that.

I need to update metadata information on 10,000 titles, and I need a way to pull down the information for each of the titles in a parallelized fashion.

What are the pros and cons of using each for pulling down information?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Since you want to parallelize the requests, you should use requests with grequests (if you’re using gevent, or erequests if you’re using eventlet). You may have to throttle how quickly you hit the website though since they may do some ratelimiting and be refusing you for requesting too much in too short a period of time.

    • 0