I am running a series of tasks using Celery and RabbitMQ which quickly reaches the rate limits of websites the tasks request from. I was wondering if it’s possible to bind an instance of celeryd to an public IP address and have multiple instances running on the same machine?
I do have multiple public IP addresses I can use but I have very little experience with networking.
Thanks in advance for any help!
Yes, you should be able to do what you want to do by running several celeryd instances and binding each to an IP using a shim. See here.
Another option would be to just assign the additional IPs to your network card using
ifconfigand let the OS send the traffic over whatever IP it chooses to. According to this you should get a distribution of traffic across the different IPs. Downside: no control which IP is used by which process.Hope that helps