Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
It seem what i can do fast crawler with python in two ways:
thread pool with block sockets
non block sockets select,asyncore,etc..
i thnk where is no real need in thread here, and solution #2 better.
which is better and why?
Twisted is usually preferred to asyncore. It is an asynchronous I/O framework that can also work with thread pools.
In Python, you should prefer asynchronous IO to threads, simply because threads are a second class citizen in its canonical implementation (CPython) due to GIL.