I have a webpage which interacts with several external APIs and in order to speed things up (the speed increase is almost linear because the majority of the time is spent waiting for http responses, etc), the code is threaded so that it pulls the content from several APIs at once.
The problem is, I am running into database locking presumably due to the fact that data from the apis is being written to the database by more than one thread simultaneously.
What can I do to eliminate this problem?
PS – please do not suggest making the api calls sequentially… The same problems will occur when multiple users are importing data at the same time, except that without the threading, it’ll be slower for everyone. And we’re talking about the difference between 15 seconds and several minutes.
Definitely don’t try this with SQLite: its major weakness is concurrency.
If MySQL shows the same problem, then you should collect all the data together and then write it from a single thread.