Question

I have about 140,000 one time API calls I need to do, the problem is that they all take about 15 seconds. This would take about 25 days to do successively, however, I’d like to get it done faster.

I’m planning on dumping the data returned into a MySQL database using a JSON parser and a function that takes the parsed data. I am familiar with Python and PHP.

What is the best way (as in, fastest and simplest to implement) to do a number of API calls concurrently and have the returned items parsed into a DB?

Answer 1

Probably you’ll have to try multithreading / multiprocessing.

This will only help if multiple parallel calls are allowed by the API (and if the machine where the API is run is fast enough to process more than one call at a time).

If the bottleneck is not your own CPU, you can simply use the threading module, as multiprocessing won’t have any more improvements.

As unchecked code, you could try:

import Queue
import threading

number_of_API_readers = 10

parameters_queue = Queue.Queue()
results_queue = Queue.Queue(100)

for par in list_of_parameters:
    parameters_queue.put(par)

def read_from_queue_and_write_to_db():
    while True:
        try:
            result = results_queue.get(timeout=30)
            #write result to DB
        except Queue.Full:
            break

def query_API_and_write_to_queue():
    while True:
        try:
            par = parameters_queue.get(timeout=5)
            result = SomeAPI.call(par)
            results_queue.put(result)
        except Queue.Empty:
            break

db_writer = threading.Thread(target=read_from_queue_and_write_to_db)
api_readers = [threading.Thread(target=query_API_and_write_to_queue) 
                for i in range(number_of_API_readers)]

db_writer.start()
for ar in api_readers:
    ar.start()

The main ideas in this code:

• have one thread writing to the database
• have many, e.g., 10 threads querying the API
• use (thread-safe) queues to synchronize the threads.