Question

I have a very big CSV file (1GB+), it has 100,000 line.

I need to write a Java program to parse each line from the CSV file to create a body for a HTTP request to send out.

In other words, I need send out 100,000 HTTP requests which are corresponding to the lines in the CSV file. It will be very long if I do these in a single thread.

I’d like to create 1,000 threads to do i) read a line from the CSV file, ii) create a HTTP request whose body contains the read line’s content, and iii) send the HTTP request out and receive response.

In this way, I need to split the CSV file into 1,000 chunks, and those chunks should have no overlapped lines in each other.

What’s the best way to such a splitting procedure?

Answer 1

Reading a single file at multiple positions concurrently wouldn’t let you go any faster (but it could slow you down considerably).

Instead of reading the file from multiple threads, read the file from a single thread, and parallelize the processing of these lines. A singe thread should read your CSV line-by-line, and put each line in a queue. Multiple working threads should then take the next line from the queue, parse it, convert to a request, and process the request concurrently as needed. The splitting of the work would then be done by a single thread, ensuring that there are no missing lines or overlaps.