Question

Is there any method so that I can split a text file in java without reading it?

I want to process a large text file in GB’s, so I want to split file in small parts and apply thread over each file and combine result for it.

As I will be reading it for small parts then splitting a file by reading it won’t make any sense as I will have to read same file for twice and it will degrade my performance.

Answer 1

Your threading attempt is ill formed. If you have to do significant processing with your file data consider following threading structure:

1 Reader Thread (Reads the File and feeds the workers )

• Queue with read chunks

1..n Worker Threads (n depends on your cpu cores, processes the data chunks from the reader thread)

• Queue or dictionary with processed chunks

1 Writer Thread ( Writes results to some file)

Maybe you could combine the Reader / Writer thread into one thread because it doesn’t make much sense to parallelize IO on the same physical harddisk.

It’s clear that you need some synchronization stuff between the threads. Especially for queues think about semaphores