Hi i have requirement to process large number of files via multithreading in java. The files will be of random size (min:100 MB , max :1.5GB).The configuration is i can only create 8 thread at the max and each thread will be allocated 8 files for processing from the source directory.The issue is sometimes huge files are getting allocated to single thread itself thus degrading the performance. I want know whether there is anyway to allocate files to thread such that all threads will be processing equal amount of size. I mean i want to balance the load among the threads on the basis of file size.
Thanks in advance 🙂
You should not even be doing parallel I/O from a single mechanical disk, as it is in fact slower than single threaded I/O. There are a lot of answers around here explaining that. Basically the mechanical head of the disk needs to spin every time to seek the next reading location. This is a costly operation. If you do this in parallel then you are just bouncing the head around as each thread gets its turn to run.
The best approach would be to just read the files one-by-one sequentially using a single producer thread and process them in parallel using a pool of worker threads.