Question

As in question. My current code is overkill for OS because it runs every wget process in seperate thread, which is fine, but I have almost 15k files to download, so I want to use a thread pool for this job. Unfortunately I must use wget for download process.

    ExecutorService executor = Executors.newFixedThreadPool(5);
  for(String filename: files) {
        try {
            String encodedFilename = URLEncoder.encode(filename, "UTF-8");
            final String cmd = "wget --no-check-certificate -O " + filename +" " + BipDownloader.bipUrl + encodedFilename;

            Runnable run = new Runnable()
            {
                public void run() {
                    try {
                        System.out.println(cmd);
                        Process process = Runtime.getRuntime().exec(cmd);
                    } catch (IOException e) {
                        // TODO Auto-generated catch block
                        e.printStackTrace();
                    }  

                }
            };
            executor.submit(run);
        } catch(IOException e) {
            System.err.println(e.getMessage());
        }    
  }

EDIT

Updated source code to use Thread Pool but my system still is unstable during download.

Answer 1

Assuming that you do need to use wget, you can use an ExecutorService to handle a threadpool for you:

ExecutorService executor = new FixedThreadPool(100); //pool of 100 threads

...

Runnable r = new Runnable() {

    public void run() {
        try {
            System.out.println(cmd);
            Process process = Runtime.getRuntime().exec(cmd);
        } catch (IOException e) {
            e.printStackTrace();
        }  
    }
}

executor.submit(r);

• The optimal size of the pool depends on various factors and it is best to test several numbers. Something between 100 and 1000 should be ok.
• If you need to monitor the progress of the executions, you can store the futures returned by executor.submit, or you can use a CompletionExecutorService.

EDIT

As noted in the comments, exec is non blocking so in theory, it is possible that all processes will be started before any of them has finished, even if the size of the pool is limited. To prevent that you should wait until in your run method until the process finishes:

Process process = Runtime.getRuntime().exec(cmd);
int exitVal = process.waitFor();