Question

I’m implementing a multithreaded Wget with the URLs in the Open Directory Project (ODP). I get the following error:

java.io.FileNotFoundException: log (Too many open files)
at java.io.FileOutputStream.openAppend(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:207)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at java.io.FileWriter.<init>(FileWriter.java:78)
at ODP.run(ODP.java:103)

I’ve read that it’s because it overpasses the limit of open file descriptors indicated in the system variable: cat /proc/sys/fs/file-max

Is there a way to limit the number of threads that can be started at the same time? I thought that it did it automatically by the number of threads that could run the CPU.

Thank you!

Edit: this is the main loop inside the main method:

        while (rs.next ()) {                
            // Process entry
            String rsc = rs.getString ("resource");
            String tpc = rs.getString("topic");             
            (new ODP(rsc, tpc, rs.getString("description"))).start();   
            BufferedWriter outLog = new BufferedWriter(new FileWriter("log", true));
    outLog.write(count + ": " + rsc + " | " + tpc  + "\n");
    outLog.close(); 
            ++count;                
        }

In the ODP.run() method I create a BufferedWritter:

        BufferedWriter out = new BufferedWriter(new FileWriter(cat.split("/")[1] + ".webtrec", true));
        out.write(webtrec);
        out.close();

Answer 1

Is there a way to limit the number of threads that can be started at
the same time?

Yes there is, by using some kind of thread pool. If you are using Java 1.5 and above, you are in luck; you can use ThreadPoolExecutor to limit the number of concurrent threads.