Question

I have written a multi-threaded Java application which reads a bunch of .jar files from a directory. This application spawns multiple threads and each threads reads bunch of jar files. I’m having trouble identifying the stopping condition for this application. How can i identify that all the files have been read?

The following is a snippet function which gets called from the run() method for each thread.

    import java.io.*;
import java.util.Enumeration;
import java.util.jar.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipException;
import java.io.FilenameFilter;

public class ArchiveFileTest implements Runnable {
     private static boolean stopAll = false;
     private static int threadNumber = 0;
     private int myNumber = 0;

     public ArchiveFileTest () {
            myNumber = threadNumber;
            threadNumber++;
     }

     public static boolean setStopAll () {
            return setStopAll(true);
     }

     public static boolean setStopAll (boolean b) {
            stopAll = b;
            return stopAll;
     }

     public static String[] listFiles (File parentDir,final String ext1,final String ext2,final String ext3,final String ext4) {
            String allFiles[] = parentDir.list(new FilenameFilter() { 
                 public boolean accept(File pDir, String fName) {
                        if (fName.endsWith("."+ext1) || fName.endsWith("."+ext2) || fName.endsWith("."+ext3) || fName.endsWith("."+ext4)) return true;
                        else return false;
                 }
            });
            for (int i=0; i<allFiles.length; i++)
                 allFiles[i] = parentDir.getAbsolutePath() + File.separator +  allFiles[i];
            return allFiles;
     }

     public ZipFile getMyZipFile (File parentDir) {
            String fn[] = listFiles(parentDir, "jar", "zip", "war", "rar");
            int fileNum = myNumber % fn.length;
            ZipFile zFile = null;
            for (int i=0; i<fn.length; i++) {
                 String jFile = fn[(fileNum + i)%fn.length];
                 try {
                        zFile = new ZipFile(jFile);
                        break;
                 } catch  (IOException e) {
                     setStopAll();
                 }
            }
            return zFile;
     }

     public void doStuff() throws Exception {
            File dName = new File("/home/sqatest/chander/sample-files");
          final int N_TIMES = 15;
            final int N_FILES = 500;
          int counter = 0;
            int fCount  = 0;
            if (!dName.isDirectory() || !dName.exists()) {
                 System.err.println("The parent directory given should point to an existing directory...");
                 setStopAll();
                 return;
            }
          while (counter < N_TIMES) { 
                 ZipFile zipFile = getMyZipFile(dName);
                 if (zipFile == null) {
                        System.err.println("No zip file entry for the Thread-" + myNumber);
                        break;
                 }
                 try {
                        Enumeration <? extends ZipEntry> zipEntries = zipFile.entries(); 
                        fCount = 0;
                        ZipEntry ze = null;
                        while (zipEntries.hasMoreElements()) {
                             ze = zipEntries.nextElement();

                             if (ze.isDirectory()) continue; // if it is a directory go to next entry

                             InputStream is = zipFile.getInputStream(ze); 
                             fCount++;
                             int readCount = 0;
                             try {
                                    while(is.read((new byte[50])) != -1 && readCount != 200) readCount++;
                                    System.out.println("Successfully Read " + zipFile.toString());

                                    //is.close();
                             } catch (IOException e) {
                                    e.printStackTrace();
                             }
                             if (fCount == N_FILES) break;  // read maximum of N_FILES
                        }
                        if (stopAll) break;
                 } catch (Exception e) {
                     e.printStackTrace();
                 } finally {
                        counter++; 
                 }
          } 
     }

     public void run () {
            try {
                 doStuff();
            } catch (IOException e) {
                 e.printStackTrace();
                 setStopAll();
            } catch (Exception e) {
                 e.printStackTrace();
            }
     }

     public static void main (String[] args) throws Exception { 
          final int MAX_THREADS = 500;
          final int MAX_HOLDING_THREADS = 5;
            int loopCount = 0;
            Thread mainThread = Thread.currentThread();
            for (int m=0; ; m++) {
                 Thread t[] = new Thread[MAX_HOLDING_THREADS];
                 for (int n=0; n<t.length; n++) {
                        t[n] = new Thread(new ArchiveFileTest());
                        t[n].start();
                        if ((m+1)*(n+1)==MAX_THREADS) {
                             System.out.println("\n" + MAX_THREADS + " reached... \nMain Sleeping for some mins..."); 
                             loopCount++;
                             try {
                                    t[n].join();
                                    System.out.println("\nMain is back... (" + loopCount + ")");
                             } catch (InterruptedException e) {
                                    e.printStackTrace();
                                    setStopAll();
                             }
                             m = 0;
                        }
                 }
            }
     }
}

Answer 1

I don’t think your application will ever stop. You’ve got an infinite loop in the main method:

for (int m=0; ; m++) {
    ....
}

Note, setting m=0 inside the body won’t break the loop, so I think you’ll never end even if you have no file. It then continuously reads all zip/jar/war/rar files in the directory (choosing the file based on a rotating counter myNumber is not very maintainable), but never exits the loop.

If you’re requirement is to read ZIP files using a number of threads, then I would go about it a different way.

1. Create a Set of files which you want to look at.
2. Create a ThreadPoolExecutor to create a fixed pool of 5 threads
3. Iterate over the set of files and create a new Runnable which does the Zip Extraction (though I’m not quite sure why you read the first 10000 bytes of a ZIP entry and then don’t do anything with it), and call the execute method. That will use the thread pool to process 5 files at a time.
4. After submitting all the runnables Use the shutdown method, which will wait for all submitted tasks to finish, and the shutdown the thread pool.