Question

I am developing a REST web service that should receive files from the clients and precess them. After that I receive the file I want to create a new thread for processing the file, so I am not obliged to wait the end of the processing function.

If I am receiving a lot of files, I will create a lot of thread. Is there any limit or danger to do this?

Answer 1

If I am receiving a lot of files, I will create a lot of thread. Is there any limit or danger to do this?

Yes there is. I’m not sure there is a limit on the number of threads but at some point you will run out of memory. Each of the threads have stack and other local storage that will add up.

I’d limit the number of threads that you have forked by not accepting new connections if the limit has been reached. Then additional connections will wait in the TCP queues until the previous requests have been completed.

A better mechanism may be to use a fixed ExecutorService thread pool instead of forking a new thread for each request:

// create a thread pool with 10 workers
ExecutorService threadPool = Executors.newFixedThreadPool(10);
while(!shutdown) {
   // receive a request from your service -- this is tricky
   // process it using the thread pool
   threadPool.submit(new MyRequest(...));
}
...
// once we have submitted all jobs to the thread pool, it should be shutdown
threadPool.shutdown();

The tricky part of this is how to determine which connections are readable. That takes NIO code, channel selectors, etc. to multiplex the connections you have to see which ones can be read.