I have a large text file in which each line requires intensive process. The design is to have a class that reads the file and delegates the processing of each line to a thread, via thread pool. The file reader class should be
blocked from reading the next line once there is no free thread in the pool to do the processing.
code// read each line {
// create a Runnable to process the line
// pass the Runnable to the thread pool
// block if there is no free thread in the pool
// }[/code]
Using Executors.newFixedThreadPool() seems not a good solution because the the created thread pool operates on a shared unbounded queue. This might cause an out-of-memory because we keep filling the Runnable to the unbounded queue, which is almost like loading the whole file into memory).
codeExecutorService tpes = Executors.newFixedThreadPool(3);
int c = 0;
while ((line = in.readLine()) != null) {
c++;
System.out.println("Manager: line #" + c);
tpes.execute(new WorkerThread(line)); // WorkerThread is a Runnable
}[/code]
Using ThreadPoolExecutor does not provide the solution as well--the execute() method of ThreadPoolExecutor does not actually block the caller; it only throws a RejectedExecutionException exception. [b]Do I need to create a special RejectedExecutionException handler to block the reading and to resubmit the same Runnable? But, how will the caller know when a thread becomes available in the pool?[/b]
codeThreadPoolExecutor tpExe = new ThreadPoolExecutor(2, 3, 30, TimeUnit.SECONDS, new ArrayBlockingQueue(5));
int c = 0;
while ((line = in.readLine()) != null) {
c++;
System.out.println("Manager: line #" + c);
try {
tpExe.execute(new WorkerThread(line));
}
catch (RejectedExecutionException e) {
System.out.println("Manager: RejectedExecutionException line #" + c);
}
}[/code]
