Question

I have a sudden issue in Production code which was not happening since the last 5-6 years. I have a Thread Pool that spawns a maximum of 64 threads, and all 64 threads read some data and put it in a common Map for further processing.

The read is done by all threads from a specific source, and I have verified that the data is indeed getting read from the source, however, one particular batch is not getting put in the Map.

Here is a code snippet (cannot put whole code due to confidentiality issues):

try {
   <read the data>
    .
    .
    <do processing>
    .
    .
    synchronized(glock) { //glock is a class attribute, Object glock = new Object[];
     map.put(<data that was read>);
     log.debug("bla bla bla")
    }
} catch(Throwable e) { 
     log.error("error") 
  }
  finally {
    log.debug("done")
 }

ISSUE: A particular thread does not go into the synchronized block, does not put into the map, does not print "bla bla bla", does not print "error" but it prints "done".

I have verified everything…nothing has changed in the code, this issue has appeared immediately from nowhere. The problem is, I cannot put any additional logs since it is production code without getting all clients agree, but that is the last part.

Has anyone faced a similar issue, or know anything about it? The data being read is huge, 6000 records and each record having a minimum 0f 30-40 columns of data.

Thanks in advance.

EDIT: Catching Throwable and not Exception

Answer 1

There is a 99% chance that something is wrong with your code. This means the code in the synchronized block throws an exception but you don’t see it.

The usual culprits are:

• Empty catch block which swallows exceptions (can be elsewhere in the code)
• Unusual log config for the logger in the catch block which, for example, writes exceptions into a different log file
• Log messages aren’t written in order for some reason (so the ERROR line is not where you expect it)
• Some unusual condition (like out of memory) plus resilient code prevents writing of the log messages. Maps can need surprising amounts of memory.
• log isn’t a standard Java logger but something else.

There is a small (< 1%) chance that you found a bug in your VM or there is a hardware issue. If you can get the same results repeatedly, it’s probably not a hardware issue.

If everything else fails, you will need to debug the issue in production. Of course, your client will object; at that time, you will let them decide which is more important to them: Some rule that you mustn’t debug the code or fixing the bug.