I am measuring “hiccups” using jHiccup tool build by Azul Systems. It collects data to identify the frequency and duration of pause times (hiccups) that occur while a JVM runs a Java application. It works on JVM level and beyond ( OS,driver and etc ).
Here is the results 
These results were gained on machine with SUSE SLERT 11 2.6.33 kernel PREEMPT RT, Intel i5, 4g memory. The process was running under cpu shielding ( 3 logical processors were isolated ) and with 99 priority (FIFO). I am wondering where this 57 mcs latency comes from. The application is pretty straightforward. It is network order processing system, so it parsing TCP packets feed, doing simple business logic and thats all. No GC, syncronizations, it is single threaded.
My guess is it might be a network issue e.g. blocking read? When I tryed non-blocking read with busy-waiting, I got similar results, but maybe i am doing it wrong. I have no clue where these hiccups can come from.
IRQ Balance will distribute the interrupt handling across your cpus. You can turn this off or control its masks so you don’t get interrupted as much (Unfortunately there are two intrrupts you cannot turn off)
Logical processes on the same core can disturb each other. For best results, I would isolate a core and only use that.
Even though you have shielded the application it has many threads. For best results I have used linux to isolate a number of cores and only assign the critical threads to those cores. i.e. the other threads in the same application don’t use those cores at all.
To control this, I have written this library Java Thread Affinity Even with this library, I see some jitter (though as much as 10x less) which could be caused by the power management or Local Timer Interrupt.