Unless there is a hardware problem with your machine, it typicaly mean interrupts are disabled for long period of time or some process is running at higher priority then Proc (very unlikely with default setup)
If the slowness is significant, say 1% or more off, you could write a program that hooks the tic interrupt and counts them. Then report every computer minute and sync it against an accurate clock. You should be able to see a problem within a few minutes.
I forgot to mention, that SMI can be responsable too. We’ve seen some computer being slow down by a crappy USB mouse that was sending garbage. It was handle through SMI because BIOS was set in emulation mode.
Thanks Mario and maschoen, you guys are correct the problem was due to the HW, there were false interrupts that was triggered by the unused com ports hence delaying the timer interrupt