I was still getting drift. So in the end I switched to a different approach to my event trace log where I store $ticks at the start of an event and then show the increment as various trace entries are made during the event handling.

Events almost always take < 1s to execute so this approach is just fine for my requirement.