Furthermore twitch's own handling, even doing it this way, is inaccurate and leads to the nicklist being wrong.
How? If it's removing canceled events it should have no effect. The only explanation would be that it's doing something other than what you are describing (like, say, outright hiding JOIN/PART events).
The issue might be related to the batching, though. If it's waiting to batch a bunch of joins, it might be waiting until after they have already spoken, leading to someone talking when they are "not in the channel"
