Not sure of the exact cause of this, but I've been having this problem for a long time (since v6.*, it didn't occur in 5.91), I just hadn't had time to put it here. I am using Windows XP Home and currently 6.12. It's a bit hard to explain, but I'll try...
mIRC keeps a buffer of all messages/joins/etc in the channel window (we all know this), but when the buffer starts to grow larger the scrollbar from within that window, be it channel, query, dcc chat, status, etc., doesn't 'update' properly.
Say the scrollbar is 3mm big, when you grab it, scroll completely up to the top of the window then back down it again, it 'grows' to about 3cm (for example), and vise versa, it grows and shrinks randomly.
Using my scrollwheel on my house causes the window to 'shake' up and down. Pressing "Page UP" only makes it jump like 6 lines. Both these problems can be fixed by grabbing the scroller and moving it up to the top and then back to the bottom, and all returns to normal.
I hope I explained it alright... What's up with this?