Sorry, but that is not a bug. The height of listbox items currently simply depends on whether the "check" property is present or not. As such, you are trying to achieve something that is not supported.
The "size" property, by the looks of it, forces the listbox to the exact given height instead of being truncated to whole items. As such, it is unrelated to your issue.
I would recommend turning this post into a feature suggestion, along the lines of "please add a property that gives list items the same height as though they had a checkbox". Even then, given that your whole design won't work without additional extensions (the synchronized-scrolling thing), you should probably start thinking about a new design, no matter how cool this looks. mIRC dialogs only provide basic functionality; for anything more advanced you'd have to use something like DCX.