This limit is most probably imposed by your OS and not by mIRC. I managed to open about 400 custom windows before mIRC gave up with an
* Error creating window error in Status. Are you sure you aren't getting the same error? If it turns out you are, read on.
On Win9x/ME there is a pretty low limit on the number of the so called GDI, User and System objects. Roughly, the more windows you have open in total (of any application), the more objects of these kinds are in use. The limit on such systems is very easily reached.
On WinNT/2000/XP there is also a GDI object limit but is much higher and very rarely reached. A more frequent limiting factor on the maximum number of windows (although not nearly as limiting as the Win9x ones) is a memory area called Desktop Heap. You can find more info in
this MSDN blog post and the Microsoft KB article it links to.