The limit is not imposed by mIRC, so from the perspective of the program, it is unlimited. The only limit is your system environment. With memory paging to disk, that extends well into the terabytes, likely well beyond the scope of any kind of data that mIRC could conceivably process anyway.
So really, the only limit is your mind.
Though, as a theoretical exercise, it is worth noting that any table with N significantly larger than the bucket size will eventually start to see O(m) (linear) performance, so if you really are dealing with gigabytes+ worth of entries, the max bucket size may actually create performance bottlenecks-- though to be fair, those bottlenecks will be much smaller than the loops/commands used to write/read/process the data in your script. Of course, if you really do end up in this situation and want to minimize impact, it would be simple enough to create multiple hash tables to spread load horizontally (commonly known as data sharding). But seriously, you're unlikely to ever hit this perf issue-- this is a theoretical issue, not a practical problem.