If this were the case, then any number would return the total number of user's with this usermode in the channel. It only does this if the "nickname" is 0 or a nick is 11 digits or more.
All that means is that the number rolls over because it's too big. What's happening is that once a number is too large, it flips to negative and works its way back up to 0 and then into the positives again.
To test if it's the length of digits or the number itself...
(Change ! to whatever mode you're checking)
//echo -a $nick($chan,10000000000,!)
//echo -a $nick($chan,11111111111,!)
The first one will give an error and the second one will give the total number of users with the mode. Now, to prove that it's actually going into the negatives (rolled over once it's too large), try:
//echo -a $nick($chan,-5,!)
You'll see it treats all negatives as 0 as well because negative numbers aren't acceptable in this location.
So you can see that it's related the the maximum number that can be used in whatever calculation mIRC's doing and not the total number of numbers.
And, just because I felt like looking into it, it appears that the number is a signed 20-bit integer. This means that the number can be anything from -10737418239 to +10737418239.
//echo -a $nick($chan,10737418239,!)
//echo -a $nick($chan,10737418240,!)
The first gives the error, the second gives the total (because it's rolled over into negative).
As to why it's a signed 20-bit integer, I have no idea. That seems a little strange to me why it isn't at least using 16-bit, 24-bit, or 32-bit.
Btw, this is the same thing starbucks_mafia was saying.