$bytes(size,3) returns the "3 digit format"-- that means in terms of significant digits, not zero padding. You can use $base($bytes(1,3),10,10,3) if you need specific zero padding, but $bytes(1,3) will not return 001.
The explanation for the first example seems to be that mIRC gives up on 3 digit representation of file sizes in the thousands-of-TB range.. It makes sense if you realize that mIRC would have otherwise switched to the next SI representation (PB, petabyte) when it reaches the 1000's of-- but since PB isn't supported (I imagine), it has nowhere to go.