IMHO this is inconsistent behaviour regardless of how division by zero should work.
If A % B returns value C, then $calc(A % B) should be equivalent to var %var = A % B. Furthermore, if A and B are both numeric, mIRC should not return a non-numeric return value.