$md5(text|&binvar|filename,[N])
Returns md5 hash value for the specified data, where N = 0 for plain text (default), 1 for &binvar, 2 for filename.

Don't put the character "," if you'll put something different of 0, 1 or 2.

/set %test te,st
//echo -a $md5(%test)
will returns:
9cfd05a790a462d5cbad1dda35e3f8dc

//echo -a $md5(te,st)
will return:
569ef72642be0fadd711d6a468d68ee1

//echo -a $md5(te)
will return:
569ef72642be0fadd711d6a468d68ee1

* I'm using mIRC 6.03.