This has nothing to do with the implementation of regular expressions (which is a separate library, PCRE, anyway). It has to do with how mirc parses identifiers in general. \) is a regex escape sequence (which indeed works with mirc's regex too) but it doesn't escape the special meaning of ")" in mirc: just as in
//echo -a $str(\),3)
"\)" doesn't prevent mirc from returning an error. Can you see why this example errors?
To pass ")" characters (or "\)" sequences for that matter) to an identifier, either escape them the mirc way (use ... $+ $chr(41) $+ ...) or put them in a variable and use that variable as an identifier parameter. For example:
//var %a = \) | echo -a $str(%a,3)
or
//echo -a $str(\ $+ $chr(41),3)
