Look at /help $regex
for info on regular expressions. The syntax you are using looks like TCL. mIRC uses the PERL (proper) syntax for regex. For more info on the syntax, go here: http://www.regular-expressions.info
Looking at your example, it seems like you want to ensure that there are equal numbers of ( and ) characters. In mIRC you can do that easily like this:
if ($count($2,$chr(40)) != $count($2,$chr(41))) return
To check if there are any invalid characters, you can use this regex:
if ($regex($2,[^\w().-])) return
Just a note, certain characters don't need to be escaped when they are within  brackets.