your problem was a simple one:
$+ %text.color $+ $regsubex($1-,@([^$chr(40)]*)($chr(40)(.*?)$chr(41))/g, $+ 04$chr(44)01 $+ \3 $+ $+ 09$chr(44)01)
the parts in red (and, in fact, the entire surrounding expression) is treated as plaintext by mIRC - this follows from the basics of mIRC syntax. the expression "@([^$chr(40)]*)($chr(40)(.*?)$chr(41))/g" gets passed to the regex engine and, because of the plaintext $chr()s, cannot possibly match any string.
you can solve this any numbers of ways, the simplest of which is to just throw an extra ) into your original expression:
now your parentheses are balanced and you can put that expression into $regsubex() without any trouble. obviously, the meaning of the expression has changed slightly, in that a ')' in between '@' and '(' is no longer acceptable. if you do expect this to appear in $1- and you don't want the meaning changed, you can use the following expression functionally equivalent to the one in your post:
as you can see, the \)? serves little purpose; it's just there to ensure balanced parentheses.
also, you have errors in your substitution text. you'll notice it after testing a working expression, nothing that moving a few $+s won't sort out