you seem to have modified the regex a tad:
alias braces return $regsubex($1-,/{.*?}/g,)
However your still going to have issues with:
//echo -a $braces({ he llo } you {} there {tag} whatsup?)
This is due the t "}" with spaces around it makes mIRC think it needs to close down.
i.e
//tokenize 32 a b c d e f g { if ($1-) echo -a $1- }
is the same as
//tokenize 32 a b c d e f g | if ($1-) echo -a $1-
So passing the parameters yourself is going to be crude:
//tokenize 32 $({,) a $(},) <-- is that being stripped ? | echo -a $braces($1-) orginal: $1-
but whenever "{ tag }" is put in an $ident by mIRC itself (i.e in $1- in on text event or $did() in a dialog) you wont have to escape { and }
i.e:
//tokenize 32 $({,) a $(},) <-- is that being stripped ? | var %x = $1- | echo -a $braces(%x)
Hope that helps

Whats actually funny about "}"
alias tr { tokenize 32 $({,) a $(},) <-- is that being stripped ? } set -l %x $1- } echo -a %x }
this works "fine" all be it you have to use set -l because it wont be sent in the aliases scope but in the main mIRC's scope , just as /var doesn't work in the editbox of mIRC.
alias tm {
var %x = $$1 * 3
} return %x
also works as expected this is because mIRC doesn't really parse by { } i.e it wont check while parsing a script if the correct ammount are used. It just keeps streaming the alias untill it can't no more.