The issue isn't from the brackets. Grouping is in fact allowed in the matctext, and works.
You cannot use : inside the text of a matchtext, as mIRC will parse that as the end of the matchtext delimiter, that is why your example fails.
The following example proves that it works fine:
on $*:TEXT:/(.)/:#:echo -a $regml(1)