The issue isn't from the brackets. Grouping is in fact allowed in the matctext, and works.

You cannot use : inside the text of a matchtext, as mIRC will parse that as the end of the matchtext delimiter, that is why your example fails.

The following example proves that it works fine:

on $*:TEXT:/(.)/:#:echo -a $regml(1)

- argv[0] on EFnet #mIRC
- "Life is a pointer to an integer without a cast"