A parenthesis is a valid character of base-64 encoded data. $decode() can work with it just fine. The only caveat to that is that mIRC must have some way of knowing that the parenthesis is part of the text to be decoded and not part of the syntax of the scripting language. The only time that would be a problem is if you are manually entering the encoded text (typing/pasting it into the command line) or perhaps working with evaluation brackets, timers or scon/scid.
Example:
//echo -a $encode(helloaaa)
(:&5L;&]A86$`
; Encoded text contains a parenthesis
//echo -a $decode($encode(helloaaa))
helloaaa
; Encoded text is taken as a parameter by $decode and returns original string
//echo 12 -a $decode((:&5L;&]A86$`)
* Invalid format: $decode
; The encoded text manually typed in causes an error as mIRC has no way to know that the parenthesis is meant as part of the parameter and not code
//var %encoded = (:&5L;&]A86$` | echo -a $decode(%encoded)
helloaaa
; By passing the encoded text to a variable (where the parenthesis means nothing special) you can then pass that variable to $decode() and get the expected result
In short, the solution is to pass the parameter in via a variable or some other means ($input() might also be useful for your purposes). This isn't a bug, just an unavoidable consequence of mIRC's scripting language not using quotes for strings to enable explicit distinction between text and code.