The problem is the presence of the ] character within the character set creating bad regular expression syntax. You can either escape it with a backslash or put it as the first character within the character set (ie. /([[color:red]]< >%{}|\^~[`])/g[/color])

I'm not sure if that would've worked with any version of mIRC (more specifically the version of PCRE used by mIRC), generally having all those characters with possible syntactic meaning in the expression will be a bit of a minefield. I would either escape them all or better yet use a more comprehensive expression that will URL-encode any non-alphanumeric character without having to specify them all in the expression.

Personally I use one of the following:
Code:
; Escapes all non-alphanumeric characters (except underscore which is fine)
alias urlencode return $regsubex($1,/([^\w\d])/g,% $+ $base($asc(\t),10,16))

; or

; As above but converts space to a +
alias urlencode return $replace($regsubex($1,/([^\w\d\x20])/g,% $+ $base($asc(\t),10,16)), $chr(32), +)

Spelling mistakes, grammatical errors, and stupid comments are intentional.