There isn't any way to change the limit on recursion because technically speaking speaking it doesn't exist in mIRC scripting anymore.
You can get around it by using indirect recursion with two aliases which call each other or by calling the command via
scon/
scid (
example here). Both of those methods are severely limited by mIRC's own call stack limit which cannot be increased, so you may find you have to avoid recursion completely.