The reason this doesn't work is that "iswm" means "is wildcard match", so mom* matches only when it begins the string, *mom matches only when it ends the string, and using regex symbols would match only when the nick contained things like literal \d characters.
mIRC also has the ISIN operator where (string1 isin string2) is basically the same thing as (*string1* iswm string2). If you want to use regex matches, you'd need to use the $regex identifier which returns negative numbers for some kinds of syntax errors, or otherwise returns the number of matches, which is 0 or 1, unless using the /g flag which allows multiple matches.
There are case-sensitive iswmcs and isincs, which you're probably not wanting. However regex by default is case-sensitive unless you wrap your search inside the /i flag. In your case, you can do something like this to check for several matches, and I get the impression you don't care whether or not it ends with a number, so there should be no need to mention them in your match checking:
if (mom isin $4) msg #filters message
if (mother isin $4) msg #filters message
is the same as
if ( (mom isin $4) || (mother isin $4) ) msg #filters message
is the same as
if ((*mom* iswm $4) || (*mother* iswm $4) ) msg #filters message
and you can do quite a few of these 'isin' type checks in a single case-insensitive regex
if ( $regex($4,/(?:mom|mother|mum)/i) > 0) msg #filters message
It's good that you're not automatically acting on matches, since 'isin' type matches can easily have false matches for strings like momentum or smother.
