$regex returns 1 for matching, 0 for not matching. If you want the total number of matches you need to use the //g modifier. $regml(0) will then return the number of matches. Use the i and S modifiers for case insensitivity and color stripping. This is now the regex:

/([a-z0-9]+ -> \$\d+ \(Σ\$\d+\))/giS

If you want to catch text and actions have them each call an alias.

Code:
on *:TEXT:* -> $* (Σ$*)*:#test:{
  testspam $1-
}

on *:ACTION:* -> $* (Σ$*)*:#test:{
  testspam $1-
}

alias testspam {
  var %regex = /([a-z0-9]+ -> \$\d+ \(Σ\$\d+\))/giS
  if ($regex($1-, %regex) {
    inc -u60 %betSpam. [ $+ [ $nick ] ] $regml(0)
  }
  if (%betSpam. [ $+ [ $nick ] ] >= 3) {
    msg $chan .timeout $nick
  }
}