I think you're wanting the "%spamlines" and "%spamnick" to restart to 0 when a mod says a message in channel. I would get rid of the "if ($nick isop $chan) { return }" line and add a check to see if the user who typed the message is a mod before adding their name to the %spamnick and adding to the %spamlines. If they are a mod, then erase/unset the %spamlines and %spamnick.

Code:
on *:text:!timeout*:#: {
  if ($nick isop $chan) {
    if ($2 == line) {
      if ($3 isnum) {
        %spamline = $abs($int($3))
        msg # Spamline is now $abs($int($3)) $+ . Messaging $abs($int($3)) times in a row will timeout you!
      }
      else { msg # $3 is not a number! | return }
    }
    elseif ($2 == time) {
      if ($3 isnum) {
        %spamtime = $abs($int($3))
        msg # Timeout time is now $abs($int($3)) $+ !
      }
      else { msg # $3 is not a number! | return }
    }
    elseif ($2 == message) {
      %spammessage = $3-
      msg # Timeout message is now : $3-
    }
    else { msg # insufficient parameters. !timeout <line,time,message> <parameter> }
  }
  else {
    msg # You dont have permission to do that!
  }
}

on *:text:*:#: {
  if ($nick == %spamnick) {
    inc %spamlines
    if (%spamlines == %spamline) {
      msg # .timeout $nick %spamtime %spammessage
      msg # $nick you're spamming too fast. slow down!
      %spamlines = 1
    }
  }
  elseif ($nick isop $chan) {
    unset %spamnick
    unset %spamlines
  }
  else {
    %spamnick = $nick
    %spamlines = 1
  }
}