When you combine the variables and a value (such as $nick) u either need an eval or the []'s. Also, your delay for ignore time is set to 10 (meanign seconds, not minutes.. From the help:

The -u# switch specifies a delay in seconds after which the ignore is automatically removed.

Try this..

Code:
ON *:TEXT:!command:*:{
  [color:green]; ths "if null" is pointless from the variable perspective.[/color]
  [color:green]; The /inc will make null values default to 1 after being executed[/color]
  /inc -u10 %4ignore [color:red][[/color] $+ [color:red][[/color] $nick [color:red]] ][/color] 1

  [color:green]; check for limit[/color]
  if (%4ignore [color:red][[/color] $+ [color:red][[/color] $nick [color:red]] ][/color] >= 10) {
      /msg $nick Maxed out. You have gone over the allotted msgs
      /msg $nick Ignoring your messages for 5 minutes.
      [color:green]; using address 1 is the same as nick!*@*[/color]
      /ignore -pcntikdu[color:red]300[/color] $address($nick, 1)
      /unset %4ignore [color:red][[/color] $+ [color:red][[/color] $nick [color:red]] ][/color]
      /echo MAX for $nick
    }
  }
}

i Didnt debug, so if you find an error, let me know, and I'll fix it.