I believe the main problem is that your %a var is not incrementing correctly. You need to remove the line /set %a $calc(%b + 1)

Here I've rearranged a few things and taken the liberty of changing $nick to $wildsite ( which is equivalent to $address(nick,2) )
to prevent a simple nick change from getting around your counter.

Code:
on @*:text:*:#:{
if ($nick !isop $chan) {
  var %a = 1
  var %b = $ini(sysbadword.ini,$chan,0)
  var %c = $ini(sysbadword.ini,$chan,%a)
  while (%a <= %b) {
    if (%c isin $1-) {
      inc %asn. [ $+ [ $wildsite ] ]
      if (%asn. [ $+ [ $wildsite ] ] == 1) { kick $chan $nick $readini(sysbadword.ini,$chan,%c) ( $+ %c $+ ) }
      if (%asn. [ $+ [ $wildsite ] ] == 2) { 
        mode $chan +b $address($nick,2)
        kick $chan $nick $readini(sysbadword.ini,$chan,%c) ( $+ %c $+ )
        timerlavaaboca 1 18000 /mode $chan -b $address($nick,2)
      }
    inc %a
  }
}

Note that the %asn.$wildsite variable is never unset. This could cause the script to fail for a frequent abuser.
You may want to unset it after banning, to give 2 more chances, or add a permanent ban for a count of 3 or more.
If you'd simply like to continue to ban for 5 hours for each subsequent offense, just change

Code:
if (%asn. [ $+ [ $wildsite ] ] == 2) {


to

Code:
if (%asn. [ $+ [ $wildsite ] ] >= 2) {