First, it helps to create your post by using the hashtag symbol to add the 'code' tag to your post, so the code appears formatting like it does in the scripts editor, so it's easier to read. You can click on 'preview' to see if your post looks right. Sometimes you need to add some extra lines in front of the closing code-tag to make the last line easier to see. If using the code tag, your code can look like:

Code
alias check_badnick {
  var %f = badnicks.txt
  var %t = $lines(%f)
  if (!%t) { return }

  var %i = 1
  while (%i <= %t) {
    var %r = $read(%f,n,%i)
    if (%r iswm $1) { return 1 }
    inc %i
  }

  return 0
}


When you have trouble with code not working correctly, one thing that's a big help to find the problem is to insert debug messages into the script.

echo -s debug $scriptline $nofile($script) event= $event nick= $nick $ $+ 1- $qt($1-) message and/or variablles and or identifiers go here

This echoes a message to your status window which lets you see that it reached a specific line number of a specific script. That helps located the problem, and also helps find it when it's time to remove them. If you see the message showing line 4 but not line 10, and it should be showing both, this helps you identify the location of the problem, though the problem can sometimes be caused by an earlier line not doing its job correctly.

It's hard to tell where your problem is, as the code looks fine. Try manually testing your check_badnick alias to see if it works correctly.

//echo -a $check_badnick(badword_goes_here)

If this returns 0 for a bad word, or 1 for a good word, then you need to fix that problem first.

It's also a good idea to check if you're OP in the channel where the nick is changing. I think the @ event prefix should work to avoid triggering the ON NICK event if the person changing nick only shares a nick where you're not an op. But in addition, something like this makes sure you don't send empty threats to someone in a channel where you're not an OP:

Code
var %chan = $comchan($newnick,%i) | if (!$nick(%chan,$me,@)) { inc %i | continue }