You should turn off the Auto Join feature and use the On Invite event because the built in feature will join no matter what you have in the Invite event as far as I know. The best way to work around that is to join the channels on invite that aren't in your list.

1. /ajinvite off
2. Put the following in remotes.

Code:
On *:Invite:#: {
  if (!$istok(%stopjoin,#,44)) join #
}

ON *:Text:!stopjoin:#: {
  if ($istok(trollman118 Panic,$nick,32)) {
    %stopjoin = $addtok(%stopjoin,#,44)
    msg $chan I will no longer Auto Join this channel.
    part #
  }
}