That is a possibilty, but the kick kicks the $nick that set the ban.. not all nicks matching the $address. So that shouldn't be a problem as far as I can see.