You should run a timer when they join and check if ($nick !isowner $chan) then.