Just add
Code:
if (($nick isop $chan) || (nick isvoice $chan)) halt

To the first line, right under your on join, and on part events.