Code:
On me:*:Join:#: {
  $+(.timerchkusers,.,$cid,.,$chan) 0 10 chkusers $chan
}

On me:*:Part:#: {
  if ($timer($+(chkusers,.,$cid,.,$chan))) $+(.timerchkusers,.,$cid,.,$chan) off
}

alias chkusers {
  if ($nick($1,0) == 1) {
    $+(.timerchkusers,.,$cid,.,$1) off
    part $1
  }
}


I'm not sure if there's a better way of doing this, and I've never really used $cid before.

When you join a channel a timer is started and checks every 10 seconds to see how many users are on the channel. If there's only 1 user (you) it parts the channel.