My approach to channel security does limit the amount of users that come in but I've only ever invited people to my own channels who I've known a while and those who use them do the same.