Limit it to only channels that you want the bot it, or limit it to only people who you actually want to allow to invite it. Trust those people to only invite when they are ops and let them know that you'll remove the priveledge of inviting the bot if they do it when they are not ops.
You really shouldn't give any control over a bot to anyone you don't trust anyhow.
