You remove them the same way you added them, except by also inserting the -r switch.

/protect foo@bar.net #foobar
/protect -r foo@bar.net #foobar

If you try to remove the address without mentioning the #channel, it won't work