This isn't a bug, as already stated, you're adding a global ignore, but trying to remove a local ignore setting.

Another alternative is opening the Address Book, clicking on the Control tab and deleting each of the ones you want removed.