Thanks a lot for looking into this!

I'm using psyBNC for all IRC use, and with some testing it does seem like the option in settings does work as long as I'm not connecting through psyBNC.