If you switch to using SASL to connect to swiftirc, does that resolve your issue?
Update: looking into this a little more, I did find some contexts where the "try next server in group" setting was not being observed. I have made a change in the next version which should resolve this.
