I think the color is based on a hash calculation against $address($nick,something), so if the nick identifies after joining the channel, then their address appears to change, and there's a good chance their color would also change.