Not only is it possible for the multiple channels/networks issue to be the cause, it is, in fact, probable.
While mIRC can connect to many networks and many channels, the more networks and the more channels it has to reference, the slower it's going to be, since mIRC is a single threaded application.
The detected slow down while using multiple channels/networks has been mentioned before many times.
