The port that you need to use really depends on each individual IRC server and the network that it belongs to. True, 6667 is used for most IRC servers, but that doesn't mean that it will work for all.
For instance, the default port for DALnet servers is 7000. What you can do is to try and connect to the network's website to see if they give you any connection instructions.
It may also be possible that you're trying to connect to an IRC server that is restricted to a very select number of people. Servers like that, aka hubs, are essentially used to connect different IRC servers on a network to each other and these special servers are not open to the public.
