I'm going to presume that you have some other method of communicating with these other people, example Yahoo or MSN Messenger. Just how, doesn't matter.
Next time you join your channel and it only shows 1, contact one of your friends and have them join the channel. If their display shows 2 (or more), then get them to type //echo -a $server
NOTE: Both // are required.
While they do this, you do the same. I'd be willing to bet that the information returned to your friend does not match what is returned to you. If this is the case, then the simple answer is for you to use the /server command and specify the server that your friend is using.
It's not unheard of for a network to have multiple servers, and if the servers aren't communicating with each other, then what you've encountered is not only possible, but probable.
