Could be the quality or bandwidth of your connection, the distance between you and the server you connect to, the amount of traffic on the path your connection follows, etc.

Is your browsing also slow?
Do you have the same problem connecting to other IRC networks?