I have been having the EXACT same problem as you. I have not been able to figure out what the cause is much less a fix.
I have tried 3 different computers in my house. I have formatted them, and tried them with no other software running.
I have also tried other IRC clients.
I tried taking the router out of the equation, and hooking straight to the cable modem.
I JUST DON'T GET WHAT THE PROBLEM IS!!!
I can download huge files from FTP, www, etc at 400 KB/s no problem what so ever, but give me a nice fast dcc and it will freeze unfreeze freeze unfreeze my PC's.
I have a friend with the exact same ISP, and he doesn't suffer this problem.
If there is anyone out there that has any clue what so ever as to what this problem could be, it would be most appreciated. This is the first time a google search has even found knowledge that someone else is having the same problem. I thought I was the only one, and it was driving me insane. At least now I know I am not alone in this.