Well if you wanna talk about REAL hackers then no, not all of them do. A hacker, by original definition, is a programmer. Codemastr, for example, is a hacker and one who hacks IRC
Crackers (not the Jacob's kind) on the other hand can, have and will use IRC as a means to gain access to someone elses machine. Albeit through a DCC-send of a trojan, a worm propagated by $decode or just by spying their IP address and using wotever tools they have handy.
Don't talk to me about these Crackers, I have dealt with them before. I am not as ignorant as you may think so I thank you not to underestimate what I know
