That is the big downside to the 'Net. A professional hacker will usually just come and go without causing any trouble or interference and you may never even know they were there. Script kiddies doing it for kicks may know how to connect after discovering a vulnerability but that is usually the extent of their l337 h4x0r1ng knowledge and before they disconnect they'll either leave a 'calling card' (either on purpose or because they don't really know what they are doing) or just stuff your machine completely. I am personally not bothered by the first group, nothing can stop them and if you don't have any personal information on your machine (I don't) then there's usually no harm done. The second group is the one I strive to keep out and they are very easy to stop.

1. Don't just share files (send or receive) with anyone that happens by. Know and trust them first.
2. Don't visit spammed websites or open sus email attachments (especially HTML pages or very small .exe files - yeah those little tricky .exe's that open your CD drawer or show a film of Fred Flintstone going for gold are usually also trojans!) People don't usually spend time making these things just for entertainment value.
3. Install and maintain a quality anti-virus programme and aslo a firewall (which you already have)
4. If someone annoys you on IRC and threatens to hack you, the last thing you should say is "I have a firewall so it's impossible" - all you are doing then is giving them a challenge to overcome. Lets face it, most kiddie h4x0rs are dole bludgers and have nothing meaningful to do so they use the 'Net to stir people.
5. Don't fill your machine with wares. Things like Windows XP and Office XP that doesn't need authentication are great but what comes with them is not. Don't even get Linux distros or other freeware from IRC, get it from the official websites.

Being 06:00 I can't think of anything else at the moment (all I can think about is eating brekky).

I'll mention one other thing, if you use an IRC network that doesn't mask your IP then you are throwing out the welcome mat to hackers. I recommend using a network that does mask your IP for you which also stops DNS lookups on a nickname (on my network anyway).