Let me map this out for you, with some very basic knowledge of how it all works.
When you go to DCC someone, it sends to that person via a notice of some sort on the IRC server your IP and your port. This special notice or CTCP or whatever it is, then triggers the other guy to say "Accept DCC from......?"
Afterwards, that person's client tries to connect to yours 'listening' on the port that you configured.
If you do not specify a correct IP, it won't work. If you do not specify an IP at all, then it's not going to work, for obvious reasons.
Not to mention, on many IRC networks (usually not smaller ones), your IP is in your hostname unless you use a bouncer program. Anyone can simply use the /dns command on your nick, and get your IP.
But for point of reference, once you direct connect that person, your IP is known. Plain and simple it's just how the internet works. You can't get around that.