Thanks I was testing with latest trunk release of Firefox on a clean Vista install.
Unfortunately I have not been able to resolve this issue. My guess is that it is an Internet Explorer/Vista security limitation which prevents Internet Explorer from using DDE to communciate with external applications. I tested this with custom:// protocols linked to other software and the same DDE issue occurred.
For now I have removed DDE support for irc:// links under Vista. This means that irc:// links under Vista will still launch mIRC correctly, however they will launch a new mIRC each time you click an irc:// link.