I ran into this problem earlier when somebody was trying to DCC send me a file with Japanese characters in its filename - the user had SJIS/JIS conversion enabled and I didn't, so the filename came through with several JIS sequences, each beginning with an ESCAPE ($chr(27), 0x1B) character. I was using a special DCC script at the time (which would deliberately send fewer DCC 'ACK' packets - see below) and the /fopen failed, even though I was sending the filename through $mkfn() to sanitize it.
Further testing verified that $mkfn does NOT properly replace ESCAPE characters with underscores (as it does for all other invalid characters) even though they are not valid in filenames (at least not on my Windows 2000 installation).
Regarding the DCC script - one user was apparently behind a router/firewall/NAT which was splitting the DCC packets extremely small, and it caused mIRC to completely consume my upstream just from sending the DCC acknowledge packets. To fix this, I wrote a special DCC script to force it to only send ACKs every 4096 bytes - in most cases, it didn't cause any problems since everybody sends ahead (without waiting for ACKs) nowadays.