After discussing this with Saturn, it turns out the issue is related to my DLL being in the default mode, and not having the UNICODE flag set. When I had originally tried to set the unicode flag, text was output as a stream of Chinese symbols, because I didn't realize there was a -DUNICODE compiler flag I was supposed to use too.
Apparently 'unicode' is effectively a UTF16 encoding, where each character always uses 2 bytes, as opposed to UTF8 where the most common values 0-127 use 1 byte in a &binvar while the rest use either 2 or 3 depending on which range of codepoints they're in.
Since in UTF16 mode the 10340 characters always uses 10340*2 bytes, that encoding works for the current buffer's byte length. However that same number of bytes is being used by mIRC to receive 10340 character from the DLL in the default unicode=NO setting, where it's possible for the same number of characters to be as long as 10340*3 bytes, causing the GPF.
For now I need to finish debugging my DLL to make sure all the functions work now, before thinking of transitioning everything to UNICODE mode for the benefit of the one function that needs 10340*3 bytes in UTF8 mode. In Unicode mode, the UTF16 is also the input format, which means the functions trying to hash strings would be returning the hash for the UTF16 bytes instead of the UTF8 bytes, so I'd need to translate incoming data so those functions would once again return the correct answer.
