You should probably be setting the instance to mIRC's hInstance. There should be a way to get that (I forget exactly how).
Anyhow, I would suggest making a non-mIRC-dll project in VS2010, just a plain old executable, and see if you can create a window there (this time you *will* need a message loop to keep the program active). If it's failing there, it's not an mIRC issue, you're simply using the CreateWindowEx function incorrectly. I can't really narrow down what you're doing wrong, since I'm not a winapi expert.
