Forums Developers How to make my DLL change control's properties

That will likely have no effect at all. Reading MSDN:

The WM_CTLCOLORBTN message is sent to the parent window of a button before drawing the button. The parent window can change the button's text and background colors. However, only owner-drawn buttons respond to the parent window processing this message.

Note the "owner-drawn buttons", the buttons in mIRC, to my knowledge, are not owner drawn. So that means they would ignore this. What you likely need to do is get the DC of the control and modify it like that.