Following up on this, I tried switching to FlashWindowEx(), which is probably what FlashWindow() calls, except that FlashWindowEx() provides extra options. Unfortunately, none of the options resolved the issue. Despite mIRC specifically setting the flash to reset or timeout, there were occasions when the button remained a solid color.
