In Windows, applications can be compiled as single-byte ANSI or multi-byte Unicode. mIRC was originally an ANSI application. It was converted to Unicode. That is what is meant by Unicode in this context. Unicode in Windows applications is UTF-16.
however there is an option for utf8 messages
This option is present because some users requested the ability to disable Unicode/UTF-8 conversion in incoming/outgoing messages.
DLL interface passes data over as a char *
DLL support was added long ago. It supports char* for backwards compatibility. When a DLL is called, mIRC automatically converts parameters/results to/from Unicode/UTF-8.