In Windows, applications can be compiled as single-byte ANSI or multi-byte Unicode. mIRC was originally an ANSI application. It was converted to Unicode. That is what is meant by Unicode in this context. Unicode in Windows applications is UTF-16.

however there is an option for utf8 messages

This option is present because some users requested the ability to disable Unicode/UTF-8 conversion in incoming/outgoing messages.

DLL interface passes data over as a char *

DLL support was added long ago. It supports char* for backwards compatibility. When a DLL is called, mIRC automatically converts parameters/results to/from Unicode/UTF-8.