If you want your nick to be displayed as desired, switch View-Font-UTF-8 to "Default" and turn UTF-8 display in the settings. mIRC does interpret every line of text as a whole, so you can not have two different encodings in that line. Algorithm: prepare the line (concat time, nick, message), check if it is utf-8 and display it in unicode if true and in default channel encoding if false.
Mind that "піў" is windows-1251 for that unicode square. So "піўпіў" (

) is the same as these two squares in unicode; and the codes of other symbols (numbers, []'s, :'s, etc) are the same for unicode and windows-1251. Hence:
1) if you write in English (or Unicode), the whole line can be interpreted as Unicode and you get 2 squares.
2) if you write in Russian, the Unicode sequence breaks and the line of text is being shown correctly.
3) if you have your nick in, like, Russian, and the symbol combination can not be found in Unicode, and you write in English or Russian in windows-1251, then you get normal text.
4) if you have your nick like that of the nr. 3, and you type in Unicode, your nickname will break unicode sequence and you'll get a lot of garbled symbols.
I hope I made myself clear.