Forums Bug Reports Binary Vars BUG

well, after khaled makes a utf-8 support this bug starts to show

//bset -t &x 1 á | echo -a $bvar(&x,0)
must return 1, but it's returning 2

thanks!

It returns 2 because it takes 2 bytes to represent that character in UTF-8. Binary variables operate on bytes, not characters.

You can use $len($bvar(&x,1-).text) if you want to know the number of characters it contains (and the binvar contains valid UTF-8 of course).

It's not a bug. The character "á" is represented with two bytes in UTF-8.

K this is just $&*&! up...

$asc(á) = 225
$chr(225) = á

It's one byte...

It's not one byte. Please learn about Unicode.

$chr() now returns Unicode codepoints. á is represented by codepoint U+00E1 or 225. As shown by the page, the UTF-8 encoding of this codepoint is 2 bytes long:

UTF-8 (hex) 0xC3 0xA1

Exactly what you get when you use it:

There's a good chart here that explains how each "code point" (in this case, 225 or U+00E1) is translated into raw byte(s).

The result is all characters between $chr(128) and $chr(2047) take up two bytes each.

They should both be two bytes... or one byte... I haven't check lately but mIRC does not properly read UTF-16 encoded files since the last time I checked... big Endian or Little Endian... has this changed?

Who said anything about UTF-16? mIRC uses UTF-8. Using UTF-16 would be pointless because it's a fixed-width encoding that uses null-bytes as filler, thereby making it unusable over IRC.

No they shouldn't

mIRC is using UTF-8 to encode the characters you list with /bset which is why you aren't getting two bytes for both. If it used UTF-16 then you'd be right.

Using $read on three files with identical text -- stored in UTF-8, UTF-16 LE, and UTF-16 BE -- they all read just fine. As far as binary variables and /bread goes though, it works fine but it's just reading raw bytes -- if the data happens to be Unicode text, you would have to determine which encoding the text is on your own.

Edit: By the way, $bvar().text only works correctly with UTF-8 encoded text.

Uh, binary variables can store any binary data, not just text. It doesn't even make sense to treat all binary data as characters.

Why exactly are you replying to me ?
My post agrees with what you and others said previously and emphasizes on the fact that binary variables works on bytes, not caracters, which is why :
is wrong

Haha, sorry, I misunderstood your previous post. It looked like you were stating "No they shouldn't" in response to "Binary variables operate on bytes, not characters." (But I realize now that's not what you meant.)

I was confused by this too.

I'm unable to read "layout.ini" in the \Windows\Prefetch\ directory (it's currently the only UTF-16 text file I've got access to atm). Using the standard $read function.

I'll do some more testing later... but it doesn't properly treat UTF-16 text files as it should...

*** Does not read UTF-16 files with no BOM...

And how would mIRC even know the file is UTF-16 encoded if there is no BOM?

There are plenty of UTF-16 files with no BOM, and the spec explains how to deal with such files:

From wikipedia:


Therefore mIRC should be able to handle these files; how to detect them is an interesting question, but not really our concern. In other words, whether mIRC performs auto-detection or we have to tell it manually isn't really the issue. There needs to be a way to access these files.

I would consider it a bug if mIRC is unable to read non-BOM UTF-16 files given that it is supposedly "unicode compatible"-- at worst, it should be a feature suggestion. Perhaps a new switch on $read and its counterparts to specify file encodings (which mIRC badly needs!) would help.

That said, this topic is totally tangential to binary vars, and we should open a separate discussion for the issue.

It's not that clear cut unfortunately, as explained by the next sentence from that Wikipedia article:



For this reason, it's common for UTF-16 files without a BOM to be LE, not BE, if they originated from a Windows program.



I generally don't like the concept of auto-detection. I'm sure you've seen the Notepad trick where you save some text, close and reopen the file, and the text becomes gibberish. UTF-8 is much more commonly used in text files anyway, so it should only be an issue in a small number of cases. However, I think it would be a good idea to add a switch to force mIRC to use a particular encoding when reading a text file.

No, the specification is very clear cut. The behaviour of certain programs is incorrect, but that shouldn't really matter. The point is that mIRC should [at least] support the specification. The other point is, again, this discussion should be in another thread. BOMs and UTF-16 support have nothing to do with this issue.

It has been a while since I researched this but if I recall correctly, the only way to determine the encoding format of a file without a BOM is to scan the contents of the file and to analyze it.

In the case of a text file that uses UTF-16, BE or LE, and is only storing characters from the Basic Latin and Latin-1 Supplement of the Unicode table, you can check for alternating zero bytes. You will need to load the file, check for the BOM and if it does not exist, continue reading the file as single-byte characters. If you come across a zero byte, the file is probably UTF-16 and you will need to start reading the file again from the beginning as UTF-16 (whether it is BE or LE depends on where the zero byte came in the sequence).

If there is an error in the file and the zero byte is not meant to be there (or if it was ANSI text saved with zero byte separators on purpose for some reason), loading it as UTF-16 results in garbage.

If there are no zero bytes, you will not be able to determine whether it is ANSI or UTF-16.

For example, if you come across a UTF-16 file with no BOM that contains thousands of characters in the range 0x0101 to 0x017F, you will not be able to tell whether it is ANSI or Unicode. However if it contains any character that has a zero byte (in the range 0x0000 to 0x00FF, or 0x0100 etc.) you can assume it is UTF-16.

While the above method will work for some text files (and mostly only for Latin text), I felt it was a little too unreliable and limited in scope, which is why I decided not to add support for it.