Forums Bug Reports Binary Vars BUG

well, after khaled makes a utf-8 support this bug starts to show

//bset -t &x 1 á | echo -a $bvar(&x,0)
must return 1, but it's returning 2

thanks!

It returns 2 because it takes 2 bytes to represent that character in UTF-8. Binary variables operate on bytes, not characters.

You can use $len($bvar(&x,1-).text) if you want to know the number of characters it contains (and the binvar contains valid UTF-8 of course).

It's not a bug. The character "á" is represented with two bytes in UTF-8.

K this is just $&*&! up...

$asc(á) = 225
$chr(225) = á

It's one byte...

It's not one byte. Please learn about Unicode.

$chr() now returns Unicode codepoints. á is represented by codepoint U+00E1 or 225. As shown by the page, the UTF-8 encoding of this codepoint is 2 bytes long:

UTF-8 (hex) 0xC3 0xA1

Exactly what you get when you use it:

There's a good chart here that explains how each "code point" (in this case, 225 or U+00E1) is translated into raw byte(s).

The result is all characters between $chr(128) and $chr(2047) take up two bytes each.

They should both be two bytes... or one byte... I haven't check lately but mIRC does not properly read UTF-16 encoded files since the last time I checked... big Endian or Little Endian... has this changed?

Who said anything about UTF-16? mIRC uses UTF-8. Using UTF-16 would be pointless because it's a fixed-width encoding that uses null-bytes as filler, thereby making it unusable over IRC.

No they shouldn't

mIRC is using UTF-8 to encode the characters you list with /bset which is why you aren't getting two bytes for both. If it used UTF-16 then you'd be right.

Using $read on three files with identical text -- stored in UTF-8, UTF-16 LE, and UTF-16 BE -- they all read just fine. As far as binary variables and /bread goes though, it works fine but it's just reading raw bytes -- if the data happens to be Unicode text, you would have to determine which encoding the text is on your own.

Edit: By the way, $bvar().text only works correctly with UTF-8 encoded text.

Uh, binary variables can store any binary data, not just text. It doesn't even make sense to treat all binary data as characters.

Why exactly are you replying to me ?
My post agrees with what you and others said previously and emphasizes on the fact that binary variables works on bytes, not caracters, which is why :
is wrong

Haha, sorry, I misunderstood your previous post. It looked like you were stating "No they shouldn't" in response to "Binary variables operate on bytes, not characters." (But I realize now that's not what you meant.)

I was confused by this too.

I'm unable to read "layout.ini" in the \Windows\Prefetch\ directory (it's currently the only UTF-16 text file I've got access to atm). Using the standard $read function.

I'll do some more testing later... but it doesn't properly treat UTF-16 text files as it should...

*** Does not read UTF-16 files with no BOM...

And how would mIRC even know the file is UTF-16 encoded if there is no BOM?

There are plenty of UTF-16 files with no BOM, and the spec explains how to deal with such files:

From wikipedia:


Therefore mIRC should be able to handle these files; how to detect them is an interesting question, but not really our concern. In other words, whether mIRC performs auto-detection or we have to tell it manually isn't really the issue. There needs to be a way to access these files.

I would consider it a bug if mIRC is unable to read non-BOM UTF-16 files given that it is supposedly "unicode compatible"-- at worst, it should be a feature suggestion. Perhaps a new switch on $read and its counterparts to specify file encodings (which mIRC badly needs!) would help.

That said, this topic is totally tangential to binary vars, and we should open a separate discussion for the issue.

It's not that clear cut unfortunately, as explained by the next sentence from that Wikipedia article:



For this reason, it's common for UTF-16 files without a BOM to be LE, not BE, if they originated from a Windows program.



I generally don't like the concept of auto-detection. I'm sure you've seen the Notepad trick where you save some text, close and reopen the file, and the text becomes gibberish. UTF-8 is much more commonly used in text files anyway, so it should only be an issue in a small number of cases. However, I think it would be a good idea to add a switch to force mIRC to use a particular encoding when reading a text file.

No, the specification is very clear cut. The behaviour of certain programs is incorrect, but that shouldn't really matter. The point is that mIRC should [at least] support the specification. The other point is, again, this discussion should be in another thread. BOMs and UTF-16 support have nothing to do with this issue.

It has been a while since I researched this but if I recall correctly, the only way to determine the encoding format of a file without a BOM is to scan the contents of the file and to analyze it.

In the case of a text file that uses UTF-16, BE or LE, and is only storing characters from the Basic Latin and Latin-1 Supplement of the Unicode table, you can check for alternating zero bytes. You will need to load the file, check for the BOM and if it does not exist, continue reading the file as single-byte characters. If you come across a zero byte, the file is probably UTF-16 and you will need to start reading the file again from the beginning as UTF-16 (whether it is BE or LE depends on where the zero byte came in the sequence).

If there is an error in the file and the zero byte is not meant to be there (or if it was ANSI text saved with zero byte separators on purpose for some reason), loading it as UTF-16 results in garbage.

If there are no zero bytes, you will not be able to determine whether it is ANSI or UTF-16.

For example, if you come across a UTF-16 file with no BOM that contains thousands of characters in the range 0x0101 to 0x017F, you will not be able to tell whether it is ANSI or Unicode. However if it contains any character that has a zero byte (in the range 0x0000 to 0x00FF, or 0x0100 etc.) you can assume it is UTF-16.

While the above method will work for some text files (and mostly only for Latin text), I felt it was a little too unreliable and limited in scope, which is why I decided not to add support for it.

Well, there is no reliable way to detect the proper encoding of a UTF-16 encoded file without a BOM-- and by extension, there is no way to detect that a file is UTF-16 without a BOM (similarly, even the BOM itself isn't always a valid way to detect an encoding format).

The auto-detection method you propose would be extremely slow since it would potentially scan the entire file, and would need to do this for every file-- this would mean that for most files (non-utf16 ones) you would be always scanning ~1000 chars prior to reading-- every time! slow!

That's why I proposed an extra switch in all $read/$fread commands to force a specific encoding. It's basically impossible (impractical) to auto-detect, so the scripter should have to tell mIRC in these cases. Telling the runtime what encoding you want to read a file as is fairly common in every language with robust encoding support. It's fine to assume UTF-8 as default, and fair to allow basic auto-detection, but for certain encodings, we need a way to enforce this manually.

Note that having an encoding switch leaves room for adding back other encoding support, at least for reading such files from the FS, in the future. Not saying we should open up a debate about supporting codepages again, but it allows for this possibility if, in the future, it becomes easier support other encodings.

Basically, the encoding parameter could be undefined for any value other than utf8, ascii, utf16(bom?), utf16le and utf16be for now. In the future, you could add things like sjis, etc.

One way to add this would be to attach the parameter to the end of $read (and $fread too):

//echo -a $read(file.txt, wn, *foo*, 1, utf16le)

Since the current last param is [N] (numeric), you can autodetect when it is present, so we could also have:

//echo -a $read(file.txt, wn, *foo*, utf16le)

And of course everything should get transcoded to UTF-8 internally (or whatever mIRC's internal rep is).

/fopen should contain the encoding switch. $read can detect this automatically or at least make a good guess. $read always reads the entire file up to a specific line. $read will read the entire file if no line is specified.

There is no way to select the OUTPUT encoding with /write or /fwrite. Though with /fopen /fwrite should use the specified encoding set with /fopen.

/fopen -e <encoding> <handle> <file path>

This really needs to be posted as a feature suggestion since this is nothing at all to do with the original bug report.