mIRC Home    About    Download    Register    News    Help

Print Thread
Page 1 of 2 1 2
Unicode in $regsubex #263162 11/06/18 01:19 AM
Joined: Jan 2004
Posts: 1,323
L
Loki12583 Offline OP
Hoopy frood
OP Offline
Hoopy frood
L
Joined: Jan 2004
Posts: 1,323
I've found that certain characters (surrogates) when passed through $regsubex do not combine as they should.

This should resolve the characters and place them next to each other to display a '10' inside a box. $replace works fine. $regsubex corrupts the string.
Code:
//echo -ag $json.unescape(\ud83d\udd1f)

//echo -ag $regsubex(aa,/(a)/gu,$chr($gettok(55357 56607,\n,32))) vs $replace(ab,a,$chr(55357),b,$chr(56607))


Code:
alias json.unescape {
  return $regsubex($1-,/\\(?:u(....)|(.))/gu,$escape.map(\t))
}
 
alias -l escape.map {
  if ($1 isalpha) return $chr(160)
  if ($1 !isalnum) return $1
  if ($base($1,16,10) > 32) return $chr($v1)
  return $chr(160)
}


Other examples, taken from https://github.com/minimaxir/big-list-of-naughty-strings/blob/master/blns.txt
Code:
0\uFE0F\u20E3 1\uFE0F\u20E3 2\uFE0F\u20E3 3\uFE0F\u20E3 4\uFE0F\u20E3 5\uFE0F\u20E3 6\uFE0F\u20E3 7\uFE0F\u20E3 8\uFE0F\u20E3 9\uFE0F\u20E3 \uD83D\uDD1F
\ud83c\uddfa\ud83c\uddf8\ud83c\uddf7\ud83c\uddfa\ud83c\uddf8 \ud83c\udde6\ud83c\uddeb\ud83c\udde6\ud83c\uddf2\ud83c\uddf8
\ud835\udce3\ud835\udcf1\ud835\udcee \ud835\udcfa\ud835\udcfe\ud835\udcf2\ud835\udcec\ud835\udcf4 \ud835\udceb\ud835\udcfb\ud835\udcf8\ud835\udd00\ud835\udcf7 \ud835\udcef\ud835\udcf8\ud835\udd01 \ud835\udcf3\ud835\udcfe\ud835\udcf6\ud835\udcf9\ud835\udcfc \ud835\udcf8\ud835\udcff\ud835\udcee\ud835\udcfb \ud835\udcfd\ud835\udcf1\ud835\udcee \ud835\udcf5\ud835\udcea\ud835\udd03\ud835\udd02 \ud835\udced\ud835\udcf8\ud835\udcf0


Last edited by Loki12583; 11/06/18 01:23 AM.
Re: Unicode in $regsubex [Re: Loki12583] #263164 11/06/18 12:06 PM
Joined: Feb 2003
Posts: 2,773
Raccoon Offline
Hoopy frood
Offline
Hoopy frood
Joined: Feb 2003
Posts: 2,773
mIRC does not uniformly support SMP (Plane 1) and above Unicode characters. You're trying to output the symbol U+1F51F "KEYCAP TEN". mIRC wasn't built for handling non-Plane 0 characters.


Well. At least I won lunch.
Good philosophy, see good in bad, I like!
Re: Unicode in $regsubex [Re: Raccoon] #263165 11/06/18 12:38 PM
Joined: Jul 2006
Posts: 3,790
W
Wims Offline
Hoopy frood
Offline
Hoopy frood
W
Joined: Jul 2006
Posts: 3,790
Assuming you are correct, how do you explain $replace (and any other area in the scripting language which are applicable) producing that character?


Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net
Re: Unicode in $regsubex [Re: Wims] #263166 11/06/18 12:52 PM
Joined: Feb 2003
Posts: 2,773
Raccoon Offline
Hoopy frood
Offline
Hoopy frood
Joined: Feb 2003
Posts: 2,773
Because it's not supported uniformly; all over the place.


Well. At least I won lunch.
Good philosophy, see good in bad, I like!
Re: Unicode in $regsubex [Re: Wims] #263167 11/06/18 04:43 PM
Joined: Apr 2004
Posts: 859
Sat Offline
Hoopy frood
Offline
Hoopy frood
Joined: Apr 2004
Posts: 859
Originally Posted By: Wims
Assuming you are correct, how do you explain $replace [..] producing [U+1F51F "KEYCAP TEN"]?

As $len on the output will show you: it does not. The resulting two surrogate halves just happen to render correctly as one character in the end, thanks to Windows rather than to mIRC. $regsubex just happens to do something special with such surrogate halves, which makes sense because the individual, isolated surrogate halves (as they are considered to be right now) are by definition not proper characters. As such, fixing $regsubex by no means makes mIRC's support for Unicode plane 1+ anywhere close to a reality. As Raccoon said, mIRC doesn't claim to support anything on that front either. So, right now, anything that does work is just a happy accident.

With that said: I for one am not at all against changing $regsubex's behavior in this regard. I just wouldn't consider it a bug..


Saturn, QuakeNet staff
Re: Unicode in $regsubex [Re: Sat] #263172 11/06/18 10:28 PM
Joined: Jul 2006
Posts: 3,790
W
Wims Offline
Hoopy frood
Offline
Hoopy frood
W
Joined: Jul 2006
Posts: 3,790
Right, the unicode article by jaytea isn't so clear about it (and it's pinned so that's kind of mIRC claims), there's nothing about this, only /var and /echo are used to produce such characters in the article and no mention that you can't use this anywhere.

That being said, if it's only the rendering function which happen to show these chars, (compared to $replace's handling being different, here), then it would also render with $regsubex, given it were correctly replacing just like $replace is.
Which makes it a bug imo, $regsubex shouldn't be fixed because it makes mIRC's support for unicode plane 1+ but because it's at least inconsistent, if not just wrong, with the way it handles surrogates.


Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net
Re: Unicode in $regsubex [Re: Loki12583] #263194 12/06/18 03:47 PM
Joined: Dec 2002
Posts: 4,954
Khaled Offline
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 4,954
$regsubex() is far more complex than $replace(). Not only does it depend on PCRE processing, it also involves making multiple conversions back and forth between Unicode and UTF-8, and making internal calls to evaluate the identifier provided.

Quote:
$regsubex(aa,/(a)/gu,$chr($gettok(55357 56607,\n,32)))

Can you describe what you are expecting to happen in each part of this call?

Re: Unicode in $regsubex [Re: Khaled] #263196 12/06/18 04:17 PM
Joined: Feb 2003
Posts: 2,773
Raccoon Offline
Hoopy frood
Offline
Hoopy frood
Joined: Feb 2003
Posts: 2,773
I rather expected your answer (multiple conversions, UTF encoding, double-byte string type limitations).

I wonder if it's possible to identify an unpaired surrogate, wait for other surrogates, then process UTF on the sequence when it's completed. I think up to 8 byte characters are possible.


Well. At least I won lunch.
Good philosophy, see good in bad, I like!
Re: Unicode in $regsubex [Re: Loki12583] #263199 13/06/18 11:17 AM
Joined: Feb 2006
Posts: 546
J
jaytea Offline
Fjord artisan
Offline
Fjord artisan
J
Joined: Feb 2006
Posts: 546
this result isn't actually limited to $regsubex; it affects all functions in mIRC that implicitly decode UTF-8, eg:

Code:
//bset -t &a 1 $chr($base(D800, 16, 10)) | echo -a $len($bvar(&a, 1-).text)


= 3

the internal UTF-8 decoding function won't touch unpaired surrogates. would tweaking this be encroaching on violating the sanctity of unicode? clearly there is invalid UTF-8 being represented at some level, so perhaps having it decoded as well as possible isn't such a tall order? laugh

btw, $regsubex() needs to encode (and later decode) the substitution parm in order to play nice with offset positions returned by PCRE (which only handles UTF-8 encoded strings). this seems necessary, and the observed bug is an unfortunate side effect.


"The only excuse for making a useless script is that one admires it intensely" - Oscar Wilde
Re: Unicode in $regsubex [Re: Khaled] #263200 13/06/18 12:40 PM
Joined: Jul 2006
Posts: 3,790
W
Wims Offline
Hoopy frood
Offline
Hoopy frood
W
Joined: Jul 2006
Posts: 3,790
We're expecting to get $chr(55357), first surrogate, to be replaced by the first replacement of 'a', and $chr(56607), second surrogate, to be replaced by the second replacement of 'a', having them both returned from $regsubex, they would form the code point when being rendered with /echo, just like with $replace.
Not sure how $replace works but it looks like that when $regsubex adds to the final string to be returned when substituing, it checks for surrogate, something $replace isn't doing (and I'm assuming most others functions will behave like $replace).
Which one is correct? are they both correct?


Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net
Re: Unicode in $regsubex [Re: Wims] #263207 14/06/18 10:53 AM
Joined: Dec 2002
Posts: 4,954
Khaled Offline
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 4,954
They are both correct.

$replace() performs an in place exchange of characters.

$regsubex() performs a more complex processing of the text that involves conversion back and forth between UTF-8 and Unicode, which involves checks on the validity of the encoding.

A possible solution might be to change mIRC to use Unicode PCRE instead of ANSI PCRE which would mean that UTF-8 conversions would not be needed. However, it is not clear how this would affect existing scripts that pass UTF-8/Unicode in strings to $regsub/$regsubex/etc.

Re: Unicode in $regsubex [Re: Khaled] #263211 14/06/18 03:48 PM
Joined: Feb 2003
Posts: 2,773
Raccoon Offline
Hoopy frood
Offline
Hoopy frood
Joined: Feb 2003
Posts: 2,773
I would attempt to post some examples on here, but this forum destroys unicode characters.

I will note however that attempting this same trick of UTF-8 Plane 0 surrogate pairs behaves he same way, and requires an extra pass of $utfdecode() wrapped around the $regsubex().

We might be able to fix consistency by enabling $utfdecode() to support Plane 1,2,3...

I do use $regsubex() to dice up BYTES regardless of encoding, so that I can handle them as BYTES.


Well. At least I won lunch.
Good philosophy, see good in bad, I like!
Re: Unicode in $regsubex [Re: Khaled] #263213 15/06/18 08:47 AM
Joined: Dec 2002
Posts: 4,954
Khaled Offline
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 4,954
I have changed the regex routines to use Unicode PCRE calls. This will be in the next beta. I implemented this change as a number of #ifdefs to insert the corresponding Unicode calls/variables in various places, bypassing the need for UTF-8 conversions. So it can be reversed easily to the original ANSI PCRE.

This means that the OP's $regsubex() call now works as expected. The change also passes my current 100+ test calls to the regex identifiers, however it will need further testing to ensure backwards compatibility.

That said, this issue will actually be present throughout mIRC because back and forth UTF-8 conversions take place in many routines when switching between Unicode/ANSI. It just happens that it was not necessary with PCRE due to Unicode PCRE calls being available.

Re: Unicode in $regsubex [Re: Khaled] #263217 15/06/18 01:07 PM
Joined: Jul 2006
Posts: 3,790
W
Wims Offline
Hoopy frood
Offline
Hoopy frood
W
Joined: Jul 2006
Posts: 3,790
Note that you tried to use the utf16 lib of pcre recently and it was breaking scripts (namely, scripts which were using the (*UTF8) control verb, iirc), is this what you mean by Unicode PCRE calls?


Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net
Re: Unicode in $regsubex [Re: Wims] #263219 15/06/18 06:58 PM
Joined: Dec 2002
Posts: 4,954
Khaled Offline
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 4,954
We will have to see how it works out in the next beta.

Re: Unicode in $regsubex [Re: Khaled] #263244 19/06/18 11:20 AM
Joined: Jan 2004
Posts: 1,323
L
Loki12583 Offline OP
Hoopy frood
OP Offline
Hoopy frood
L
Joined: Jan 2004
Posts: 1,323
Forgot this item in the beta notes

The following now works as expected in the original post:


Code:
//echo -ag $json.unescape(\ud83d\udd1f)

alias json.unescape {
  return $regsubex($1-,/\\(?:u(....)|(.))/Fig,$escape.map(\1,\2))
}

alias -l escape.map {
  if ($1) return $chr($base($1,16,10))
  if ($2 !isalnum) return $2
  return $chr(160)
}

Re: Unicode in $regsubex [Re: Khaled] #263245 19/06/18 01:08 PM
Joined: Jul 2006
Posts: 3,790
W
Wims Offline
Hoopy frood
Offline
Hoopy frood
W
Joined: Jul 2006
Posts: 3,790
And as expected the change is breaking backward compatibility, (*UTF8) is now not recognized by pcre. I believe this should be implemented with a $prop: $regsubex().utf16

$regsubex(,/(.)/g,a) should be sending two bytes to pcre and since (*UTF8) is not used, two matches should happen and the output should be "aa". On the latest beta this is only "a"

Last edited by Wims; 19/06/18 01:22 PM.

Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net
Re: Unicode in $regsubex [Re: Wims] #263246 19/06/18 01:21 PM
Joined: Dec 2008
Posts: 1,511
westor Offline
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2008
Posts: 1,511
It breaking backward compatibility, i agree with Wims to add an property for that purpose.

Code:
//echo -a $regsubex(,/(.)/g,a)


Need Online mIRC help or an mIRC Scripting Freelancer? -> https://irc.chathub.org <-
Re: Unicode in $regsubex [Re: Wims] #263248 19/06/18 03:23 PM
Joined: Dec 2002
Posts: 4,954
Khaled Offline
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 4,954
Quote:
$regsubex(,/(.)/g,a) should be sending two bytes to pcre and since (*UTF8) is not used, two matches should happen and the output should be "aa". On the latest beta this is only "a"

Right, so basically there is no way to resolve this because scripters may have assumed that an accented e character will be seen as two characters in PCRE, ie. as UTF-8, instead of one character, athough I am not sure whether that is a reasonable assumption or how important it is. But for the sake of backwards compatibility I will be reversing this change in the next beta and noting the issue down in the code comments.

Re: Unicode in $regsubex [Re: Khaled] #263249 19/06/18 03:50 PM
Joined: Jul 2006
Posts: 3,790
W
Wims Offline
Hoopy frood
Offline
Hoopy frood
W
Joined: Jul 2006
Posts: 3,790
I do not agree.
on 6.x it is expected by mIRC to use the 8bits pcre lib so the byte 233 would be sent.
on 7.x mIRC still use (until this change) the 8bits lib but use utf8 so the two bytes 195 169 are passed to pcre.
This is what scripters rely on, there is no assumption as to how characters will be seen by pcre.
Using the 16bits lib is not a bad idea as it allows no conversion, (and therefore no loss of lone surrogate?), do you think adding a property to $regsubex is a bad idea?


Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net
Page 1 of 2 1 2