mIRC Home    About    Download    Register    News    Help

Page 1 of 2 1 2 >
Topic Options
#263162 - 11/06/18 02:19 AM Unicode in $regsubex
Loki12583 Offline
Hoopy frood

Registered: 22/01/04
Posts: 1242
I've found that certain characters (surrogates) when passed through $regsubex do not combine as they should.

This should resolve the characters and place them next to each other to display a '10' inside a box. $replace works fine. $regsubex corrupts the string.
Code:
//echo -ag $json.unescape(\ud83d\udd1f)

//echo -ag $regsubex(aa,/(a)/gu,$chr($gettok(55357 56607,\n,32))) vs $replace(ab,a,$chr(55357),b,$chr(56607))


Code:
alias json.unescape {
  return $regsubex($1-,/\\(?:u(....)|(.))/gu,$escape.map(\t))
}
 
alias -l escape.map {
  if ($1 isalpha) return $chr(160)
  if ($1 !isalnum) return $1
  if ($base($1,16,10) > 32) return $chr($v1)
  return $chr(160)
}


Other examples, taken from https://github.com/minimaxir/big-list-of-naughty-strings/blob/master/blns.txt
Code:
0\uFE0F\u20E3 1\uFE0F\u20E3 2\uFE0F\u20E3 3\uFE0F\u20E3 4\uFE0F\u20E3 5\uFE0F\u20E3 6\uFE0F\u20E3 7\uFE0F\u20E3 8\uFE0F\u20E3 9\uFE0F\u20E3 \uD83D\uDD1F
\ud83c\uddfa\ud83c\uddf8\ud83c\uddf7\ud83c\uddfa\ud83c\uddf8 \ud83c\udde6\ud83c\uddeb\ud83c\udde6\ud83c\uddf2\ud83c\uddf8
\ud835\udce3\ud835\udcf1\ud835\udcee \ud835\udcfa\ud835\udcfe\ud835\udcf2\ud835\udcec\ud835\udcf4 \ud835\udceb\ud835\udcfb\ud835\udcf8\ud835\udd00\ud835\udcf7 \ud835\udcef\ud835\udcf8\ud835\udd01 \ud835\udcf3\ud835\udcfe\ud835\udcf6\ud835\udcf9\ud835\udcfc \ud835\udcf8\ud835\udcff\ud835\udcee\ud835\udcfb \ud835\udcfd\ud835\udcf1\ud835\udcee \ud835\udcf5\ud835\udcea\ud835\udd03\ud835\udd02 \ud835\udced\ud835\udcf8\ud835\udcf0



Edited by Loki12583 (11/06/18 02:23 AM)

Top
#263164 - 11/06/18 01:06 PM Re: Unicode in $regsubex [Re: Loki12583]
Raccoon Offline
Hoopy frood

Registered: 18/02/03
Posts: 2368
mIRC does not uniformly support SMP (Plane 1) and above Unicode characters. You're trying to output the symbol U+1F51F "KEYCAP TEN". mIRC wasn't built for handling non-Plane 0 characters.
_________________________
doiní things a particle can

Top
#263165 - 11/06/18 01:38 PM Re: Unicode in $regsubex [Re: Raccoon]
Wims Offline
Planetary brain

Registered: 31/07/06
Posts: 3367
Loc: France
Assuming you are correct, how do you explain $replace (and any other area in the scripting language which are applicable) producing that character?
_________________________
Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net

Top
#263166 - 11/06/18 01:52 PM Re: Unicode in $regsubex [Re: Wims]
Raccoon Offline
Hoopy frood

Registered: 18/02/03
Posts: 2368
Because it's not supported uniformly; all over the place.
_________________________
doiní things a particle can

Top
#263167 - 11/06/18 05:43 PM Re: Unicode in $regsubex [Re: Wims]
Sat Offline
Hoopy frood

Registered: 19/04/04
Posts: 827
Loc: The Netherlands
Originally Posted By: Wims
Assuming you are correct, how do you explain $replace [..] producing [U+1F51F "KEYCAP TEN"]?

As $len on the output will show you: it does not. The resulting two surrogate halves just happen to render correctly as one character in the end, thanks to Windows rather than to mIRC. $regsubex just happens to do something special with such surrogate halves, which makes sense because the individual, isolated surrogate halves (as they are considered to be right now) are by definition not proper characters. As such, fixing $regsubex by no means makes mIRC's support for Unicode plane 1+ anywhere close to a reality. As Raccoon said, mIRC doesn't claim to support anything on that front either. So, right now, anything that does work is just a happy accident.

With that said: I for one am not at all against changing $regsubex's behavior in this regard. I just wouldn't consider it a bug..
_________________________
Saturn, QuakeNet staff

Top
#263172 - 11/06/18 11:28 PM Re: Unicode in $regsubex [Re: Sat]
Wims Offline
Planetary brain

Registered: 31/07/06
Posts: 3367
Loc: France
Right, the unicode article by jaytea isn't so clear about it (and it's pinned so that's kind of mIRC claims), there's nothing about this, only /var and /echo are used to produce such characters in the article and no mention that you can't use this anywhere.

That being said, if it's only the rendering function which happen to show these chars, (compared to $replace's handling being different, here), then it would also render with $regsubex, given it were correctly replacing just like $replace is.
Which makes it a bug imo, $regsubex shouldn't be fixed because it makes mIRC's support for unicode plane 1+ but because it's at least inconsistent, if not just wrong, with the way it handles surrogates.
_________________________
Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net

Top
#263194 - 12/06/18 04:47 PM Re: Unicode in $regsubex [Re: Loki12583]
Khaled Offline


Planetary brain

Registered: 04/12/02
Posts: 4141
Loc: London, UK
$regsubex() is far more complex than $replace(). Not only does it depend on PCRE processing, it also involves making multiple conversions back and forth between Unicode and UTF-8, and making internal calls to evaluate the identifier provided.

Quote:
$regsubex(aa,/(a)/gu,$chr($gettok(55357 56607,\n,32)))

Can you describe what you are expecting to happen in each part of this call?

Top
#263196 - 12/06/18 05:17 PM Re: Unicode in $regsubex [Re: Khaled]
Raccoon Offline
Hoopy frood

Registered: 18/02/03
Posts: 2368
I rather expected your answer (multiple conversions, UTF encoding, double-byte string type limitations).

I wonder if it's possible to identify an unpaired surrogate, wait for other surrogates, then process UTF on the sequence when it's completed. I think up to 8 byte characters are possible.
_________________________
doiní things a particle can

Top
#263199 - 13/06/18 12:17 PM Re: Unicode in $regsubex [Re: Loki12583]
jaytea Offline
Fjord artisan

Registered: 23/02/06
Posts: 539
this result isn't actually limited to $regsubex; it affects all functions in mIRC that implicitly decode UTF-8, eg:

Code:
//bset -t &a 1 $chr($base(D800, 16, 10)) | echo -a $len($bvar(&a, 1-).text)


= 3

the internal UTF-8 decoding function won't touch unpaired surrogates. would tweaking this be encroaching on violating the sanctity of unicode? clearly there is invalid UTF-8 being represented at some level, so perhaps having it decoded as well as possible isn't such a tall order? laugh

btw, $regsubex() needs to encode (and later decode) the substitution parm in order to play nice with offset positions returned by PCRE (which only handles UTF-8 encoded strings). this seems necessary, and the observed bug is an unfortunate side effect.
_________________________
"The only excuse for making a useless script is that one admires it intensely" - Oscar Wilde

Top
#263200 - 13/06/18 01:40 PM Re: Unicode in $regsubex [Re: Khaled]
Wims Offline
Planetary brain

Registered: 31/07/06
Posts: 3367
Loc: France
We're expecting to get $chr(55357), first surrogate, to be replaced by the first replacement of 'a', and $chr(56607), second surrogate, to be replaced by the second replacement of 'a', having them both returned from $regsubex, they would form the code point when being rendered with /echo, just like with $replace.
Not sure how $replace works but it looks like that when $regsubex adds to the final string to be returned when substituing, it checks for surrogate, something $replace isn't doing (and I'm assuming most others functions will behave like $replace).
Which one is correct? are they both correct?
_________________________
Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net

Top
#263207 - 14/06/18 11:53 AM Re: Unicode in $regsubex [Re: Wims]
Khaled Offline


Planetary brain

Registered: 04/12/02
Posts: 4141
Loc: London, UK
They are both correct.

$replace() performs an in place exchange of characters.

$regsubex() performs a more complex processing of the text that involves conversion back and forth between UTF-8 and Unicode, which involves checks on the validity of the encoding.

A possible solution might be to change mIRC to use Unicode PCRE instead of ANSI PCRE which would mean that UTF-8 conversions would not be needed. However, it is not clear how this would affect existing scripts that pass UTF-8/Unicode in strings to $regsub/$regsubex/etc.

Top
#263211 - 14/06/18 04:48 PM Re: Unicode in $regsubex [Re: Khaled]
Raccoon Offline
Hoopy frood

Registered: 18/02/03
Posts: 2368
I would attempt to post some examples on here, but this forum destroys unicode characters.

I will note however that attempting this same trick of UTF-8 Plane 0 surrogate pairs behaves he same way, and requires an extra pass of $utfdecode() wrapped around the $regsubex().

We might be able to fix consistency by enabling $utfdecode() to support Plane 1,2,3...

I do use $regsubex() to dice up BYTES regardless of encoding, so that I can handle them as BYTES.
_________________________
doiní things a particle can

Top
#263213 - 15/06/18 09:47 AM Re: Unicode in $regsubex [Re: Khaled]
Khaled Offline


Planetary brain

Registered: 04/12/02
Posts: 4141
Loc: London, UK
I have changed the regex routines to use Unicode PCRE calls. This will be in the next beta. I implemented this change as a number of #ifdefs to insert the corresponding Unicode calls/variables in various places, bypassing the need for UTF-8 conversions. So it can be reversed easily to the original ANSI PCRE.

This means that the OP's $regsubex() call now works as expected. The change also passes my current 100+ test calls to the regex identifiers, however it will need further testing to ensure backwards compatibility.

That said, this issue will actually be present throughout mIRC because back and forth UTF-8 conversions take place in many routines when switching between Unicode/ANSI. It just happens that it was not necessary with PCRE due to Unicode PCRE calls being available.

Top
#263217 - 15/06/18 02:07 PM Re: Unicode in $regsubex [Re: Khaled]
Wims Offline
Planetary brain

Registered: 31/07/06
Posts: 3367
Loc: France
Note that you tried to use the utf16 lib of pcre recently and it was breaking scripts (namely, scripts which were using the (*UTF8) control verb, iirc), is this what you mean by Unicode PCRE calls?
_________________________
Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net

Top
#263219 - 15/06/18 07:58 PM Re: Unicode in $regsubex [Re: Wims]
Khaled Offline


Planetary brain

Registered: 04/12/02
Posts: 4141
Loc: London, UK
We will have to see how it works out in the next beta.

Top
#263244 - 19/06/18 12:20 PM Re: Unicode in $regsubex [Re: Khaled]
Loki12583 Offline
Hoopy frood

Registered: 22/01/04
Posts: 1242
Forgot this item in the beta notes

The following now works as expected in the original post:


Code:
//echo -ag $json.unescape(\ud83d\udd1f)

alias json.unescape {
  return $regsubex($1-,/\\(?:u(....)|(.))/Fig,$escape.map(\1,\2))
}

alias -l escape.map {
  if ($1) return $chr($base($1,16,10))
  if ($2 !isalnum) return $2
  return $chr(160)
}

Top
#263245 - 19/06/18 02:08 PM Re: Unicode in $regsubex [Re: Khaled]
Wims Offline
Planetary brain

Registered: 31/07/06
Posts: 3367
Loc: France
And as expected the change is breaking backward compatibility, (*UTF8) is now not recognized by pcre. I believe this should be implemented with a $prop: $regsubex().utf16

$regsubex(ť,/(.)/g,a) should be sending two bytes to pcre and since (*UTF8) is not used, two matches should happen and the output should be "aa". On the latest beta this is only "a"


Edited by Wims (19/06/18 02:22 PM)
_________________________
Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net

Top
#263246 - 19/06/18 02:21 PM Re: Unicode in $regsubex [Re: Wims]
westor Offline
Hoopy frood

Registered: 27/12/08
Posts: 1483
Loc: Greece
It breaking backward compatibility, i agree with Wims to add an property for that purpose.

Code:
//echo -a $regsubex(ť,/(.)/g,a)
_________________________
Need Online mIRC help or an mIRC Scripting Freelancer? -> http://westor.ucoz.com/contact <-

Top
#263248 - 19/06/18 04:23 PM Re: Unicode in $regsubex [Re: Wims]
Khaled Offline


Planetary brain

Registered: 04/12/02
Posts: 4141
Loc: London, UK
Quote:
$regsubex(ť,/(.)/g,a) should be sending two bytes to pcre and since (*UTF8) is not used, two matches should happen and the output should be "aa". On the latest beta this is only "a"

Right, so basically there is no way to resolve this because scripters may have assumed that an accented e character will be seen as two characters in PCRE, ie. as UTF-8, instead of one character, athough I am not sure whether that is a reasonable assumption or how important it is. But for the sake of backwards compatibility I will be reversing this change in the next beta and noting the issue down in the code comments.

Top
#263249 - 19/06/18 04:50 PM Re: Unicode in $regsubex [Re: Khaled]
Wims Offline
Planetary brain

Registered: 31/07/06
Posts: 3367
Loc: France
I do not agree.
on 6.x it is expected by mIRC to use the 8bits pcre lib so the byte 233 would be sent.
on 7.x mIRC still use (until this change) the 8bits lib but use utf8 so the two bytes 195 169 are passed to pcre.
This is what scripters rely on, there is no assumption as to how characters will be seen by pcre.
Using the 16bits lib is not a bad idea as it allows no conversion, (and therefore no loss of lone surrogate?), do you think adding a property to $regsubex is a bad idea?
_________________________
Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net

Top
Page 1 of 2 1 2 >