mIRC Home    About    Download    Register    News    Help

Topic Options
#227897 - 27/11/10 06:24 PM $gettok fails with empty tokens
Tuxman Offline
Ameglian cow

Registered: 24/12/05
Posts: 28
We noticed that $gettok() does not actually like empty tokens.

//echo -ag $gettok(1#2#3#4#5#6#7,3,35)
3

//echo -ag $gettok(1##3##5##7,3,35)
5

OK, this MIGHT be correct in a way, as 5 is the third "non-empty" token. However, sometimes there are reasons to take the third token regardless of its contents, so in this case mIRC does not seem to provide a way.

I would consider it a bug, as it does not work as pronounced in the /help.
_________________________
Gamers.IRC team - gamersirc.net
#Gamers.IRC on QuakeNet (sometimes we're there).

Top
#227898 - 27/11/10 07:15 PM Re: $gettok fails with empty tokens [Re: Tuxman]
Wims Offline
Planetary brain

Registered: 31/07/06
Posts: 3496
Loc: France
This has been discussed in the past, this is because mIRC use the C strtok() function internally.
The help file could precise that nul tokens are ignored, however the behavior is intended.
Quote:
sometimes there are reasons to take the third token regardless of its contents, so in this case mIRC does not seem to provide a way
True, but for these specials cases you could make your own gettok function, you could also suggest a $prop for all $*tok function to keep nul token

edit : quick regex version that handle nul tokens :

Code:
alias gettokn var %pattern /((.*?) $+ $chr($$3) $+ ){ $+ $2}/ | noop $regex($1,%pattern) | return $regml(2)



Edited by Wims (27/11/10 07:46 PM)
_________________________
Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net

Top
#227899 - 27/11/10 07:22 PM Re: $gettok fails with empty tokens [Re: Tuxman]
drum Offline
Fjord artisan

Registered: 08/12/02
Posts: 339
The help file doesn't actually define what a "token" is. This is the way mIRC has always worked so don't expect it to change.

Top
#227900 - 27/11/10 07:34 PM Re: $gettok fails with empty tokens [Re: drum]
Tuxman Offline
Ameglian cow

Registered: 24/12/05
Posts: 28
Originally Posted By: drum
The help file doesn't actually define what a "token" is.

So it is rather safe to assume that null tokens are tokens, too, right?
Hm, I don't care how this is solved, a prop would be OK, but I think it should. smile
_________________________
Gamers.IRC team - gamersirc.net
#Gamers.IRC on QuakeNet (sometimes we're there).

Top
#227901 - 27/11/10 07:58 PM Re: $gettok fails with empty tokens [Re: Tuxman]
drum Offline
Fjord artisan

Registered: 08/12/02
Posts: 339
Originally Posted By: Tuxman
So it is rather safe to assume that null tokens are tokens, too, right?


No. I'm saying there are different definitions of what a token is. In mIRC's implementation, there's no such thing as a null token.

Don't get me wrong. I see the usefulness of what you're describing. I'm just explaining that it's not a "bug".


Edited by drum (27/11/10 08:04 PM)

Top
#227902 - 27/11/10 08:16 PM Re: $gettok fails with empty tokens [Re: Tuxman]
argv0 Offline
Planetary brain

Registered: 13/10/03
Posts: 3918
Loc: Montreal, QC, Canada
There's nothing to solve. This is also the basic reason why multiple spaces don't work. If mIRC assumed empty tokens were tokens, then input like:

"the    quick brown fox"

Would break scripts expecting $2 to be "quick". That is, it would break just about every script. Note that $N is just shorthand for $gettok($1-,N,32). This is how mIRC parses out $1- internally for every command and identifier, it's a design feature of the language, and therefore is not a bug and probably not going to change.

You're free to write your own tokenization function that handles empty tokens.

edit: Here's one for you:

Code:
alias gettokex {
  var %text = $1, %n = $2, %delim = $chr($3)
  var %count = $calc($count(%text,%delim) + 1)
  if (%n == 0) return %count
  if (%n < 0) %n = $calc(%count + %n)
  if (%n > %count) return
  var %from = $iif(%n == 1,1,$calc($poscs(%text,%delim,$calc(%n - 1)) + 1))
  var %to = $poscs(%text,%delim,$iif(%n == 1,1,%n))
  if (%from == $null) %from = $calc($poscs(%text,%delim,-1) + 1)
  if (%to == $null) %to = $calc($len(%text) + 1)
  var %diff = %to - %from
  if (%diff == 0) return
  return $mid(%text,%from,%diff)
}


//echo -a $gettok(foo::bar,1,58)
foo
//echo -a $gettok(foo::bar,2,58)
(tries to echo empty string)
//echo -a $gettok(foo::bar,3,58)
bar

or

//echo -ag $gettokex(1##3##5##7,3,35)
3


Edited by argv0 (27/11/10 08:44 PM)
_________________________
- argv[0] on EFnet #mIRC
- "Life is a pointer to an integer without a cast"

Top
#227905 - 27/11/10 09:26 PM Re: $gettok fails with empty tokens [Re: Tuxman]
DJ_Sol Offline
Hoopy frood

Registered: 04/01/07
Posts: 1156
Loc: Seattle
Solve it by storing data correctly & responsibly. If I find that a list I gather dynamically can return a null value I have to script for that. If I am storing values that contain a period, I know not to use $chr(46) for a delimiter.

Top
#227908 - 27/11/10 10:41 PM Re: $gettok fails with empty tokens [Re: argv0]
Tuxman Offline
Ameglian cow

Registered: 24/12/05
Posts: 28
Ah. I tried and failed to build something like that with Regex.
Thank you very much then, argv0! smile
_________________________
Gamers.IRC team - gamersirc.net
#Gamers.IRC on QuakeNet (sometimes we're there).

Top
#227909 - 27/11/10 10:43 PM Re: $gettok fails with empty tokens [Re: DJ_Sol]
Tuxman Offline
Ameglian cow

Registered: 24/12/05
Posts: 28
Originally Posted By: DJ_Sol
Solve it by storing data correctly & responsibly. If I find that a list I gather dynamically can return a null value I have to script for that. If I am storing values that contain a period, I know not to use $chr(46) for a delimiter.

I often use comma-separated strings for storing lists (like: channel passwords and stuff), so something they might be null. I just never thought about what happens when I want to work around that.
_________________________
Gamers.IRC team - gamersirc.net
#Gamers.IRC on QuakeNet (sometimes we're there).

Top
#227910 - 28/11/10 01:09 AM Re: $gettok fails with empty tokens [Re: Tuxman]
Wims Offline
Planetary brain

Registered: 31/07/06
Posts: 3496
Loc: France
Quote:
I tried and failed to build something like that with Regex.
I edited my post with a regex exemple after submitting, I guess both of you didn't see it.
_________________________
Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net

Top
#227912 - 28/11/10 06:10 AM Re: $gettok fails with empty tokens [Re: Wims]
Tuxman Offline
Ameglian cow

Registered: 24/12/05
Posts: 28
Indeed, I missed it. smile
Thank you for the hint.
_________________________
Gamers.IRC team - gamersirc.net
#Gamers.IRC on QuakeNet (sometimes we're there).

Top
#232851 - 27/06/11 02:11 AM Re: $gettok fails with empty tokens [Re: Tuxman]
Tuxman Offline
Ameglian cow

Registered: 24/12/05
Posts: 28
Seems like mIRC 7 broke the $gettokex alias from above?

Quote:
//echo -ag $gettokex(1##3##5##7,3,35)
1##3##5##7 3 35


Wtf.
_________________________
Gamers.IRC team - gamersirc.net
#Gamers.IRC on QuakeNet (sometimes we're there).

Top
#232852 - 27/06/11 02:45 AM Re: $gettok fails with empty tokens [Re: Tuxman]
argv0 Offline
Planetary brain

Registered: 13/10/03
Posts: 3918
Loc: Montreal, QC, Canada
It returns the same value as I originally posted for me in the latest mIRC. I think you should check that you copied the alias properly, and don't have any other versions of it lying around.
_________________________
- argv[0] on EFnet #mIRC
- "Life is a pointer to an integer without a cast"

Top
#232855 - 27/06/11 04:14 AM Re: $gettok fails with empty tokens [Re: Tuxman]
Tuxman Offline
Ameglian cow

Registered: 24/12/05
Posts: 28
Weird. Got to reinvestigate then...
_________________________
Gamers.IRC team - gamersirc.net
#Gamers.IRC on QuakeNet (sometimes we're there).

Top
#232865 - 27/06/11 09:11 PM Re: $gettok fails with empty tokens [Re: Tuxman]
Jigsy Offline
Hoopy frood

Registered: 18/11/04
Posts: 798
Loc: I live inside your computer. S...
qwerty posted this one a few years back:

Code:
alias gettokn {
  var %c = \x $+ $base($3,10,16,2)
  returnex $remove($gettok($regsubex($1,/(?<=^| %c )(?=$| %c )/gx,$lf),$2,$3),$lf)
}


Works afaik in 7.19.
_________________________
This signature is currently out of order. We apologize for the inconvenience.

Top
#232870 - 28/06/11 09:03 AM Re: $gettok fails with empty tokens [Re: Jigsy]
jaytea Offline
Fjord artisan

Registered: 23/02/06
Posts: 546
nice! might i suggest an expansion on it to support all token identifiers, as well as including a thing or two for exceptional cases and generality, given that we can now use delimiters that occupy multiple bytes in UTF-8:

Code:
/*
{ -------------------------------------------------------------------------------
 
  $tok(<name>, <text>, [, parm, parm, ...])[.cs]
 
  Returns the value of a call to $<name>tok[cs](<text>, [, parm1, parm2, ...])
  but acknowledges null tokens both in <text> and appropriate parms.
  
  Use the .cs property to call the case sensitive version of the token identifier.
 
  Examples:
 
  $tok(num, a..b..c, 46) = 5
  $tok(rem, a..b..c, , 2, 46) = a..b.c
  $tok(put, a..c, b, 2, 46) = a.b.c
  $tok(rep, a..c, c, , 46) = a..
  $tok(find, .a.A, A, 46).cs = 3
 
  Note that if you plan to use this for sorting, the randomly selected placeholder,
  chosen from the 256-55295 code point range will result in null tokens being placed 
  after characters in the typically used 1-255 range. 
 
} -------------------------------------------------------------------------------
*/
 
alias tok {
  while ($chr($rand(256, 55295)) isin $2) /
  var %marker = $v1 | ; random placeholder
 
  var %C = $iif($1 == sort, $3, $eval($ $+ $0, 2))
  if (%C !isnum) return
 
  var %i = 3, %parms, %delim = $+(\Q, $chr(%C), \E), %cs
  if ($prop == cs) && ($istok(add find is match rem rep sort wild, $1, 32)) %cs = cs 
 
  if ($0 > 2) && ($3 == $null) && ($istok(add find ins is put rem rep wild, $1, 32)) {
    %i = 4
    %parms = ,%marker
  }
 
  while (%i <= $0) {
    if (%i == 4) && ($1 == rep) && ($4 == $null) {
      %parms = %parms ,%marker
    }
    else {
      %parms = %parms ,$ $+ %i
    }
    inc %i
  }
 
  returnex $remove($eval($+($, $1, tok, %cs, ( $eval(                 $&
    $regsubex(__tokn,$2, /(?<=^| %delim )(?=\z| %delim )/gx, %marker) $&
    , 0) %parms )), 2), %marker)
}
 
/*
{ -------------------------------------------------------------------------------
 
  $tokcs(<name>, <text>, [, parm, parm, ...])
 
  Syntactic sugar for $tok().cs.
 
  For example:
 
  $tokcs(rep, A..a., a, , 46) = $tok(rep, A..a., a, , 46).cs = A...
 
} -------------------------------------------------------------------------------
*/
 
alias tokcs {
  var %i = 1, %parms
  while (%i <= $0) {
    %parms = %parms ,$ $+ %i
    inc %i
  }
  returnex $tok( [ $mid(%parms, 2) ] ).cs
}


_________________________
"The only excuse for making a useless script is that one admires it intensely" - Oscar Wilde

Top