mIRC Home    About    Download    Register    News    Help

Print Thread
$gettok fails with empty tokens #227897 27/11/10 06:24 PM
Joined: Dec 2005
Posts: 28
T
Tuxman Offline OP
Ameglian cow
OP Offline
Ameglian cow
T
Joined: Dec 2005
Posts: 28
We noticed that $gettok() does not actually like empty tokens.

//echo -ag $gettok(1#2#3#4#5#6#7,3,35)
3

//echo -ag $gettok(1##3##5##7,3,35)
5

OK, this MIGHT be correct in a way, as 5 is the third "non-empty" token. However, sometimes there are reasons to take the third token regardless of its contents, so in this case mIRC does not seem to provide a way.

I would consider it a bug, as it does not work as pronounced in the /help.


Gamers.IRC team - gamersirc.net
#Gamers.IRC on QuakeNet (sometimes we're there).
Re: $gettok fails with empty tokens [Re: Tuxman] #227898 27/11/10 07:15 PM
Joined: Jul 2006
Posts: 3,559
W
Wims Offline
Hoopy frood
Offline
Hoopy frood
W
Joined: Jul 2006
Posts: 3,559
This has been discussed in the past, this is because mIRC use the C strtok() function internally.
The help file could precise that nul tokens are ignored, however the behavior is intended.
Quote:
sometimes there are reasons to take the third token regardless of its contents, so in this case mIRC does not seem to provide a way
True, but for these specials cases you could make your own gettok function, you could also suggest a $prop for all $*tok function to keep nul token

edit : quick regex version that handle nul tokens :

Code:
alias gettokn var %pattern /((.*?) $+ $chr($$3) $+ ){ $+ $2}/ | noop $regex($1,%pattern) | return $regml(2)


Last edited by Wims; 27/11/10 07:46 PM.

Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net
Re: $gettok fails with empty tokens [Re: Tuxman] #227899 27/11/10 07:22 PM
Joined: Dec 2002
Posts: 339
D
drum Offline
Fjord artisan
Offline
Fjord artisan
D
Joined: Dec 2002
Posts: 339
The help file doesn't actually define what a "token" is. This is the way mIRC has always worked so don't expect it to change.

Re: $gettok fails with empty tokens [Re: drum] #227900 27/11/10 07:34 PM
Joined: Dec 2005
Posts: 28
T
Tuxman Offline OP
Ameglian cow
OP Offline
Ameglian cow
T
Joined: Dec 2005
Posts: 28
Originally Posted By: drum
The help file doesn't actually define what a "token" is.

So it is rather safe to assume that null tokens are tokens, too, right?
Hm, I don't care how this is solved, a prop would be OK, but I think it should. smile


Gamers.IRC team - gamersirc.net
#Gamers.IRC on QuakeNet (sometimes we're there).
Re: $gettok fails with empty tokens [Re: Tuxman] #227901 27/11/10 07:58 PM
Joined: Dec 2002
Posts: 339
D
drum Offline
Fjord artisan
Offline
Fjord artisan
D
Joined: Dec 2002
Posts: 339
Originally Posted By: Tuxman
So it is rather safe to assume that null tokens are tokens, too, right?


No. I'm saying there are different definitions of what a token is. In mIRC's implementation, there's no such thing as a null token.

Don't get me wrong. I see the usefulness of what you're describing. I'm just explaining that it's not a "bug".

Last edited by drum; 27/11/10 08:04 PM.
Re: $gettok fails with empty tokens [Re: Tuxman] #227902 27/11/10 08:16 PM
Joined: Oct 2003
Posts: 3,918
A
argv0 Offline
Hoopy frood
Offline
Hoopy frood
A
Joined: Oct 2003
Posts: 3,918
There's nothing to solve. This is also the basic reason why multiple spaces don't work. If mIRC assumed empty tokens were tokens, then input like:

"the    quick brown fox"

Would break scripts expecting $2 to be "quick". That is, it would break just about every script. Note that $N is just shorthand for $gettok($1-,N,32). This is how mIRC parses out $1- internally for every command and identifier, it's a design feature of the language, and therefore is not a bug and probably not going to change.

You're free to write your own tokenization function that handles empty tokens.

edit: Here's one for you:

Code:
alias gettokex {
  var %text = $1, %n = $2, %delim = $chr($3)
  var %count = $calc($count(%text,%delim) + 1)
  if (%n == 0) return %count
  if (%n < 0) %n = $calc(%count + %n)
  if (%n > %count) return
  var %from = $iif(%n == 1,1,$calc($poscs(%text,%delim,$calc(%n - 1)) + 1))
  var %to = $poscs(%text,%delim,$iif(%n == 1,1,%n))
  if (%from == $null) %from = $calc($poscs(%text,%delim,-1) + 1)
  if (%to == $null) %to = $calc($len(%text) + 1)
  var %diff = %to - %from
  if (%diff == 0) return
  return $mid(%text,%from,%diff)
}


//echo -a $gettok(foo::bar,1,58)
foo
//echo -a $gettok(foo::bar,2,58)
(tries to echo empty string)
//echo -a $gettok(foo::bar,3,58)
bar

or

//echo -ag $gettokex(1##3##5##7,3,35)
3

Last edited by argv0; 27/11/10 08:44 PM.

- argv[0] on EFnet #mIRC
- "Life is a pointer to an integer without a cast"
Re: $gettok fails with empty tokens [Re: Tuxman] #227905 27/11/10 09:26 PM
Joined: Jan 2007
Posts: 1,156
D
DJ_Sol Offline
Hoopy frood
Offline
Hoopy frood
D
Joined: Jan 2007
Posts: 1,156
Solve it by storing data correctly & responsibly. If I find that a list I gather dynamically can return a null value I have to script for that. If I am storing values that contain a period, I know not to use $chr(46) for a delimiter.

Re: $gettok fails with empty tokens [Re: argv0] #227908 27/11/10 10:41 PM
Joined: Dec 2005
Posts: 28
T
Tuxman Offline OP
Ameglian cow
OP Offline
Ameglian cow
T
Joined: Dec 2005
Posts: 28
Ah. I tried and failed to build something like that with Regex.
Thank you very much then, argv0! smile


Gamers.IRC team - gamersirc.net
#Gamers.IRC on QuakeNet (sometimes we're there).
Re: $gettok fails with empty tokens [Re: DJ_Sol] #227909 27/11/10 10:43 PM
Joined: Dec 2005
Posts: 28
T
Tuxman Offline OP
Ameglian cow
OP Offline
Ameglian cow
T
Joined: Dec 2005
Posts: 28
Originally Posted By: DJ_Sol
Solve it by storing data correctly & responsibly. If I find that a list I gather dynamically can return a null value I have to script for that. If I am storing values that contain a period, I know not to use $chr(46) for a delimiter.

I often use comma-separated strings for storing lists (like: channel passwords and stuff), so something they might be null. I just never thought about what happens when I want to work around that.


Gamers.IRC team - gamersirc.net
#Gamers.IRC on QuakeNet (sometimes we're there).
Re: $gettok fails with empty tokens [Re: Tuxman] #227910 28/11/10 01:09 AM
Joined: Jul 2006
Posts: 3,559
W
Wims Offline
Hoopy frood
Offline
Hoopy frood
W
Joined: Jul 2006
Posts: 3,559
Quote:
I tried and failed to build something like that with Regex.
I edited my post with a regex exemple after submitting, I guess both of you didn't see it.


Looking for a good help channel about mIRC? Check #mircscripting @ irc.swiftirc.net
Re: $gettok fails with empty tokens [Re: Wims] #227912 28/11/10 06:10 AM
Joined: Dec 2005
Posts: 28
T
Tuxman Offline OP
Ameglian cow
OP Offline
Ameglian cow
T
Joined: Dec 2005
Posts: 28
Indeed, I missed it. smile
Thank you for the hint.


Gamers.IRC team - gamersirc.net
#Gamers.IRC on QuakeNet (sometimes we're there).
Re: $gettok fails with empty tokens [Re: Tuxman] #232851 27/06/11 01:11 AM
Joined: Dec 2005
Posts: 28
T
Tuxman Offline OP
Ameglian cow
OP Offline
Ameglian cow
T
Joined: Dec 2005
Posts: 28
Seems like mIRC 7 broke the $gettokex alias from above?

Quote:
//echo -ag $gettokex(1##3##5##7,3,35)
1##3##5##7 3 35


Wtf.


Gamers.IRC team - gamersirc.net
#Gamers.IRC on QuakeNet (sometimes we're there).
Re: $gettok fails with empty tokens [Re: Tuxman] #232852 27/06/11 01:45 AM
Joined: Oct 2003
Posts: 3,918
A
argv0 Offline
Hoopy frood
Offline
Hoopy frood
A
Joined: Oct 2003
Posts: 3,918
It returns the same value as I originally posted for me in the latest mIRC. I think you should check that you copied the alias properly, and don't have any other versions of it lying around.


- argv[0] on EFnet #mIRC
- "Life is a pointer to an integer without a cast"
Re: $gettok fails with empty tokens [Re: Tuxman] #232855 27/06/11 03:14 AM
Joined: Dec 2005
Posts: 28
T
Tuxman Offline OP
Ameglian cow
OP Offline
Ameglian cow
T
Joined: Dec 2005
Posts: 28
Weird. Got to reinvestigate then...


Gamers.IRC team - gamersirc.net
#Gamers.IRC on QuakeNet (sometimes we're there).
Re: $gettok fails with empty tokens [Re: Tuxman] #232865 27/06/11 08:11 PM
Joined: Nov 2004
Posts: 800
Jigsy Offline
Hoopy frood
Offline
Hoopy frood
Joined: Nov 2004
Posts: 800
qwerty posted this one a few years back:

Code:
alias gettokn {
  var %c = \x $+ $base($3,10,16,2)
  returnex $remove($gettok($regsubex($1,/(?<=^| %c )(?=$| %c )/gx,$lf),$2,$3),$lf)
}


Works afaik in 7.19.


What do you do at the end of the world? Are you busy? Will you save us?
Re: $gettok fails with empty tokens [Re: Jigsy] #232870 28/06/11 08:03 AM
Joined: Feb 2006
Posts: 546
J
jaytea Offline
Fjord artisan
Offline
Fjord artisan
J
Joined: Feb 2006
Posts: 546
nice! might i suggest an expansion on it to support all token identifiers, as well as including a thing or two for exceptional cases and generality, given that we can now use delimiters that occupy multiple bytes in UTF-8:

Code:
/*
{ -------------------------------------------------------------------------------
 
  $tok(<name>, <text>, [, parm, parm, ...])[.cs]
 
  Returns the value of a call to $<name>tok[cs](<text>, [, parm1, parm2, ...])
  but acknowledges null tokens both in <text> and appropriate parms.
  
  Use the .cs property to call the case sensitive version of the token identifier.
 
  Examples:
 
  $tok(num, a..b..c, 46) = 5
  $tok(rem, a..b..c, , 2, 46) = a..b.c
  $tok(put, a..c, b, 2, 46) = a.b.c
  $tok(rep, a..c, c, , 46) = a..
  $tok(find, .a.A, A, 46).cs = 3
 
  Note that if you plan to use this for sorting, the randomly selected placeholder,
  chosen from the 256-55295 code point range will result in null tokens being placed 
  after characters in the typically used 1-255 range. 
 
} -------------------------------------------------------------------------------
*/
 
alias tok {
  while ($chr($rand(256, 55295)) isin $2) /
  var %marker = $v1 | ; random placeholder
 
  var %C = $iif($1 == sort, $3, $eval($ $+ $0, 2))
  if (%C !isnum) return
 
  var %i = 3, %parms, %delim = $+(\Q, $chr(%C), \E), %cs
  if ($prop == cs) && ($istok(add find is match rem rep sort wild, $1, 32)) %cs = cs 
 
  if ($0 > 2) && ($3 == $null) && ($istok(add find ins is put rem rep wild, $1, 32)) {
    %i = 4
    %parms = ,%marker
  }
 
  while (%i <= $0) {
    if (%i == 4) && ($1 == rep) && ($4 == $null) {
      %parms = %parms ,%marker
    }
    else {
      %parms = %parms ,$ $+ %i
    }
    inc %i
  }
 
  returnex $remove($eval($+($, $1, tok, %cs, ( $eval(                 $&
    $regsubex(__tokn,$2, /(?<=^| %delim )(?=\z| %delim )/gx, %marker) $&
    , 0) %parms )), 2), %marker)
}
 
/*
{ -------------------------------------------------------------------------------
 
  $tokcs(<name>, <text>, [, parm, parm, ...])
 
  Syntactic sugar for $tok().cs.
 
  For example:
 
  $tokcs(rep, A..a., a, , 46) = $tok(rep, A..a., a, , 46).cs = A...
 
} -------------------------------------------------------------------------------
*/
 
alias tokcs {
  var %i = 1, %parms
  while (%i <= $0) {
    %parms = %parms ,$ $+ %i
    inc %i
  }
  returnex $tok( [ $mid(%parms, 2) ] ).cs
}




"The only excuse for making a useless script is that one admires it intensely" - Oscar Wilde