Forums Bug Reports $remtok bug

$remtok(abcd,xyz,1,32) == abcd
var %a = xyz
$remtok(abcd,%a,1,32) == abcd
$remtok(abcd,%b,1,32) == $null
var %b
$remtok(abcd,%b,1,32) == $null
$remtok(abcd,,1,32) == $null
$remtok(abcd,$null,1,32) == $null

If the second parameter of $remtok is an unassigned variable, an empty variable, an empty parameter, or a $null parameter... then the $remtok returns $null even though the string of tokens doesn't contain any $null to be removed.

In every case above, $remtok should have returned abcd.

---

Given this bug, $remtok cannot be used without $iif.

$iif($remtok(%a,%b,1,32),$v1,%a)

Thanks for your bug report. There have been a number of requests over the years to change how the token identifiers work, unfortunately changing them now would break many scripts.

The behaviour you describe applies to other token identifiers as well. If a change was made to $remtok() to change its behaviour, the same change would really need to be applied to other token identifiers for consistency, which would increase the likelihood of breaking scripts.

That said, I will experiment with this change (applied to all of the token identifiers) to see how it works out with my token test scripts.

Ok,

$instok(a b c d,,1,32) == $null (should return "a b c d"?)
$puttok(a b c d,,1,32) == $null (should return "b c d"?)
$reptok(a b c d,a,,1,32) == $null (should return "b c d"?)

but

$addtok(a b c d,,32) == a b c d
$deltok(a b c d,,32) == a b c d

It doesn't seem very consistent behavior, given these functions all modify the token string (i excluded the identifiers that just retrieve, count or search).

Yes, it's not pretty. Looking through the token identifiers code, there is a lot that I would like to change about them but it just isn't possible without breaking scripts. Even small changes that seem innocuous can have major side-effects. As you noted, the $instok(), $puttok(), and $reptok() identifiers would all need to be changed in this case - and $remtok() I think. Their behaviours are also a bit murky if other eg. position or separator parameters are left out. Ideally, they should have halted the script in such cases. Some of the identifiers also contain legacy code that needs to remain in place to make them compatible with previous incarnations of an identifier, where it was later extended in some way. Tricky.

Anyway, I have made changes that allow some parameters to be empty. Multiple test cases for each identifier, using combinations of empty values for all parameters, show that results have only changed for the following parameter combinations:

7.38: $instok(a b c d,,1,32) ==
beta: $instok(a b c d,,1,32) == a b c d

7.38: $puttok(a b c d,,1,32) ==
beta: $puttok(a b c d,,1,32) == b c d

7.38: $remtok(a b c d,,1,32) ==
beta: $remtok(a b c d,,1,32) == a b c d

7.38: $reptok(a b c d,d,,1,32) ==
beta: $reptok(a b c d,d,,1,32) == a b c

7.38: $reptok(a b c d,,e,1,32) ==
beta: $reptok(a b c d,,e,1,32) == a b c d

I will include these changes in the public beta for testing and if they look okay, they will be included in the final release.

Glad to hear you would like to change those things
Now, about doing it, I think it's important to note that while you're potentially preserving scripts by keeping some behaviors, you're actually at the same time preventing real new users from using the correct behavior (that is, the behavior which makes more sense when the original intended behavior is unclear). I think that functionnality should be top priority. I think that whenever you have no functionality in a current behavior, it should be fixed because it simply can't break scripts, if it does, it means the script was broken in the first place, that it used a flawed logic. There can't be any script out there using $remtok(list,$null,C) and expect to get $null because there is no functionality in that, they could just use $null if that's what they wanted to get.
Similar example would be not fixing the 'nested while loop on one line' bug, meaning you're thinking there is a script out there actually using that behavior on purpose to get the outer loop to be entirely skipped over..