This really should be called a bug, but I think some may disagree that's it's a bug. It certainly sucks to work-around.
//tokenize 44 1,2,3 | echo -a $1 - $2 - $3 == 1 - 2 - 3
//tokenize 44 1,2,3 | echo -a $1 - $2 - $3 == 1 - 3 -
Do I really need to explain why it isn't good that /tokenize act this way? If I write a script that expects $3 to be "3", it's now $null. The reason /tokenize does this is probably linked with X>1 spaces is always one space (why do that?!?!?!).
Also but related, token handling identifers (such as $gettok), should support $null tokens for the same reason.
Naquada
