While the returned values are like expected if used in the command part, $N indeed returns "unexpected" values if used in the menuitem definition:
Code:
menu nicklist,channel {
  Test (selected $snick(#,0) nicks)
  .$ $+ 0 = $0 : echo -ag $ $+ 0 = $0
  .$ $+ 1 = $1 : echo -ag $ $+ 1 = $1
  .$ $+ 2 = $2 : echo -ag $ $+ 2 = $2
}
- If no nick is selected, shows "$0 = 1" but echoes "$0 = 0"
- If two nicks are selected, shows "$0 = 1" but echoes "$0 = 2"
- If two nicks are selected, it shows "$2 =" but echoes "$2 = <2nd selected nick>"

The examples of the helpfile don't use $N in the menuitem definition at all, and I'm not sure what the "correct" behaviour is: $N's in the menuitem definition are used in $submenu calls, maybe that's why $0 is evaluated to 1 and $2 is empty.

To fix your particular script, use:
$snicks (a comma-delimited string of all selected nicknames of the active channel)
$snick(#,N) (the Nth selected nick of channel #; if N = 0 the total number of selected nicks. If you dont specify N it returns a string like $snicks)