it doesn't work because you are misusing the $rand function.
One possible solution would be:
on *:TEXT:Hi:#: var %a = $rand(1,3) | msg # $iif(%a == 1,Hi there $nick,$iif(%a == 2,Hi,Whats up $nick))
This works in two ways. If you supply it with numbers for v1 and v2, it returns a random number between v1 and v2. If you supply it with letters, it returns a random letter between letters v1 and v2.
$rand(a,z) returns a letter in the range a,b,c,...,z
$rand(A,Z) returns a letter in the range A,B,C,...,Z
$rand(0,N) returns a number in the range 0,1,2,...,N