heres code that should work, I dont use [ ] becuase they are really for something else, i use $+( ) to conenct the needed parts, and $( ,2) to evaluate to the vars contents
I was also a bit concerned about you switching from var %x to set %x which i wasnt sure if its going to use the local var one or the global set one, when accessing %x
Lastly i added a else { return } becuase u dont want to do the rest if the $nick doing the line isnt in the game at all.

Code:
on *:TEXT:!mmafia out:#:{
  var %x
  if (%plyr1 == $nick) { var %x = 1 }
  elseif (%plyr2 == $nick) { var %x = 2 }
  elseif (%plyr3 == $nick) { var %x = 3 }
  else { return }
  while (%x < 3) {
    set $+(%,plyr,%x) $($+(%,plyr,$calc(%x + 1)),2)
    inc %x
  }
  set $+(%,plyr,%x) none
  notice $nick You are no longer signed up.
  dec %plyrs
}

However considering your just using 3 players, i would hardcode the changes.
Code:
  on *:TEXT:!mmafia out:#:{
  if (%plyr1 == $nick) { set %plyr1 %plyr2 | set %plyr2 %plyr3 | set %plyr3 none }
  elseif (%plyr2 == $nick) { set %plyr2 %plyr3 | set %plyr3 none }
  elseif (%plyr3 == $nick) { set %plyr3 none }
  else { return }
  notice $nick You are no longer signed up.
  dec %plyrs
}