%md#channel can be only in 3 states:
  • ae # -at
  • sm
  • $null
which means we can tidy up the script slightly by simply using if !$eval( ) (which is true in the $null state). Here's how I would have written the same functions (cleaning up your code slightly):
Code:

autoecho {
  if $eval($+(%,md,$chan), 2) == sm {
    echo 4 ERROR: Cannot use Auto Echo with Self-Mod on
  }
  elseif !$eval($+(%,md,$chan), 2) { 
    if !$1- {
      echo 7 -at Auto Echo ON (default params)
      set %md $+ $chan ae 14 -at 
    }
    else {
      echo 7 -at Auto Echo ON
      set %md $+ $chan ae $1- 
    }
  }
  else {
    echo 7 -at Auto Echo OFF
    unset %md $+ $chan 
  }
}
selfmod {
  if ae * iswm $eval($+(%,md,$chan), 2) {
    echo 4 ERROR: Cannot use Self-Mod with Auto Echo on
  }
  elseif !$eval($+(%,md,$chan), 2) {
    echo 7 -at Self-Mod ON
    set %md $+ $chan sm 
  }
  else {
    echo 7 -at Self-Mod OFF
    unset %md $+ $chan 
  }
}
on *:input:#:{
  if $left($1,1) != / {
    if $eval($+(%,md,$chan), 2) == sm {
      echo 4 -at SELF-MOD IS ON: 7Cannot chat in this channel
      halt
    }
    if ae * iswm $eval($+(%,md,$chan), 2) {
      echo $right($eval($+(%,md,$chan), 2), -3) $1-
      halt
    }
  }
}

Your two errors are:
  • In the /selfmod alias, $left must touch its ( .. yours had a space between them. This causes the initial condition (the ae check) to fail since "($eval($+(%,md,$chan) ,2),2)" will be treated as the string to compare to "ae" which is obviously no match and the condition fails. The elseif will fail because "%md#channel" is not $null. That leaves only the else, which correctly echo "Self-Mod off" and unset the variable.
  • In the on INPUT, you can simply use $right($eval($+(%,md,$chan), 2), -3) instead of doing all the calculations. Your calculations were completely hosed, anyway, as were your mismatched parentheses. $right($eval($+(%,md,$chan), 2), $calc($len($eval($+(%,md,$chan), 2)) - 3) would be correct, though cumbersome. Also, your $len needed a $, in addition to needing a $calc( ) to calculate everything. You could also have used $mid($eval($+(%,md,$chan), 2), 4) if that is more intuitive for you.
There's no need for a halt in the aliases, if you slightly restructure your if statements as I have done above. Once the code inside the matching if-elseif condition is completed, control drops out of the if-elseif-else control structure with no other code other than that control structure.