You need a way to exit your loop, no matter what style of looping you choose to do.
Example (using your code from above rather than rewriting with WHILE):
alias massdeop {
:begindeop
unset %nicks
if ($nick($$1,0,o) > 1) {
var %i = 0
:checkdeop
inc %i
[color:red]if ($nick($$1,0,o) < %i) { halt }[/color]
set %curnick $nick($$1, $+ %i $+ ,o)
if (%curnick != $me) { set %nicks $addtok(%nicks, $+ %curnick $+ ,32) }
if ($numtok( $+ %nicks $+ ,32) == 3) { .mode $$1 -ooo %nicks | goto begindeop }
else { goto checkdeop }
}
}
Note that I removed the else { goto end } and :end parts at the end of your script. They do nothing for you.
Also, note that you can do the same addition in a while loop.
*EDIT* Here it is with a WHILE loop... I couldn't just leave it with GOTO.

alias massdeop {
var %c = 1
var %i = $nick($$1,0,o)
while (%c <= %i && %i > 1) {
if ($nick($$1,%c,o) != $me) {
var %nicks = %nicks $nick($$1,%c,o)
}
if ($numtok(%nicks,32) == 3) {
.mode $$1 -ooo %nicks
unset %nicks
}
inc %c
}
if (%nick != $null) {
.mode $$1 -oo %nicks
}
}
Note that this one also will deop the remaining ops. You previous code will deop groups of 3, but if 1 or 2 remain in %nicks at the end, you wouldn't deop them.