You need to delay by some interval to execute the check, instead of immediately checking for clones. The problem you're facing is it detects > 1 match, and deals with them. so 1-2 is already gone before you test #3, testing #3 fails because you already kick/banned 1-2, so there can't be more than 1 match...

You also have some potential exploits here, since there's no error checking on nicknames. You probably don't want to kick anyone with some form of channel access mode (op/halfop/voice/etc...) and you also should ensure that the nickname is NOT you, with the current setup, if I were to launch 3 clones with the beginning nick matching the first 5 characters of your nickname, you're also gonna kick yourself!

There's another problem here, IRCd's have a limitation on multiple modes/targets and this is defined by raw 005 MODES=X and TARGMAX=X. If you get a rather lengthly list, how you're kicking based on just a tokenized list on 44, your kick could potentially have two outcomes... miss some because the list is beyond the maximum targets allowed, or potentially ignored all-together because the targets list is too long.

IMPORTANT NOTE: As far as I know, mIRC does not have an $identifier for all things passed in 005, one of which is TARGMAX, this has a potential to BREAK because I just assumed $modespl and TARGMAX-kick are identical.

Example 005 line from an IRCd:

Here you will see a list of maximum targets for KICK is 4, yet MODES is 12. You may need to parse raw 005 yourself and log the maximum number of allowable targets for kick and change the variable: %FixMeFrom005 from "$modespl" for this value located above the last while loop of this example, or just set this to a lower number, like from this server, just set it to 4, instead of $modespl.

the targmax portion:

Here is a re-worked version that "should" do what you're after, based on what I gather from the code you posted, and the example paste of the end result you wish to see as an outcome. More than one similar nickname (first 5 characters matching) joining within 3 seconds. I also used $ialchan rather than a custom data-storage (hashtable/ini/flatfile) so there is no global variables or storage tables. For this to work, /ial MUST be on (who doesn't have this on anymore?)

NOTE: there is one potential unfortunate side-effect with the method I used (ialchan), there's no way to determine if a nicks idle was set due to a recent join, or from recently spoken (action or text) so it "could" potentially kick someone who matches the pattern, has no channel access list (op/voice/etc..) and has spoken within the 3sec timeframe of the join flood, but this "should" be a very rare occurrence.

I heavily commented what it is doing so you can follow the method on how and why it works.

on *:JOIN:#: {
  ;=== Are we some form of op or ircop?
  if ($nick(#,$me,@&~%) || o isin $usermode) { 
    ;=== Is the nickname that joined at least 5 characters?
    if ($len($nick) >= 5) { 
      ;=== Define variables for the pattern and get number of matches for pattern.
      var %pattern = $+($mid($nick,1,5),?*!*@*) , %matches = $ialchan(%pattern,#,0)
      ;=== Do we have more than one match?
      if (%matches > 1) {
        ;=== Define a variable to house our timer name, get its tokens if it exists already and append our pattern to it.
        var %timer = $+(-clones-,$cid,-,#) , %tokens = $addtok($gettok($timer(%timer).com,2-,32),%pattern,32)
        ;=== (Re)set the timer to execute in 1/4 of a second (250 ms) to filter clones with our new pattern(s).
        ; THIS is an important step! if we don't delay, we could miss a few! say 3 join in succession, this will trigger on #2 (> 1)
        ; which will kick 1-2, which means next join trigger on #3 will only have 1 match because we already dealt with #1-2!
        ; by simply delaying a little bit we can nab rapid succession joins that match all in one go!
        $+(timer,%timer) -m 1 250 FilterClones # %tokens
alias -l FilterClones {
  var %chan = $1 , %patterns = $2- , %total = $numtok(%patterns,32)
  ;=== Are we still some form of op or ircop?
  if ($nick(%chan,$me,@&~%) || o isin $usermode) { 
    ;=== Iterate over our passed patterns
    var %x = 0
    while (%x < %total) {
      inc %x
      ;=== Define variables for our pattern, and number of pattern matches
      var %pattern = $gettok(%patterns,%x,32) , %matches = $ialchan(%pattern,%chan,0)
      ;=== Do we have more than one match?
      if (%matches > 1) {
        ;=== clear out results/bans (potential leftover from last pattern) and iterate over our matches
        var %y = 0 , %result = $null , %bans = $null
        while (%y < %matches) {
          inc %y
          ;=== Define variables for the address, nickname, and the nicknames idle time.
          var %address = $ialchan(%pattern,%chan,%y) , %nick = $ialchan(%pattern,%chan,%y).nick , %idle = $nick(%chan,%nick).idle
          ;=== Is our matching nick NOT some form of access level or $me, and idle for <= 3 seconds? If so they either 
          ; just joined, or spoken. Add to results/bans lists. NOTE: We probably don't want nicks that just spoken, but this is
          ; an unfortunate side-effect of this method that should be a rare case, they must match the pattern, and 
          ; have spoken within the 3 sec timeframe of the join flood.
          if (!$nick(%chan,%nick,$prefix) && %nick != $me && %idle <= 3) { var %result = $addtok(%result,%nick,44) , %bans = $addtok(%bans,$mask(%address,2),32) }
        ;=== Get the number of results
        var %results = $numtok(%result,44)
        ;=== Do we have more than one nick left that met all of the above criteria?
        if (%results > 1) {
          ;=== set mode +MR and ban the pattern
          mode %chan +MRb %pattern
          ;=== set a timer to unlock modes "MR" in 5 seconds
          $+(.timer,-unlock-,$cid,-,%chan) 1 5 mode %chan -MR
          ;=== Iterate bans by modespl and ban them
          while ($gettok(%bans,$+(1-,$modespl),32)) { 
            mode %chan $+(+,$str(b,$numtok($v1,32))) $v1
            var %bans = $gettok(%bans,$+($calc($modespl +1),-),32)
          ;=== Iterate nicks by TARGMAX-kick and kick them
          var %FixMeFrom005 = $modespl
          while ($gettok(%result,$+(1-,%FixMeFrom005),44)) { 
            kick %chan $v1 IDenticaL NICK(s)
            var %result = $gettok(%result,$+($calc(%FixMeFrom005 +1),-),44)