mIRC Home    About    Download    Register    News    Help

Print Thread
Filtering problem #209707 21/02/09 11:53 AM
Joined: Sep 2003
Posts: 156
B
bleach Offline OP
Vogon poet
OP Offline
Vogon poet
B
Joined: Sep 2003
Posts: 156
Hello, I want to filter a text file that includes duplicated numbers. I wrote a module below but it works too slow. I would be pleased for better modules/ideas, thanks a lot...




Code:
alias d.detect {
  var %i = 1, %file = $+(",$sfile(*.txt,select file...,start),")
  var %o.file = $replace(%file,$remove($gettok(%file,-1,92),.txt"),$+($remove($gettok(%file,-1,92),.txt"),-ok))
  while $read(%file,%i) {  
    if ($cwnum(%o.file,$read(%file,%i)) == ok) write %o.file $read(%file,%i)
    inc %i
  }
}
alias cwnum {
  var %i = 1, %n = $gettok($1-,-1,32), %file = $remove($gettok($1-,1-,32),$gettok($1-,-1,32))
  if (!$exists(%file)) write %file ::begin::
  while $read(%file,%i) {
    if (%n == $read(%file,%i)) var %x = false
    inc %i
  }
  if (!%x) return ok
}

Re: Filtering problem [Re: bleach] #209708 21/02/09 12:48 PM
Joined: Feb 2006
Posts: 546
J
jaytea Offline
Fjord artisan
Offline
Fjord artisan
J
Joined: Feb 2006
Posts: 546
hi bleach! a lot of $read() overuse there, coupled with a bit of a "naive" algorithm, and i can totally understand the slow running time laugh

this method should prove to be substantially faster:

Code:
alias d.detect {
  var %file = $+(",$sfile(*.txt,select file...,start),")
  var %o.file = $replace(%file,.txt",-ok.txt")
  .fopen d.in %file
  if (!$ferr) {
    .fopen -o d.out %o.file
    if (!$ferr) {
      if ($hget(d.detect)) hfree d.detect
      hmake d.detect $iif($calc($lines(%file) / 10 + 1) > 10000,10000,$v1) 
      while (!$fopen(d.in).eof) {
        var %line = $encode($fread(d.in) $+ $crlf,m)
        if (!$hget(d.detect,%line)) .fwrite d.out $decode(%line,m)
        hinc d.detect %line
      }
    }
    .fclose d.out
  }
  .fclose d.in
}


that should run in time proportional to the size of the file, whereas your current alias runs several orders of magnitude slower


"The only excuse for making a useless script is that one admires it intensely" - Oscar Wilde
Re: Filtering problem [Re: jaytea] #209739 22/02/09 09:14 AM
Joined: Sep 2003
Posts: 156
B
bleach Offline OP
Vogon poet
OP Offline
Vogon poet
B
Joined: Sep 2003
Posts: 156
what can I say, I don't know. that's the best shocked . thanks a lot smile

Last edited by bleach; 22/02/09 09:16 AM.