mIRC Homepage
Posted By: Mike2Mike Issue with large txt files - 09/12/19 12:06 PM
Hey There,

I would need some help with my textfile processing.

Following situation:
Textfile with around 20.000 lines.
(every line contains a msg message)

My script is Processing as follows:

1) read the first line from txt (with read command)
2) delete the first line from txt (with write command)
3) if the txt file is not empty start again with 1) (in a loop)
4) if the txt file is empty - halt

Issue:
I think the issue which i provide as follows comes from the txt file size.

At the beginning of the processing the script only reads 3-5 lines per second.
Later it is getting faster and faster up to 20-40 lines per second.

Is there a way to process large text files faster?
Posted By: Wims Re: Issue with large txt files - 09/12/19 12:26 PM
they are better ways to read a text file but it would depends on what you need, if you know you want to process all the line or most of them, you can use /filter, if you want some specific line(s) but only a few, File handling can be used.
Seems like you're processing all the lines, so you should check /filter: https://en.wikichip.org/wiki/mirc/commands/filter
Posted By: Dazuz Re: Issue with large txt files - 09/12/19 12:30 PM
You could just use /filter, and then /write -c to clear the file.

As an example: //filter -fk mymessages.txt myalias | write -c mymessages.txt

Or if you just want to test WITHOUT CLEARING THE FILE, try: /filter -fk mymessages.txt myalias
Code
alias myalias {
  ;$1 contains the whole line.
  echo -s Here's the whole line: $1

  ;This thing here splits the input line into tokens separated by character 32, which is a normal empty space.
  tokenize 32 $1
  echo -s There are $0 words! First word: $1 Second word: $2 Third word: $3 $+ ...
}


/help /filter
/help /write
Posted By: Mike2Mike Re: Issue with large txt files - 09/12/19 12:55 PM
Great that is what I was looking for - now I have another issue - is there a way to limit the output to 12 messages per second?
Posted By: maroon Re: Issue with large txt files - 09/12/19 03:47 PM
/timername -m 0 $calc(1000//12) alias_that_does_one_line

or you can have a timer that functions once per second and processes a burst of 12.
You could also have all the lines processed to another disk file using the complete output, then /play it with interval of $calc(1000//12) milliseconds.
Posted By: Dazuz Re: Issue with large txt files - 09/12/19 05:02 PM
Just to throw in one more option:
Code
alias mytimer {
  set -e %myvariable 1
  .timermytimer -o 0 1 myfilter
}

alias -l myfilter {
  filter -fkr $+(%myvariable,-,$calc(11+%myvariable)) mymessages.txt myalias
  if ($filtered) inc %myvariable 12
  else {
    unset %myvariable
    .timermytimer off
    write -c mymessages.txt
    echo -a Done!
  }
}

alias -l myalias {
  echo -s Here's the whole line: $1
}

/mytimer starts a timer that spits out 12 lines at a time, and clears the file after there's no more lines left.

If you stop it early and start again, it will repeat the same lines.
© mIRC Discussion Forums