mIRC Home    About    Download    Register    News    Help

Topic Options
#144427 - 10/03/06 08:31 AM Filter vs $read - fastest?
Abe Offline
Pikka bird

Registered: 09/03/06
Posts: 11
If u were to search a big text file (eg. 200k lines), what would be fastest method? filter or $read?

#144428 - 10/03/06 10:05 AM Re: Filter vs $read - fastest?
RusselB Offline
Planetary brain

Registered: 03/08/04
Posts: 7252
Loc: Ontario, Canada
It's my understanding that /filter wouldbe the faster of the two methods for this. Of course, you could set a variable then test one method, record the time required, set a second variable (could even be the same one you used the first time), test the second method, then compare the time required for the second method to the first.

#144429 - 10/03/06 04:04 PM Re: Filter vs $read - fastest?
jaytea Offline
Fjord artisan

Registered: 23/02/06
Posts: 546
do you want to count the number of matches? or just check if a string exists

the quickest equivalent to chekcing if ($read(file,w,string) != $null) is of course

//filter -ff file nul string | if ($filtered) {

i just tested the two over 500 iterations a bunch of times, and found them to be extremely close together. didnt even bother testing further, i find it safe to conclude that $read is more suitable since its the more logical method for this purpose (the method of using an invalid outfile such as nul to speed it up in this case isnt documented, and /filter is usually used to deal with multiple lines) and shorter/simpler

if you wanted to further use the line, like if ($read(...)) { commands $v1 } of course the way i used /filter wouldnt make that as easy. youd need to filter to another location, and retrieve it blabla.. youd lose time there ;D

so to summarize, id use $read when its most suitable: when im only interested in the first match. id use /filter in cases where im looking for several lines
"The only excuse for making a useless script is that one admires it intensely" - Oscar Wilde

#144430 - 10/03/06 08:04 PM Re: Filter vs $read - fastest?
FiberOPtics Offline
Hoopy frood

Registered: 05/02/04
Posts: 2019
Loc: Leuven, Belgium
You cant say that /filter is faster than $read, as it entirely depends on what your goal is.

As far as hard disk efficiency goes, $read and filter are no different. They both open and close the file once. The difference is that /filter runs through all lines of the file and dumps the matching lines to some sort of output (a file, window or alias etc).

If you know this, $read can be faster than /filter (we'd have to bench it to be sure). To illustrate this: let's say you want to return 1 line from a file that matches a certain string. $read opens the file, and checks each line to see for a match, when it finds one it returns the result. Now lets say this result was found in the middle of the file (there were 5000 lines, so the 2500th).

Compare this to what you would do with filter: First of all, you need to create a window or a file, which will hold the filtered results. There is already overhead in this which $read doesn't suffer from, as it returns the result since it is an identifier. Second, /filter goes through all lines in the file (5000 instead of 2500), even though we were already satisfied with the line at line number 2500.

The point here is: you cannot say one is faster than the other, you'd have to take into account in which setting it is to be used, and to be certain you'd even have to bench.

What you can know on the other hand, is that a /filter will become more and more preferable over $read when there are multiple $read calls necessary. That's why while loops with $read are usually frowned upon by good scripters, as they know that with each call of $read, the file is opened and closed. Whereas with /filter, the file is opened once, all lines are gone through, and then the file is closed and you can do something with the results. If I have to choose between opening/closing a file 200 times in a row and opening/closing the file just once, then I know which method I'm going to prefer.

In conclusion, if he only wants to return 1 result from a file, then $read is the way to go, and not filter, as filter implies extra code to return the result to the place where you want it.

Note that if he is going to do frequent access to this file, a much better alternative would be a hidden window where he can then use $fline on. Or a hash table, where he can use $hfind on. Note that you can't take advantage of the hashing algorithm when you use $hfind. Instead, mIRC must loop through each item doing an iswm check on each of them until it finds a match, making it pretty much the same as what $fline does. I mention this, as some people might think hash tables are a magical thing that are always fast, though the truth is it depends on what you want to do and what kind of data you are holding.

#144431 - 10/03/06 09:43 PM Re: Filter vs $read - fastest?
DaveC Offline
Planetary brain

Registered: 26/09/03
Posts: 4230
As FiberOptics has said, it really comes down to what ya doing.

If you need to just find if the words in the file then use $read(file,ntw,*word*)

If you need a total number of matches use filter -ff file nul *word* and then $filtered has the matches

If you need a list of matching lines use filter -fwc file @win *word* and then $filtered has the matches And @win holds the list (u need to make the window first window -h @win makes a nice hidden window

Heres one i sometimes need...
If you need a list of the matches AND the line numbers there on in the file then use filter -fwcn file @win *word*
This gives you the list of matching lines but at the start of each line is the linenumber its from in the file followed by a space. ie: <linenumber> <line in file>