|
Joined: Oct 2005
Posts: 16
Pikka bird
|
OP
Pikka bird
Joined: Oct 2005
Posts: 16 |
Hey, I was just wondering if a switch can be added to the $read command so it can start reading from the bottom, all the way to the top.
I know certain tricks can be made using the /filter command, but it takes a long time if the file is big (eg 50mb's)
Thanks in advance
|
|
|
|
Joined: Sep 2003
Posts: 4,230
Hoopy frood
|
Hoopy frood
Joined: Sep 2003
Posts: 4,230 |
Be a nice idea, as you said, logs can get big, and its normally the latest occurance of something you want. I would like to see it in another identifier rather than a switch, maybe like $endread only becuase i would like to see the last line of the file be treated like its line1, 2nd to last as line2 etc, i think this would cut down of disk access as well, since the file wouldnt need to be read forward to the line number you want and then a search take place reading back again.
example -- filename.txt -- fred carpert dog marketed -- filename.txt --
$endread(filename.txt,1) => marketed : $endreadn = 1 $endread(filename.txt,w,*r*e*d*,2) => carpert : $endreadn = 3
Since i would assume the normal $read(,w,N) reads the file down to line N (lines being defined by $lf or $crlf), then searchs from there on, I wouldnt see it overly hard to read the file backwards from the end toward the begining, breaking lines when encountering $lf or $lf then a $cr (working backward). (Obviuosly i think either way actually loads blocks of the file into ram and processes it from there, rather than actually making byte by byte disk requests.)
I guess its something that could be created using custom aliases and bread's but built in would of course be faster.
|
|
|
|
Joined: May 2006
Posts: 122
Vogon poet
|
Vogon poet
Joined: May 2006
Posts: 122 |
$read is usually used in loops, why not just run the loops backwards?? instead of:
var %x = 1
while ($read(file,%x)) {
...
inc %x 1
}
use:
var %x = $lines(file)
while ($read(file,%x)) {
...
dec %x 1
}
I see what u mean Dave but I fail to see a use for it. Depends what you are using it for I guess.
Last edited by jizzy; 25/05/06 12:01 PM.
If only women came with popup menus and online help.
|
|
|
|
Joined: Feb 2004
Posts: 2,019
Hoopy frood
|
Hoopy frood
Joined: Feb 2004
Posts: 2,019 |
When you use $read with the "w, r or s" switches to scan the file, scanning starts from the beginning of the file, not the end. I think he wants to scan the other way round. If that's not what he meant, then this feature suggestion doesn't make much sense imo, as you can just loop backwards as you point out.
Btw in your while condition, you should put while ($read(file,%x) != $null) because that loop would stop once the $read evalutes to a zero, an empty line, or $false. But anyway, I wouldn't loop through a text file, that's a bad idea (continuous disk access) but rather loadbuf it into a hidden window, atleast then it's buffered in memory.
Gone.
|
|
|
|
Joined: May 2006
Posts: 122
Vogon poet
|
Vogon poet
Joined: May 2006
Posts: 122 |
Im sorry but I still don't see a use for it.
If only women came with popup menus and online help.
|
|
|
|
Joined: Feb 2004
Posts: 2,019
Hoopy frood
|
Hoopy frood
Joined: Feb 2004
Posts: 2,019 |
Because if you know that the latest occurance of somethign will be more likely to be at the end than at the top, it is much more beneficial to start scanning from the back. When you scan from the front to the end on a huge text file such as the 50mb one that the OP said, that takes much longer, because internally each line of the file is tested for a match. Imo though, this feature suggestion is a little too specific for this one feature of reading the logs, I can't see it overly useful in general.
Gone.
|
|
|
|
Joined: May 2006
Posts: 122
Vogon poet
|
Vogon poet
Joined: May 2006
Posts: 122 |
I agree I dont see any 50mb files lying on my hdd that arn't programs files or movies ect. and I certianly have no need to use $read on them.
Last edited by jizzy; 25/05/06 12:17 PM.
If only women came with popup menus and online help.
|
|
|
|
Joined: May 2006
Posts: 19
Pikka bird
|
Pikka bird
Joined: May 2006
Posts: 19 |
/last { if ($1 == $null) { %counter = 10 } else { %counter = $1 } %lines = $lines( versions.txt) while (%counter != 0) { set %displine $read( versions.txt,n,$calc(%lines - %counter)) if (%displine != $null) echo %displine dec %counter } } Of course, you can alter that to supply your own file and check it exists. Regards Nigel / Nelgin
|
|
|
|
Joined: Sep 2003
Posts: 4,230
Hoopy frood
|
Hoopy frood
Joined: Sep 2003
Posts: 4,230 |
You seem to have missed the point of the idea, if you can read from the end of the file back you dont need the HUGE level of disk accesses that would be required to read from the begining of the file.
An example Assume the file has 50Mb of data average line length 50 characters (including $crlf) so 1,000,000 lines
you want to read the last 10, so var %m = $lines(filename), %i = %m - 9 | while (%i <= %m) { echo -a . $read(filename,nt,%i) | inc %i } ok $lines takes 1,000,000 lines read to get the number of lines, %m is 1000000 and %i is 999991 $read(filename,999991) reads 999991 lines, 999992 reads 999992 lines etc to 1000000 reads 1000000 lines Total lines read 10,999,955 all to get 10 lines out.
now amagine $endread var %i = 10 | while (%i) { echo -a . $endread(filename,nt,%i) | dec %i } $endread(filename,10) read the last 10 lines, 9 last 9 , 8 last 8 etc to 1 last 1 Total lines read 55
10,999,955 vs 55 (well i know which one i would use)
* i dont make any assumptons on how the file is internally read, i know its not read line by line from the disk, but rather in blocks, but the same amount of physical disk accesses would be required as the file is read through to reach the lines requested*
|
|
|
|
Joined: Oct 2005
Posts: 16
Pikka bird
|
OP
Pikka bird
Joined: Oct 2005
Posts: 16 |
scanning starts from the beginning of the file, not the end. I think he wants to scan the other way round. That's exactly what I'm talking about. As far as loop scripts go, I'd like to see someone loop a read script on a 50mb file or bigger, the amount of time it takes to search through it is just too long (takes about a minute, if not more, depending on the PC). Also I'm a little surprised by the amount of people who suggest this feature is useless since they don't need it - we all obviously have diffrent needs, I don't use the /readini command, does it mean I have to rally against it? Thanks for the alternative suggestions on the thread so far, I'll have a look at how they work and will most likley use them for now, but a command like $endread would be very handy.
|
|
|
|
Joined: Sep 2003
Posts: 4,230
Hoopy frood
|
Hoopy frood
Joined: Sep 2003
Posts: 4,230 |
im hiding out the back office at work, hoping no one wants me for anything, so this means im also a little bored thus im gonna try and script a $endread for ya, wont be as fast as an internal command but might help you out (if i manage it) As a mater of interest what was using /filter like for performance on a 50meg file? (still to slow?)
|
|
|
|
Joined: Oct 2005
Posts: 16
Pikka bird
|
OP
Pikka bird
Joined: Oct 2005
Posts: 16 |
As a mater of interest what was using /filter like for performance on a 50meg file? (still to slow?) Thanks alot for helping /filter .txt -> @window is kinda slow on a big file, but it's the only choice available at the moment, although anything else from then takes no time at all... It'd just be convinent if I had $endread for example where I would skip quite a few steps and cpu usage from the top, and just make it //say $endread(log.txt,w,*hello*) or wherever its needed.
|
|
|
|
Joined: Jan 2003
Posts: 2,523
Hoopy frood
|
Hoopy frood
Joined: Jan 2003
Posts: 2,523 |
Signed, this would be a nice feature, one that I wanted for a long time. In the meantime, if you're interested in a scripted workaround, here it is: alias endread {
if (!$isid) return
if $0 > 2 {
if w isin $2 || r isin $2 { var %m = $v1, %l = 1, %s = $iif($4 isnum 2-,$4,1) }
else return
}
elseif $2 isnum 1- { var %m = l, %l = $2 }
else return
.fopen e $qt($1)
if ($ferr) return
var %p = $file($1) - 920, %i = 0
while %p > -920 {
if %p > 0 {
.fseek e %p
.fseek -n e
}
else .fseek e 0
var %d = $calc(920 - $fopen(e).pos + %p), %p = $fopen(e).pos
noop $fread(e,%d,&a)
var %t = $bvar(&a,1-).text, %n = $regex(%t,/\r?\n/g) + 1
if !%i && * $+ $crlf iswm %t {
%t = $left(%t,-2)
dec %n
}
if %m == l {
if %l <= %n {
.fclose e $regex(e,%t,/(?<=^|\n)(.*?)(?=$|\r?\n)/g)
return $regml(e,$calc(%n - %l + 1))
}
}
elseif %s <= $calc(%i + %n) {
var %t = $replace(%t,$crlf,$lf), $&
%j = $regex(e,%t,/(?<=^|\n)(.*?)(?=$|\r?\n)/g)
if (%m == w) var %c = $!wildtok($regml(e,%j),$3,0,0)
else var %c = $!regex($regml(e,%j),$3)
while %j {
if $calc(%i + %n - %j + 1) >= %s && $(%c,2) {
.fclose e
return $regml(e,%j)
}
dec %j
}
}
dec %p 920
inc %i $calc(%n - 1)
}
.fclose e
} Its usage is exactly like $read(), except that not all switches are supported. More specifically, w, r are supported s, p aren't supported at all n, t are implied Valid examples are: $endread(file.txt,1) = last line in the file $endread(file.txt,w,*blah*) = first line from the end that matches *blah* $endread( versions.txt,r,/mode(first|last)/,10) = line that matches "modefirst" or "modelast", but the file is searched from the 10th line from the end and upwards. Oh and don't try to use it with files that contain lines longer than 900 characters, it will break most probably. Note that even though I've tested this quite a lot, it may still have bugs. If you decide to use it, please do report back any bugs you find. EDIT: Fixed a typo in the 5th line from the endEDIT 2: Removed that line, as it's completely unnecessary... probably a leftover from early triesEDIT 3: Yet another mistake corrected... hopefully the lastEDIT 4: Wishful thinking The alias is problematic, I'm looking into it but I may miss the 2-hour-edit-time window and have to repost
Last edited by qwerty; 26/05/06 08:26 PM.
/.timerQ 1 0 echo /.timerQ 1 0 $timer(Q).com
|
|
|
|
Joined: Jan 2003
Posts: 2,523
Hoopy frood
|
Hoopy frood
Joined: Jan 2003
Posts: 2,523 |
Here's a hopefully correct version: alias endread {
if (!$isid) return
if $0 > 2 {
if w isin $2 || r isin $2 { var %m = $v1, %l = 1, %s = $iif($4 isnum 2-,$4,1) }
else return
}
elseif $2 isnum 1- { var %m = l, %l = $2 }
else return
.fopen e $qt($1)
if ($ferr) return
var %p = $file($1) - 920, %i = 0
while %p > -920 {
if %p > 0 {
.fseek e %p
.fseek -n e
}
else .fseek e 0
var %d = $calc(919 - $fopen(e).pos + %p), %p = $fopen(e).pos - 920
if !%i {
var %n = $fread(e,$calc(%d + 1),&a)
if $bvar(&a,%n) == 10 { bset &a %n 13 }
}
else noop $fread(e,%d,&a)
var %t = $bvar(&a,1-).text, %n = $regex(%t,/\r?\n/g) + 1
inc %i %n
if %m == l {
if %l <= %i {
.fclose e $regex(e,%t,/(?<=^|\n)(.*?)(?=$|\r?\n)/g)
return $remove($regml(e,$calc(%i - %l + 1)),$cr)
}
}
elseif %s <= %i {
var %j = $regex(e,%t,/(?<=^|\n)(.*?)(?=$|\r?\n)/g)
if (%m == w) var %c = $!wildtok($regml(e,%j),$3,0,0)
else var %c = $!regex($regml(e,%j),$3)
while %j {
if $calc(%i - %j + 1) >= %s && $(%c,2) {
.fclose e
return $remove($regml(e,%j),$cr)
}
dec %j
}
}
}
.fclose e
}
/.timerQ 1 0 echo /.timerQ 1 0 $timer(Q).com
|
|
|
|
|