mIRC Home    About    Download    Register    News    Help

Print Thread
#149869 24/05/06 11:55 PM
Joined: Oct 2005
Posts: 16
S
saxorr Offline OP
Pikka bird
OP Offline
Pikka bird
S
Joined: Oct 2005
Posts: 16
Hey, I was just wondering if a switch can be added to the $read command so it can start reading from the bottom, all the way to the top.

I know certain tricks can be made using the /filter command, but it takes a long time if the file is big (eg 50mb's)

Thanks in advance

#149870 25/05/06 02:36 AM
Joined: Sep 2003
Posts: 4,230
D
Hoopy frood
Offline
Hoopy frood
D
Joined: Sep 2003
Posts: 4,230
Be a nice idea, as you said, logs can get big, and its normally the latest occurance of something you want.
I would like to see it in another identifier rather than a switch, maybe like $endread only becuase i would like to see the last line of the file be treated like its line1, 2nd to last as line2 etc, i think this would cut down of disk access as well, since the file wouldnt need to be read forward to the line number you want and then a search take place reading back again.

example
-- filename.txt --
fred
carpert
dog
marketed
-- filename.txt --

$endread(filename.txt,1) => marketed : $endreadn = 1
$endread(filename.txt,w,*r*e*d*,2) => carpert : $endreadn = 3


Since i would assume the normal $read(,w,N) reads the file down to line N (lines being defined by $lf or $crlf), then searchs from there on, I wouldnt see it overly hard to read the file backwards from the end toward the begining, breaking lines when encountering $lf or $lf then a $cr (working backward).
(Obviuosly i think either way actually loads blocks of the file into ram and processes it from there, rather than actually making byte by byte disk requests.)

I guess its something that could be created using custom aliases and bread's but built in would of course be faster.

#149871 25/05/06 11:59 AM
Joined: May 2006
Posts: 122
J
Vogon poet
Offline
Vogon poet
J
Joined: May 2006
Posts: 122
$read is usually used in loops, why not just run the loops backwards??
instead of:
Code:
var %x = 1
while ($read(file,%x)) {
...
inc %x 1
}

use:
Code:
var %x = $lines(file)
while ($read(file,%x)) {
...
dec %x 1
}

I see what u mean Dave but I fail to see a use for it.
Depends what you are using it for I guess.

Last edited by jizzy; 25/05/06 12:01 PM.

If only women came with popup menus and online help.
#149872 25/05/06 12:02 PM
Joined: Feb 2004
Posts: 2,019
Hoopy frood
Offline
Hoopy frood
Joined: Feb 2004
Posts: 2,019
When you use $read with the "w, r or s" switches to scan the file, scanning starts from the beginning of the file, not the end. I think he wants to scan the other way round. If that's not what he meant, then this feature suggestion doesn't make much sense imo, as you can just loop backwards as you point out.

Btw in your while condition, you should put while ($read(file,%x) != $null) because that loop would stop once the $read evalutes to a zero, an empty line, or $false. But anyway, I wouldn't loop through a text file, that's a bad idea (continuous disk access) but rather loadbuf it into a hidden window, atleast then it's buffered in memory.


Gone.
#149873 25/05/06 12:06 PM
Joined: May 2006
Posts: 122
J
Vogon poet
Offline
Vogon poet
J
Joined: May 2006
Posts: 122
Im sorry but I still don't see a use for it.


If only women came with popup menus and online help.
#149874 25/05/06 12:08 PM
Joined: Feb 2004
Posts: 2,019
Hoopy frood
Offline
Hoopy frood
Joined: Feb 2004
Posts: 2,019
Because if you know that the latest occurance of somethign will be more likely to be at the end than at the top, it is much more beneficial to start scanning from the back. When you scan from the front to the end on a huge text file such as the 50mb one that the OP said, that takes much longer, because internally each line of the file is tested for a match. Imo though, this feature suggestion is a little too specific for this one feature of reading the logs, I can't see it overly useful in general.


Gone.
#149875 25/05/06 12:15 PM
Joined: May 2006
Posts: 122
J
Vogon poet
Offline
Vogon poet
J
Joined: May 2006
Posts: 122
I agree I dont see any 50mb files lying on my hdd that arn't programs files or movies ect. and I certianly have no need to use $read on them.

Last edited by jizzy; 25/05/06 12:17 PM.

If only women came with popup menus and online help.
#149876 25/05/06 05:04 PM
Joined: May 2006
Posts: 19
N
Pikka bird
Offline
Pikka bird
N
Joined: May 2006
Posts: 19
/last {
if ($1 == $null) {
%counter = 10 }
else {
%counter = $1 }
%lines = $lines(versions.txt)
while (%counter != 0) {
set %displine $read(versions.txt,n,$calc(%lines - %counter))
if (%displine != $null) echo %displine
dec %counter
}
}

smile Of course, you can alter that to supply your own file and check it exists.

Regards
Nigel / Nelgin

#149877 25/05/06 08:24 PM
Joined: Sep 2003
Posts: 4,230
D
Hoopy frood
Offline
Hoopy frood
D
Joined: Sep 2003
Posts: 4,230
You seem to have missed the point of the idea, if you can read from the end of the file back you dont need the HUGE level of disk accesses that would be required to read from the begining of the file.

An example
Assume the file has 50Mb of data average line length 50 characters (including $crlf) so 1,000,000 lines

you want to read the last 10, so
var %m = $lines(filename), %i = %m - 9 | while (%i <= %m) { echo -a . $read(filename,nt,%i) | inc %i }
ok $lines takes 1,000,000 lines read to get the number of lines, %m is 1000000 and %i is 999991
$read(filename,999991) reads 999991 lines, 999992 reads 999992 lines etc to 1000000 reads 1000000 lines
Total lines read 10,999,955 all to get 10 lines out.

now amagine $endread
var %i = 10 | while (%i) { echo -a . $endread(filename,nt,%i) | dec %i }
$endread(filename,10) read the last 10 lines, 9 last 9 , 8 last 8 etc to 1 last 1
Total lines read 55

10,999,955 vs 55 (well i know which one i would use)


* i dont make any assumptons on how the file is internally read, i know its not read line by line from the disk, but rather in blocks, but the same amount of physical disk accesses would be required as the file is read through to reach the lines requested*

#149878 25/05/06 09:35 PM
Joined: Oct 2005
Posts: 16
S
saxorr Offline OP
Pikka bird
OP Offline
Pikka bird
S
Joined: Oct 2005
Posts: 16
Quote:
scanning starts from the beginning of the file, not the end. I think he wants to scan the other way round.

That's exactly what I'm talking about. As far as loop scripts go, I'd like to see someone loop a read script on a 50mb file or bigger, the amount of time it takes to search through it is just too long (takes about a minute, if not more, depending on the PC).

Also I'm a little surprised by the amount of people who suggest this feature is useless since they don't need it - we all obviously have diffrent needs, I don't use the /readini command, does it mean I have to rally against it?

Thanks for the alternative suggestions on the thread so far, I'll have a look at how they work and will most likley use them for now, but a command like $endread would be very handy.

#149879 25/05/06 11:29 PM
Joined: Sep 2003
Posts: 4,230
D
Hoopy frood
Offline
Hoopy frood
D
Joined: Sep 2003
Posts: 4,230
im hiding out the back office at work, hoping no one wants me for anything, so this means im also a little bored smile thus im gonna try and script a $endread for ya, wont be as fast as an internal command but might help you out (if i manage it)

As a mater of interest what was using /filter like for performance on a 50meg file? (still to slow?)

#149880 26/05/06 12:52 AM
Joined: Oct 2005
Posts: 16
S
saxorr Offline OP
Pikka bird
OP Offline
Pikka bird
S
Joined: Oct 2005
Posts: 16
Quote:
As a mater of interest what was using /filter like for performance on a 50meg file? (still to slow?)

Thanks alot for helping smile

/filter .txt -> @window is kinda slow on a big file, but it's the only choice available at the moment, although anything else from then takes no time at all...

It'd just be convinent if I had $endread for example where I would skip quite a few steps and cpu usage from the top, and just make it //say $endread(log.txt,w,*hello*) or wherever its needed.

#149881 26/05/06 07:25 PM
Joined: Jan 2003
Posts: 2,523
Q
Hoopy frood
Offline
Hoopy frood
Q
Joined: Jan 2003
Posts: 2,523
Signed, this would be a nice feature, one that I wanted for a long time. In the meantime, if you're interested in a scripted workaround, here it is:
Code:
alias endread {
  if (!$isid) return
  if $0 &gt; 2 {
    if w isin $2 || r isin $2 { var %m = $v1, %l = 1, %s = $iif($4 isnum 2-,$4,1) } 
    else return
  }
  elseif $2 isnum 1- { var %m = l, %l = $2 }
  else return
  .fopen e $qt($1)
  if ($ferr) return

  var %p = $file($1) - 920, %i = 0
  while %p &gt; -920 {
    if %p &gt; 0 {
      .fseek e %p
      .fseek -n e
    }
    else .fseek e 0
    var %d = $calc(920 - $fopen(e).pos + %p), %p = $fopen(e).pos
    noop $fread(e,%d,&amp;a)
    var %t = $bvar(&amp;a,1-).text, %n = $regex(%t,/\r?\n/g) + 1
    if !%i &amp;&amp; * $+ $crlf iswm %t { 
      %t = $left(%t,-2)
      dec %n
    }
    if %m == l {
      if %l &lt;= %n { 
        .fclose e $regex(e,%t,/(?&lt;=^|\n)(.*?)(?=$|\r?\n)/g)
        return $regml(e,$calc(%n - %l + 1))
      }
    }
    elseif %s &lt;= $calc(%i + %n) {
      var %t = $replace(%t,$crlf,$lf), $&amp;
        %j = $regex(e,%t,/(?&lt;=^|\n)(.*?)(?=$|\r?\n)/g)
      if (%m == w) var %c = $!wildtok($regml(e,%j),$3,0,0)
      else var %c = $!regex($regml(e,%j),$3)
      while %j { 
        if $calc(%i + %n - %j + 1) &gt;= %s &amp;&amp; $(%c,2) {
          .fclose e
          return $regml(e,%j)
        }
        dec %j
      }
    }
    dec %p 920
    inc %i $calc(%n - 1)
  }
  .fclose e
}


Its usage is exactly like $read(), except that not all switches are supported. More specifically,
w, r are supported
s, p aren't supported at all
n, t are implied

Valid examples are:
$endread(file.txt,1) = last line in the file
$endread(file.txt,w,*blah*) = first line from the end that matches *blah*
$endread(versions.txt,r,/mode(first|last)/,10) = line that matches "modefirst" or "modelast", but the file is searched from the 10th line from the end and upwards.

Oh and don't try to use it with files that contain lines longer than 900 characters, it will break most probably.

Note that even though I've tested this quite a lot, it may still have bugs. If you decide to use it, please do report back any bugs you find.

EDIT: Fixed a typo in the 5th line from the end

EDIT 2: Removed that line, as it's completely unnecessary... probably a leftover from early tries

EDIT 3: Yet another mistake corrected... hopefully the last

EDIT 4: Wishful thinking frown The alias is problematic, I'm looking into it but I may miss the 2-hour-edit-time window and have to repost

Last edited by qwerty; 26/05/06 08:26 PM.

/.timerQ 1 0 echo /.timerQ 1 0 $timer(Q).com
#149882 27/05/06 01:03 AM
Joined: Jan 2003
Posts: 2,523
Q
Hoopy frood
Offline
Hoopy frood
Q
Joined: Jan 2003
Posts: 2,523
Here's a hopefully correct version:
Code:
alias endread {
  if (!$isid) return
  if $0 &gt; 2 {
    if w isin $2 || r isin $2 { var %m = $v1, %l = 1, %s = $iif($4 isnum 2-,$4,1) } 
    else return
  }
  elseif $2 isnum 1- { var %m = l, %l = $2 }
  else return
  .fopen e $qt($1)
  if ($ferr) return

  var %p = $file($1) - 920, %i = 0
  while %p &gt; -920 {
    if %p &gt; 0 {
      .fseek e %p
      .fseek -n e
    }
    else .fseek e 0
    var %d = $calc(919 - $fopen(e).pos + %p), %p = $fopen(e).pos - 920
    if !%i { 
      var %n = $fread(e,$calc(%d + 1),&amp;a)
      if $bvar(&amp;a,%n) == 10 { bset &amp;a %n 13 }
    }
    else noop $fread(e,%d,&amp;a)
    var %t = $bvar(&amp;a,1-).text, %n = $regex(%t,/\r?\n/g) + 1
    inc %i %n
    if %m == l {
      if %l &lt;= %i { 
        .fclose e $regex(e,%t,/(?&lt;=^|\n)(.*?)(?=$|\r?\n)/g)
        return $remove($regml(e,$calc(%i - %l + 1)),$cr)
      }
    }
    elseif %s &lt;= %i {
      var %j = $regex(e,%t,/(?&lt;=^|\n)(.*?)(?=$|\r?\n)/g)
      if (%m == w) var %c = $!wildtok($regml(e,%j),$3,0,0)
      else var %c = $!regex($regml(e,%j),$3)
      while %j { 
        if $calc(%i - %j + 1) &gt;= %s &amp;&amp; $(%c,2) {
          .fclose e
          return $remove($regml(e,%j),$cr)
        }
        dec %j
      }
    }
  }
  .fclose e
}


/.timerQ 1 0 echo /.timerQ 1 0 $timer(Q).com

Link Copied to Clipboard