mIRC Home    About    Download    Register    News    Help

Print Thread
Sorting a hash table output #206323 12/11/08 08:58 PM
Joined: Oct 2007
Posts: 214
B
Buggs2008 Offline OP
Fjord artisan
OP Offline
Fjord artisan
B
Joined: Oct 2007
Posts: 214
Hello,

I have this code working so far to output the contents of a fairly large hash table to a txt file.

My question is, how do you sort the ouput of it alphabetically?

Code:
  var %f = $+($scriptdir,Report_IPInfo.txt), %t = ipinfo

  write -c %f Report runtime: $asctime(ddd mmm dd HH:nn:ss) $crlf $crlf

  write %f No. of Items: $hget(%t, 0).item $crlf $crlf

  if ($hfind(%t,*,0,w) > 0) {
    var %i = 1
    while ($hget(%t, %i).item != $null) {
      tokenize 44 $hget(%t,$v1) 
      write %f Nickname: $1
      write %f Address: $2
      write %f IP: $3
      write %f E-mail $4 $crlf $crlf
      inc %i 
    }
  }



Thanks a bunch in advance!

Cheers,

Jay

Re: Sorting a hash table output [Re: Buggs2008] #206330 12/11/08 10:55 PM
Joined: Nov 2006
Posts: 1,559
H
Horstl Offline
Hoopy frood
Offline
Hoopy frood
H
Joined: Nov 2006
Posts: 1,559
You could use some kind of table/chart structure, if you do not put nick / address / IP etc on separate lines but separate the "cells" with a char that may not be part of nicks/addresses/IPs/emails, like $chr(9) = TAB or $chr(160) = hardspace. Now, if you create a "dataset" like:

nick1<SEP>address1<SEP>IP1<SEP>mail1
nick2<SEP>address2<SEP>IP2<SEP>mail2
nick3<SEP>address3<SEP>IP3<SEP>mail3
nick4<SEP>address4<SEP>IP4<SEP>mail4

...you can now apply the powerfull /filter command, to sort your table/dataset by any "collumn", ascending or descending, numerically or alphanumerically...

For example, if your <SEP> had been $chr(9), sorting the table acc. to third collumn would be: "/filter -ffct 3 9 file.txt file.txt"

Add "heading lines", runtime etc. after sorting the data, for example with /write -i.

Re: Sorting a hash table output [Re: Horstl] #206331 12/11/08 11:25 PM
Joined: Jan 2007
Posts: 1,156
D
DJ_Sol Offline
Hoopy frood
Offline
Hoopy frood
D
Joined: Jan 2007
Posts: 1,156
lol I was replying saying the same thing. Ya beat me! lol you did a much better job considering Ive never had to use filter. But yeah I saw that -c switch and thought, couldn't you save the hash file to a txt file with /hsave and then use the character mirc puts in between the data as your column seperator?

Re: Sorting a hash table output [Re: DJ_Sol] #206334 13/11/08 12:17 AM
Joined: Nov 2006
Posts: 1,559
H
Horstl Offline
Hoopy frood
Offline
Hoopy frood
H
Joined: Nov 2006
Posts: 1,559
Yepp, it all depends on how you initially store the data to the hash table.
/hsave -i could ease the job, one might remove the created "ini topic" [hashtable] then and perform a /filter to remove all non-matches for a specific search term, together with (or prior to) sorting the data via /filter.

Simple example for the hsave/remove topic/sort (without the "filter for matches" part):
Code:
alias sorttest {
  ; add some items
  hadd -m testtable nick1 address4=ip2=mail3
  hadd testtable nick3 address3=ip3=mail2
  hadd testtable nick2 address2=ip1=mail4
  hadd testtable nick4 address1=ip4=mail1

  if ($isfile(test.txt)) { .remove test.txt }
  ; save table in ini format
  hsave -i testtable test.txt
  ; remove ini topic
  write -dl1 test.txt
  ; sort by a collumn
  filter -ffct $1 61 test.txt test.txt
  ; add a "header" line
  write -il1 test.txt Data Sorted by collumn $1 :
  run test.txt
  hfree testtable
}

/sorttest <N of collumn>

Re: Sorting a hash table output [Re: Buggs2008] #206351 13/11/08 02:36 PM
Joined: Oct 2007
Posts: 214
B
Buggs2008 Offline OP
Fjord artisan
OP Offline
Fjord artisan
B
Joined: Oct 2007
Posts: 214
Wow!,

Very nice Hosrtl

I am very impressed on how quick and easy it organizes data, by the specified column.

I know the data is kind of scrunched together, is there anyway to throw in a space between each line of the sorted output? (for easier viewing)

Other than that, I very impressed with the code and examples you have provided to me.

Thanks again all,

Cheers

Jay

Re: Sorting a hash table output [Re: Buggs2008] #206397 14/11/08 06:24 AM
Joined: Nov 2006
Posts: 1,559
H
Horstl Offline
Hoopy frood
Offline
Hoopy frood
H
Joined: Nov 2006
Posts: 1,559
It's possible to /filter to an alias (instead of file-to file atm), and have this alias do some "text formatting".
But it's more easey (and might be sufficient) to loop all the lines of the sorted data (in this case: your file.txt), and e.g. write a text-formatted "ready-to-use" version to another file.
Something like (assuming $chr(61) as in the last example was your "separating char"):

Code:
[...filter to file.txt is done...]

  var %f = formatted.txt

  ; write heading lines
  write -c %f Total matches: $lines(file.txt)
  write %f $crlf
  write %f $crlf

  ; do something with the lines of file.txt and insert blank lines
  var %n = 1
  while ($read(file.txt,%n)) {
    tokenize 61 $v1
    write %f Nick: $1 Address: $2 IP: $3 eMail: $4
    write %f $crlf
    inc %n
  }

[...]

Note that you might use (hidden) @windows to "dump" and /filter data as well. I just wanted to demonstrate how /filter might be used to manipulate a dataset easily, if the data has been arranged on a line-per-line order.

Re: Sorting a hash table output [Re: Buggs2008] #206412 14/11/08 05:44 PM
Joined: Oct 2007
Posts: 214
B
Buggs2008 Offline OP
Fjord artisan
OP Offline
Fjord artisan
B
Joined: Oct 2007
Posts: 214
Hello Horstl,

Great Work!

I love it, except now im getting 2 datasets, one with it all bunched up and the other, which was sortted nicely with the spaces and all nice a neat.

I think I prefer your way of dumping it into this "hidden window" and then writting it back into a text file.

How would you go about achieving this, I have to admit that I am not the best when it comes to data manipulation.

Great suggestions and comments so far im getting there,

Heres what I have so far relating to your previous comments in other posts:

Dataset:
3ngsxp=3ngsxp,04f9cbc0610c273baeee6757d9631b05,41.208.65.133,awb_123@yahoo.com
Code:
alias 1234 {

  var %file = $+($scriptdir,Report_IPInfo.txt)

  if ($isfile(%file)) { .remove %file }

  hsave -i ipinfo %file

  write -dl1 %file

  filter -ffct $1 61 %file %file

  ;write -il1 %file Data Sorted by collumn $1 : $crlf $crlf

  ; write heading lines
  write %file Total matches: $lines(%file)
  write %file $crlf
  write %file $crlf

  ; do something with the lines of file.txt and insert blank lines
  var %n = 1
  while ($read(%file,%n)) {
    tokenize 44 $v1
    write %file Nick: $GETTOK($1,1,61) Address: $2 IP: $3 eMail: $4
    write %file $crlf
    inc %n
  }


  run %file
}




;Edit Corrected Alias based on Horstl's advice:

Code:
;1 = Compile by Nickname
;2 = Compile by Address
;3 = Compile by IP Address
;4 = Compile by Email

;usage ipinfo_compile 1

alias ipinfo_compile {

  var %x = $1 | %x = $replace(%x,1,Nickname,2,Address,3,IP Address,4,E-mail)


  if (!$window(@test)) { window -hw0 @test }
  else { clear @test }
  var %i = 1, %t = ipinfo, %file = $+($scriptdir,Report_IPInfo.txt)

  if ($isfile(%file)) { .remove %file }

  while $hget(%t, %i).item != $null { 
    aline @test $hget(%t, $v1) $crlf $crlf
    inc %i
  }

  filter -wfct $1 44 @test %file

  write %file ----------------------------------------------------------------------------------
  write %file Report runtime: $+($asctime(ddd mmm dd @ HH:nn:ss),.) There are $hget(%t,0).item item(s) listed
  write %file Report compiled by: %x

  run %file
}


Last edited by Buggs2008; 14/11/08 09:10 PM.
Re: Sorting a hash table output [Re: Buggs2008] #206430 15/11/08 04:53 AM
Joined: Nov 2006
Posts: 1,559
H
Horstl Offline
Hoopy frood
Offline
Hoopy frood
H
Joined: Nov 2006
Posts: 1,559
The "problem" (in terms of possible options) with your current structure is that you store the data acc. to:
hash table item: nick
hash table data: nick,address,IP,eMail
therefore, a /hsave -i produces: nick=nick,address,IP,eMail
As you see, the data isn't separated by the same char ("=") which would be "perfect" for /filter -t. On the other hand side it's OK in this case, and "=" may be part of email addresses... Sorting the $chr(44)-separated column 1 works none the less.

Because you want to "dump" the whole hash table here, hsave -i should work better (faster) than a while loop through all hash table items like:
Code:
while $hget(%t, %i).item != $null { 
  aline @test $hget(%t, $v1) $crlf $crlf
  inc %i
}

...just get rid of the dump-file after creating the "final" file. smile

A hidden window may be the better choice IF you're scanning for specific items in the hash table ($hfind > aline). And IF there actually would be multiple filter/search operations to do, a dumped file might catch up again: as you could perform multiple /filter commands on the same source file. But again, this isn't the case here.

The code below is more or less an alternative version of your code, maybe you like one or two details of it.

Code:
; usage: /listinfo [nick|address|IP|mail]

alias listinfo {
  if ($findtok(nick address IP mail,$1,32)) { var %column = $v1 }
  else { echo -a Syntax: /info [nick|address|IP|mail] | return }

  if (!$hget(ipinfo,1).item) { echo -a table "ipinfo" is empty. }
  else {
    ; $mircdir, to ensure write permission on all systems. $qt to play safe with spaces in folder names.
    var %tempfile = $qt($+($mircdir,IPinfoDump.txt)), %outfile = $qt($+($mircdir,Report_IPInfo.txt)), %n = 1

    ; dump table to tempfile (write -c to ensure an empty tempfile)
    write -c %tempfile
    hsave -i ipinfo %tempfile
    ; sort data in tempfile
    write -dl1 %tempfile
    filter -ffct %column 44 %tempfile %tempfile

    ; heading lines of outfile
    write -c %outfile Report runtime: $+($asctime(ddd mmm dd @ HH:nn:ss),.) There are $lines(%tempfile) item(s) listed.
    write %outfile Report compiled by: $replace($1,ip,IP address,mail,E-mail) 
    write %outfile $str(-,30) $crlf $crlf

    ; for the "show the sort-by column in first place" idea
    var %denote = Nick: Address: IP: eMail:

    ; loop lines of tempfile
    while ($read(%tempfile,%n)) {
      var %read = $gettok($v1,2-,61), %line, %t = 1
      ; insert denotations
      while ($gettok(%read,%t,44)) {
        var %line = $addtok(%line,$gettok(%denote,%t,32) $v1,44)
        inc %t
      }
      ; write data to outfile; sortby-column in first place
      tokenize 44 $deltok(%line,%column,44)
      write %outfile $gettok(%line,%column,44) >>> $1- $crlf $crlf
      inc %n
    }

    ; remove the tempfile and run the outfile
    .remove %tempfile
    run %outfile

  }
}


Note that the while loop (performing multiple /write operations) can be accelerated using "file handlers", but that's a different story and only importand if your dataset is huge.

Last edited by Horstl; 15/11/08 05:08 AM.
Re: Sorting a hash table output [Re: Buggs2008] #206432 15/11/08 05:36 AM
Joined: Oct 2007
Posts: 214
B
Buggs2008 Offline OP
Fjord artisan
OP Offline
Fjord artisan
B
Joined: Oct 2007
Posts: 214
Wow!,

I really don't know where to start thanking you Horstl for writting and comming up such a complex, fast and unique code for me.

I guess I'll eventually get there myself, but your code is far more superior and effective than mine.

Need I say that was exactly what I wanted to achieve as the final result.

Thank you so much for helping me figure this out.

The dataset will get larger in due time, id also be interesting to find out how to implement these "data accelerators" ive never even heard of that term to begin with.

Again, thank you for all your time, help and support.

The cheque is definetly in the mail!

Cheers,

Jay.

Re: Sorting a hash table output [Re: Buggs2008] #206434 15/11/08 06:29 AM
Joined: Nov 2006
Posts: 1,559
H
Horstl Offline
Hoopy frood
Offline
Hoopy frood
H
Joined: Nov 2006
Posts: 1,559
...nothing wrapped in mystery smile
In short, file handling is very usefull if you're writing a lot of lines at once. To circumscribe the difference of /write and /fwrite:
Each /write command "accesses" the file in question, performs the file manipulation and "releases" the file right after (for example, the file can be accessed by other apps again).
Using the file handling method, your mIRC is getting "access" to the file via /fopen, then you perform as many /fwrite operations as you like, and finally you "release" the file via /fclose (Maybe someone else can depict the difference more precise...).

To demonstrate the different performance:
Code:
; /test.write <N>

; = while loop performing N /write
alias test.write {
  var %n = 1, %start = $ticks
  while (%n <= $1) {
    write test.txt <This is a test text>
    inc %n
  }

  ECHO -ag wrote $lines(test.txt) lines via write, this took $calc($ticks - %start) ticks.
  .remove test.txt
}

; /test.fwrite <N>

; = while loop performing N /fwrite
alias test.fwrite {
  var %n = 1, %start = $ticks
  .fopen -o testfile test.txt 
  while (%n <= $1) {
    .fwrite -n testfile <This is a test text>
    inc %n 
  }
  .fclose testfile 

  ECHO -ag wrote $lines(test.txt) lines via fwrite, this took $calc($ticks - %start) ticks.
  .remove test.txt
}

If you put a high number, e.g. 5000, it's as different as day and night (on my system it's 20x faster). smile

Re: Sorting a hash table output [Re: Buggs2008] #206435 15/11/08 07:14 AM
Joined: Oct 2007
Posts: 214
B
Buggs2008 Offline OP
Fjord artisan
OP Offline
Fjord artisan
B
Joined: Oct 2007
Posts: 214
Hello Horstl,

Wow, your work ceases to amaze my ever growing mIRC fascinations to their fullest extent possible.

The performance of the Fopen / Fwrite stuff is unbelievable 220ms to 15ms for 100 lines.

How in the world, would you relate this function back into your previous example that you had provided to me? This would make it unbeleivably fast and probably not to mention use less resources to compile it.

Code:
; usage: /listinfo [nick|address|IP|mail]

alias listinfo {
  if ($findtok(nick address IP mail,$1,32)) { var %column = $v1 }
  else { echo -a Syntax: /info [nick|address|IP|mail] | return }

  if (!$hget(ipinfo,1).item) { echo -a table "ipinfo" is empty. }
  else {
    ; $mircdir, to ensure write permission on all systems. $qt to play safe with spaces in folder names.
    var %tempfile = $qt($+($mircdir,IPinfoDump.txt)), %outfile = $qt($+($mircdir,Report_IPInfo.txt)), %n = 1

    ; dump table to tempfile (write -c to ensure an empty tempfile)
    write -c %tempfile
    hsave -i ipinfo %tempfile
    ; sort data in tempfile
    write -dl1 %tempfile
    filter -ffct %column 44 %tempfile %tempfile

    ; heading lines of outfile
    write -c %outfile Report runtime: $+($asctime(ddd mmm dd @ HH:nn:ss),.) There are $lines(%tempfile) item(s) listed.
    write %outfile Report compiled by: $replace($1,ip,IP address,mail,E-mail) 
    write %outfile $str(-,30) $crlf $crlf

    ; for the "show the sort-by column in first place" idea
    var %denote = Nick: Address: IP: eMail:

    ; loop lines of tempfile
    while ($read(%tempfile,%n)) {
      var %read = $gettok($v1,2-,61), %line, %t = 1
      ; insert denotations
      while ($gettok(%read,%t,44)) {
        var %line = $addtok(%line,$gettok(%denote,%t,32) $v1,44)
        inc %t
      }
      ; write data to outfile; sortby-column in first place
      tokenize 44 $deltok(%line,%column,44)
      write %outfile $gettok(%line,%column,44) >>> $1- $crlf $crlf
      inc %n
    }

    ; remove the tempfile and run the outfile
    .remove %tempfile
    run %outfile

  }
}


Thanks for showing me this....im sure ill be using this now once I figure this out completely.

Thanks again for everything!

Cheers,

Jay.

Last edited by Buggs2008; 15/11/08 07:20 AM.
Re: Sorting a hash table output [Re: Buggs2008] #206437 15/11/08 08:14 AM
Joined: Nov 2006
Posts: 1,559
H
Horstl Offline
Hoopy frood
Offline
Hoopy frood
H
Joined: Nov 2006
Posts: 1,559
with file handling; both for the tempfile (multiple $read >>> $fread) and outfile (multiple write >>> fwrite)
Code:
; /listinfo [nick|address|IP|mail]

alias listinfo {
  if ($findtok(nick address IP mail,$1,32)) { var %column = $v1 }
  else { echo -a Syntax: /info [nick|address|IP|mail] | return }

  if (!$hget(ipinfo,1).item) { echo -a table "ipinfo" is empty. }
  else {
    var %tempfile = $qt($+($mircdir,IPinfoDump.txt))
    var %outfile = $qt($+($mircdir,Report_IPInfo.txt))
    var %denote = Nick: Address: IP: eMail:

    ; dump table to tempfile
    write -c %tempfile
    hsave -i ipinfo %tempfile

    ; sort data in tempfile
    write -dl1 %tempfile
    filter -ffct %column 44 %tempfile %tempfile

    ; fopen outfile
    if ($fopen(listinfo.out)) { .fclose listinfo.out }
    .fopen -o listinfo.out %outfile

    ; fwrite heading lines to outfile
    .fwrite -n listinfo.out Report runtime: $+($asctime(ddd mmm dd @ HH:nn:ss),.) There are $lines(%tempfile) item(s) listed.
    .fwrite -n listinfo.out Report compiled by: $replace($1,ip,IP address,mail,E-mail) 
    .fwrite -n listinfo.out $str(-,30) $crlf $crlf

    ; fopen tempfile
    if ($fopen(listinfo.temp)) { .fclose listinfo.temp }
    .fopen listinfo.temp %tempfile

    ; loop lines of tempfile
    while ($fread(listinfo.temp)) {
      var %read = $gettok($v1,2-,61), %line, %t = 1
      ; insert denotations
      while ($gettok(%read,%t,44)) {
        var %line = $addtok(%line,$gettok(%denote,%t,32) $v1,44)
        inc %t
      }
      ; fwrite lines to outfile; sortby-column in first place
      tokenize 44 $deltok(%line,%column,44)
      .fwrite -n listinfo.out $gettok(%line,%column,44) >>> $1- $crlf
    }

    ; fclose tempfile and outfile
    .fclose listinfo.*

    .remove %tempfile
    run %outfile
  }
}



with ~600 test items, it took ~650 ticks (instead of previously ~1650 ticks) on my system, of which ~500 are consumed by hsave -i

Re: Sorting a hash table output [Re: Buggs2008] #206454 15/11/08 07:59 PM
Joined: Oct 2007
Posts: 214
B
Buggs2008 Offline OP
Fjord artisan
OP Offline
Fjord artisan
B
Joined: Oct 2007
Posts: 214
Hello Horstl,

Your new code works great and the speed of which that it generates the new file is really something quite impressive.

A huge thanks for everything again and all you help.

Cheers,

Jay