mIRC Home    About    Download    Register    News    Help

Print Thread
Faster file writing or faster sorting? #159181 14/09/06 09:04 PM
Joined: Aug 2006
Posts: 22
N
netsplit Offline OP
Ameglian cow
OP Offline
Ameglian cow
N
Joined: Aug 2006
Posts: 22
Okay here's the setup. I have a list of data that needs sorted alphabetically, channel nicks actually, as well as some other data. So I write it to a file and use /filter thing is the /write is really slow. Normally the data is small so it isn't noticeable, however in large channels it causes mirc to pause when ran...

lag! D:

I don't want to use an @window. Any dlls for faster write? Faster ways to code data writing? Better ways to sort a hash table?

Re: Faster file writing or faster sorting? #159182 14/09/06 09:22 PM
Joined: Aug 2004
Posts: 7,252
R
RusselB Offline
Hoopy frood
Offline
Hoopy frood
R
Joined: Aug 2004
Posts: 7,252
The main thing limiting your write speed for a file is your hard drive speed. The only way of increasing that is to get a faster hard drive. I've got a feeling that this is not a reasonable option at this point.
Hash tables are stored in an unsorted format, and are (realistically) unsortable. Regarding DLL's, there might be, but I'm unaware of any.
Realistically (if not literally) using an @window would be your fastest option, so I have to wonder why you don't want to use an @window?
If coded properly, you could use the @window to do the sorting, and (probably) not even notice the window (ie: using a hidden window to do the sorting, and then close it automatically when complete)

Re: Faster file writing or faster sorting? #159183 14/09/06 09:42 PM
Joined: Oct 2004
Posts: 8,327
Riamus2 Offline
Hoopy frood
Offline
Hoopy frood
Joined: Oct 2004
Posts: 8,327
Well, hash tables can easily be sorted, but it requires writing it to a file first. The information on how is located in the forum if anyone feels like trying to find it. It was a long time ago and I *think* it may have been written originally by DaveC, though after so long, I could be wrong.

Anyhow, RusselB is right that windows are the best option if you really have that much trouble writing the information and they can be hidden like he said. I'm wondering, though, why it's giving you so much trouble to write it to a file. It could be that your script needs to be improved. Writing even a few hundred lines shouldn't really lag you any.


Invision Support
#Invision on irc.irchighway.net
Re: Faster file writing or faster sorting? #159184 14/09/06 09:58 PM
Joined: Aug 2004
Posts: 7,252
R
RusselB Offline
Hoopy frood
Offline
Hoopy frood
R
Joined: Aug 2004
Posts: 7,252
Yes, I didn't have any test information, but I just did a quick script to see how long my system is lagged for how many lines written, and the results were a 2 millisecond delay (using ticks to calculate) when writing 10,000 lines. Now I know I have a fast system, but even if you multiplied that by a factor of 100, that would still mean a 2 millisecond delay for 100 lines.

Re: Faster file writing or faster sorting? #159185 14/09/06 10:31 PM
Joined: Aug 2006
Posts: 22
N
netsplit Offline OP
Ameglian cow
OP Offline
Ameglian cow
N
Joined: Aug 2006
Posts: 22
Cool, so it's the multiple write operations overhead. Not the actual writing it's self.

I'll just put it all in a big variable, how do I put the line break in between the variables in a single write operation?

Re: Faster file writing or faster sorting? #159186 14/09/06 11:05 PM
Joined: Aug 2004
Posts: 7,252
R
RusselB Offline
Hoopy frood
Offline
Hoopy frood
R
Joined: Aug 2004
Posts: 7,252
you can do that by appending $crlf to the end of each line, but note that variables have a maximum length of about 950 characters (which includes the variable name, the = sign used to assign the variable and all of the characters in the variable). To append the $crlf you have to put like <line1> $+ $crlf $+ <line2> $+ $crlf

So if you have 4 lines at 200 characters each, and you call the variable %a, then that's going to give you a variable that's about 839 characters in length (still under the 950, but hopefully it gives you an idea as to how quickly that 950 character mark can be reached)

I still have to recommend that you consider using @windows for your sorting, especially as you've not said why you don't want to use them.

Re: Faster file writing or faster sorting? #159187 14/09/06 11:37 PM
Joined: Aug 2006
Posts: 22
N
netsplit Offline OP
Ameglian cow
OP Offline
Ameglian cow
N
Joined: Aug 2006
Posts: 22
Thank you both much appreciated. smile

The main reason is I'd have to recode the data reloader. Also I'm kinda worried about the window stealing the focus. It's not an issue except in very large channels (2000+). That and I'm still learning @windows.

Re: Faster file writing or faster sorting? #159188 14/09/06 11:55 PM
Joined: Oct 2004
Posts: 8,327
Riamus2 Offline
Hoopy frood
Offline
Hoopy frood
Joined: Oct 2004
Posts: 8,327
Hide the window and it can't steal focus and you won't even know it is even there.


Invision Support
#Invision on irc.irchighway.net
Re: Faster file writing or faster sorting? #159189 15/09/06 03:31 AM
Joined: Feb 2005
Posts: 342
R
Rand Offline
Fjord artisan
Offline
Fjord artisan
R
Joined: Feb 2005
Posts: 342
Code:
; Create and hide the window
window -hsle @info
; Use a loop below if necessary, to put data into window.
aline @info WHATEVER

; Here you have two choices,
; Make the window active and put it back into switchbar/treeview.
; Or instead of making it active you could just close it.

; Uncommenting the line below will make the window active.
; window -aw3 @info
; Uncommenting the line below will close the window.
; window -c @info


This makes the @info window a "sorted list box" and since it's "hidden" to start off with, it allows your text to be written to the window a bit faster, AND prevents it from stealing focus. Especially if you choose to close the window after you're finished instead of making it the active window.

If you still prefer to actually *write* to a file, you can use:

fopen myfile the_file_here.txt
fwrite -n myfile %data
fwrite -n myfile %data2
fwrite -n myfile %data3
fclose myfile

The above works much faster then the standard /write.

Re: Faster file writing or faster sorting? #159190 15/09/06 03:51 AM
Joined: Oct 2005
Posts: 1,741
G
genius_at_work Offline
Hoopy frood
Offline
Hoopy frood
G
Joined: Oct 2005
Posts: 1,741
This code can be used to filter a #channel to a file.txt.

Code:
alias listfilter {
  ;$1 = channel, $2 = filename

  if (!$2) { echo -a /listfilter &lt;channel&gt; &lt;filename&gt; | return }

  if (%lf.busy) { echo -a Already filtering %lf.target | return }
  set %lf.busy 1
  var %lf.start = $ctime
  set %lf.target $1

  if ($window($+(@sort.,$1))) window -c $+(@sort.,$1)
  window -hl $+(@sort.,$1)

  if ($exists($2)) .remove $2
  filter -wlk $1 sfadd *
  filter -wftl 1 32 $+(@sort.,$1) $2 *
  if ($exists($2)) run $$2

  echo -s Channel: $1 Lines: $filtered Time: $calc($ctime - %lf.start) $+ s

  :error
  unset %lf.*
  if ($window($+(@sort.,$1))) window -c $+(@sort.,$1)
  reseterror
}

alias -l sfadd aline $+(@sort.,%lf.target) $regsubex(,$1,/^[ $+ $prefix $+ ]/g,)


Syntax: /listfilter #channel file.txt

The /listfilter alias can only be called once at a time. A variable ensures that the alias is not called again before the first occurance has completed.

I tested it on a channel of 1000+ users, and it took less than 1 seconds to complete.

-genius_at_work

Re: Faster file writing or faster sorting? #159191 15/09/06 03:55 AM
Joined: Sep 2003
Posts: 261
S
Scorpwanna Offline
Fjord artisan
Offline
Fjord artisan
S
Joined: Sep 2003
Posts: 261
What about dumping all the lines of a text file in $sorttok dividing them by some oddball character that you know noone uses. then while loop it back into writing it sorted line by line into the file? Since your just looking to do this with nicks,
Code:
alias sortfilebyline {
  var %thefile = $+(", $scriptdirnicklist.txt, ")
  var %newfile = $+(", $scriptdirnicklist2.txt, ")
  var %lines = $lines(%thefile)
  var %i = 1
  var %data
  while (%i &lt;= %lines) {
    %data = %data $+ $read(%thefile, %i) $+ :
    inc %i
  }
  %data = $sorttok(%data,58)
  %i = 1
  %lines = $wildtok(%data,*,0,58)
  while (%i &lt;= %lines) {
    write %newfile $gettok(%data,%i,58)
    inc %i 
  }
  .remove %thefile
  .rename %newfile %thefile
}
 

Re: Faster file writing or faster sorting? #159192 15/09/06 04:42 AM
Joined: Feb 2005
Posts: 342
R
Rand Offline
Fjord artisan
Offline
Fjord artisan
R
Joined: Feb 2005
Posts: 342
Seriously, read the original post.

He wanted to avoid writing to a file in the first place, because "WRITE" takes too long. The "example" you gave is just a bit too process intensive.

This is why my above post suggests the use of using a sorted list window /filter the nicklist into the sorted @window and then do whatever he needs to based on the @window. As far as writing to files goes, it's either /filter OR /fopen, /fwrite, and /fclose.

Re: Faster file writing or faster sorting? #159193 15/09/06 05:18 AM
Joined: Sep 2003
Posts: 261
S
Scorpwanna Offline
Fjord artisan
Offline
Fjord artisan
S
Joined: Sep 2003
Posts: 261
In his original post he doesn't mention not writing to a file. Yes he says it's slow, but his post wasn't just about writing to a file, but also sorting too. My post was just to display some $sorttok info not actually telling him to write to a file. Get the info from something else and send it to $sorttok for %varing then do something with it. I have noway of knowing the posters scripting experience that's why I just made something simple.

However, yeah you're right I have no business replying to someone else's post even if it is to offer some type of help. Touch you and your godlyness. Bow down before the mighty scripter. Me and my crappy "example" might as well stay away from here if we are just going to be put down.

Re: Faster file writing or faster sorting? #159194 15/09/06 05:20 AM
Joined: Sep 2003
Posts: 4,230
D
DaveC Offline
Hoopy frood
Offline
Hoopy frood
D
Joined: Sep 2003
Posts: 4,230
Getting right back to the first thing, I would assume your doing something drasticly wrong with the /WRITE operation, are u just adding lines to the file, or are you inserting them?
becuase as Russel said, just writing 10,000 lines takes stuff all time. And if its not the write I wonder if its the loop your using to fetch each line for writing, maybe its flawed.
I have seen several reasons a script that must loop well start falling on its butt, one of the biggest and simplest mistakes someone well make looks like this

var %i = 1 | while (%i <= $fline(#channel,*matchtext*,0)) { .......code using $fline(#channel,*matchtext*,%i)....... | inc %i }
^ that goes and searches for every matchtext getting the total every single time it loops!

var %i = 1, %m = $fline(#channel,*matchtext*,0) | while (%i <= %m) { .......code using $fline(#channel,*matchtext*,%i)....... | inc %i }
^ thats much better since it need only check the total once

window -h @window | filter -wwc #channel @window *matchtext* | var %i = 1, %m = $line(@window,0) | while (%i <= %m) { .......code using $line(@window,%i)....... | inc %i } | window -c @window
^ that should be faster again (but uses a @window, i however assume you didnt want to use one from lack of knowledge on them as you said)

alias filter.alias.name { .......code using $1....... } | ; note $1 contains the WHOLE LINE! see tokenize to seperate!
&
filter -wk #channel filter.alias.name *matchtext*
^ that is about as fast as i can think of looping with code that you have to execute

filter -wf #channel filename.txt *matchtext*
^ and thats about as fast as fast as you can get selected lines out to a file

---

Without sample code to show us what your doing i doubt you well find a solution. thats well make your problem go away.
The best i can suggest if its JUST write speed is that you switch to using /FWRITE whcih requeres you to open, write and close the file your writting to.

Re: Faster file writing or faster sorting? #159195 15/09/06 10:21 PM
Joined: Oct 2004
Posts: 73
M
Mardeg Offline
Babel fish
Offline
Babel fish
M
Joined: Oct 2004
Posts: 73
Instead of the second /filter to a file, could he use /savebuf of the custom listwindow if it is opened with the -s auto-sort parameter?
Code:
 window -hls $+(@sort.,$1) 

I'm not sure if it's quicker or if /savebuf applies to a listwindow, and if the set window buffersize in options -> other is too small this method may fail.

Re: Faster file writing or faster sorting? #159196 15/09/06 11:40 PM
Joined: Sep 2003
Posts: 4,230
D
DaveC Offline
Hoopy frood
Offline
Hoopy frood
D
Joined: Sep 2003
Posts: 4,230
I see what your meaning however...
Using a list window has overheads likely to do with the selectability functions of a list window, even more so if your sorting the list.
It would have been better to use a plain window in your example, also using -hlS does not work as the -l alone makes the main window area the listbox whcih means u would have needed the -s option to sort it, i however assumed you had ment this rather than the -S.

Heres a small alias and the results i got using it on different window types.
Code:
alias test {
  window -c @test | window $1 @test
  var %i = 1100000, %ticks = $ticks
  while (%i &gt; 1000000) { aline @test line %i | dec %i }
  echo -s Using " /window $1 " took $calc($ticks - %ticks) ticks.
  window -c @test
}

Using " /window -h " took 5078 ticks. result unsorted
Using " /window -hs " took 5110 ticks.
Using " /window -hl " took 10828 ticks. result unsorted
Using " /window -hls " took 73547 ticks.

As you can see a main text window if sorted or not takes near the same time
A listbox took twice the time while unsorted and over 12 times the time when sorted

From what i gather if you are sorting a window it is likely sorted to a memory buffer before being output to its destination.
I expect that this well make it faster than having to construct a hidden window and write to it letting it sort and then savebuf it to a file.

Re: Faster file writing or faster sorting? #159197 16/09/06 06:57 PM
Joined: Feb 2004
Posts: 2,019
FiberOPtics Offline
Hoopy frood
Offline
Hoopy frood
Joined: Feb 2004
Posts: 2,019
Quote:
Well, hash tables can easily be sorted, but it requires writing it to a file first.


Bit of nitpicking, but when you write the file, and sort the file, you are not sorting the hash table. You're simply sorting the file which holds the information from the hash table. Your hash table is unchanged. Hash tables by nature aren't supposed to be sorted, that would defeat their purpose.

I know you meant it differently, but I have seen many people state they sort a hash table (fex with /filter) which is simply a silly thing to say.


Gone.
Re: Faster file writing or faster sorting? #159198 17/09/06 03:29 AM
Joined: Aug 2006
Posts: 22
N
netsplit Offline OP
Ameglian cow
OP Offline
Ameglian cow
N
Joined: Aug 2006
Posts: 22
Quote:

Originally posted by Rand
If you still prefer to actually *write* to a file, you can use:

fopen myfile the_file_here.txt
fwrite -n myfile %data
fwrite -n myfile %data2
fwrite -n myfile %data3
fclose myfile

The above works much faster then the standard /write.



Sweet deal. No lag with that. Thanks heaps.

Last edited by netsplit; 17/09/06 03:37 AM.