Forums Scripts & Popups Question about LOADBUF

At the moment i'm dealing with large text files & I use the loadbuf command to load up a file I want to examine into a Dialog List (which is a feature of an text editor i'm working on)

What I find is that if a file is particularly large, say 50000 lines, it will freeze the mirc for quite some time , 30+ seconds

On the other hand when I do the same thing & load it into a custom @window there doesn't seem to be that freezing.

Can anyone help me get rid of this freeze. I don't particularly want to discard my dialog list.

Is there another way of loading up files into the dialog list that doesn't cause this freezing? Alternatively, if it's a non-fix situation, is there anyway I can incorporate a @window to essentially replace the dialog List, and not have it as a floating window but rather a part of the editor.

Note: I don't want to use dll's for this but was wondering if there was a scripting solution

I know someone who was having a similar problem... and im not quite sure this is exactly the same , but i do remember it was an issue with the dialog opening when large amounts of text were involved.
Thread
I hope thats what you needed.

It's not quite what I was after. Thank you though for posting the suggestion.

I actually did try the fopen, looped fread, fclose option and my results were:

for a file 140000 lines long:- took 13min 50 seconds

which is slower than the loadbuf: 9 min 46 seconds.

I don't need to process the text, just load it up.

Interestingly, loadbuf into a custom @window was instant. no waiting whatsoever. However I found the vertical sidebar not really functioning properly. Only could scroll down and view a portion of the file.

The lines, were all definately there though as I could see how many lines and echo lines at end of file using //echo -a $line(@name,N,T)

In the fopen etc option - I took the liberty of adding a simple progress bar to see how it was performing and I could see the process steadily slow down. So I presume it's a memory thing having to perform did -a repeatedly.

I just don't understand how loadbuf into a custom window versus loadbuf into dialog can give such different results.

If anyones found a solution that doesn't involve dll's I'm very keen to hear from them.

I understand that /loadbuf is faster than /fopen. However, I thought using /loadbuf into dialogs took the same time as using /loadbuf into custom @windows. I guess I'll have to try that myself. o_O

I did some tests of my own, and indeed, loadbuffing or filtering to a listbox in a dialog takes ages, whereas to a window is instantly.

The only way is to either use a dll like nhtml or mdx to dock a window in a dialog, or to use whilefix.dll so that your mIRC doesn't freeze. Either way it involves using a dll, and I don't think there's much you can do.

Sorry, Chris2015 just informed me that it didnt do what he needed either.

What I gave him works fine, tested myself, he does need 6.15 for it to work.

There is of course also the possibility that what he asked for (and the way he stated his question) wasn't really what he wanted in reality

This thread is in no relation with Chris' question though.

Greets

He could use a fopen, and a .TIMER 0 0 to load the dialog one line per timer event that self deletes (and calls maybe another alias if more needs doing) when it reaches eof.

On another subject i thought i might ask about...
In my personal experence WhileFix is good when you cant do much else, but I notice a hard to describe laggyness or stuttering laggyness after a very large loop has ended, If i had to amagine what it was, it would be like saying all the calls to whilefix dll, were all being ended, so there is a kinda slowwing as this is being done, or some internal table is being cleared out or a refrence to each call, I felt this due to if the loop was real large this odd slowing (NOT a normal mirc freeze) appears to get longer in accordance to the loop length. Things like changing window etx well be slow to respond, even tho u know nothing else is happening, This whole thing only lasts for maybe 10 seconds, and thats on an old 600mhz pc. Anyone else noticed this, or is it my amagination?

Ahh the timer is a great idea, didn't think it would work, but since the timer works independently it performs nicely.

Did this setup:



Then performed the following:

//fopen big big.txt | dialog -m big big | .timeraddline 0 0 addline

Added the 50000 lines, without freezing mIRC, excellent

Regarding whilefix.dll, I've never really used it, just 1 or 2 times, you could post on ms.org in the comments section of the dll, and the author might reply with some feedback.

Greets

thanks for this suggestion.

Gave it a try and wohoooooo!

no freeze!!

The trade-off it seems is the time it takes to load the entire file though.
I took out the stop-watch and used the same 140000 line file and it took 30 minutes and 30 seconds for all the lines to load up.

Not being one to give up. I tried to manipulate the time using -m flag, which made no difference. Then it occurred to me that instead of using fopen/fread/fclose, why not use the timer on loadbuf and load it up incrementally.

and it works. No freeze whatsoever and the time it takes to load up a 140000 file is 26 minutes & 20 seconds.- small improvement. This is loading up 50 lines at a time.

Now I'm trying to work out what would be the optimum way of incrementally loading up using loadbuf, ie would modifying how many lines per increment make a difference, also would adding a small delay help improve clear memory (assuming its a memory issue?) , eg if timer were set to do after 1 or 2 seconds rather than immediate - would this improve performance.

BTW i did this crude change to the code you posted



then typed this instead:

//dialog -m big big | set %x 1 | set %y 50 | .timeraddline 3000 0 addline

I have a feeling that no matter what you do your in for a considerable delay, I had a thought of using a hidden window whcih well loaad 140000 lines in seconds if even that. Then a filter to put the lines into the dialog, I tested it with a small amount of data first to get it going, 1400 (1/100th), and something I noticed when changing to the large file, it accessed slower outright, ie: if it took 2 seconds to do 1400 when only 1400 were there, it took 6 seconds to do it when 140000 were there, I was thinking of ways around this, and thought i better test to make sure i was right about it going slowwer, so I just repeatedl;y loaded the same 100 lines of a 100 line hidden window into the dialog, and what i saw made me realize there are far more problems involved than just accessing the data to put in quickly, The big problem was , as more and more lines were added to the dialog, the time to add each 100 slowed down.

In the end i used this


I used filename c:\bigfile.txt, but you could change that
If u run this you well see how the time to complete each 1000 line pass increases, dramaticly by the end of the file.

PS: i do acknoledge that as this moves down the file, the loadbuf command takes longer anyway, since it must read further down the file, however when i did a loadbuff to a window instead of a dialog, each pass was still well under 1 second, while the loadbuf to a dialog was taking 20 odd seconds each pass near the end.


Conclusion. Its only ever going to get So fast and thats that. Just loading the dialog slows the more you add.

I have a feeling you're right.

I did notice however that the amount of lines per pass makes a big difference in the overall result.

In my example, I used 50 lines per pass which took 26 minutes. Using your code with 1000 lines per pass dropped it down to 11 minutes on the laptop I use.

I did try and manipulate timers once again thinking it is a memory thing. I'm not too technically minded about software, but my thoughts were that a certain amount of memory is allocated each time loadbuf executed (which clears once the event is finished), and if its executed several times before the first pass is over, you are simply allocated more memory at the expense of speed, and not allowing it to clear.

Anyway, like I said I'm not that knowledgable about the inner workings of software so I can be excused it all of the above is nonsense. But what I did try to do was space out the timer so that instead of being called immediately, it would occur every 3 seconds. That didn't make any difference.

Next, I tried to increment the timer, so that every time it reached a 10000 mark in the lines it was to buffer, it would increment one second,

I used the %y variable in the timer which started at zero & change the following line in testit


as well as:


Anyway, that didn't work either. Then I tried to create a "pause" every 20000 lines, thinking, if the whole process were to pause for a second or two, it would allow some memory to be freed up then continue fresh at a faster speed.

so instead of increasing %y, it remained 0, but instead i changed my line in addline to:

if (20 // %a) { var %v = 1 | while (%v <= 100000) { inc %v } | echo -st Pause }

Anyway. That only made the whole thing take longer than 11 minutes.

So anyway, I think the only real solution can't be found in scripting a solution to this, but rather with performance enhancing dll's - at least until this bug is fixed (if it can even be called a bug - it's probably more a case of me trying to do something that dialog lists were never designed for)