Forums Developers Large file handling

Hi all, I need to read a random string from a 40000 line file, do some "work" on it, then write it to another file, and delete the line in the original file, then do it all over again, until the original file is empty.

My script works perfectly, but it takes AGES.

If I can just scramble all the lines in the original (to another file, or in itself) would help as well, and I will find another way to manipulate the lines after it is scrambled.

Thanks...

loadbuf the file to @window, do the stuff and /savebuf

Maybe you can speed it up with /fopen and $fread and reading it into a hash file and shuffle that one, if possible while writing out...

But 40k lines seems like a lot for mIRC scripts, so if you don't want your connections to timeout, you might have to do it in 10 times of 4000 lines each or something like that.

Otherwise use dll's or external shuffle programs for this task.

There is no way in mIRC to do this with decent speed, unless if the shuffling is done, as mentioned by Kelder, by an external program or code.

Could you tell us what exactly you want to change? Perhaps there are better ways of achieving what you want, instead of reading a random line and then deleting that same random line.

What's the purpose? What do you want changed in that random line? And why do you have a text file with 40k lines? There are better means to manage large amonts of data, than a text file. A hash table for example.

Greets

OK, first of all, when I said that it works perfectly and it is just slow - this is not the case (too much beer ).

It is for a trivia-game, what I want to do is read a random line, remove the question number from the string, then write it into a new file with question number 1, then 2, 3 etc.

This way the questions get shuffled, and has different numbers. (because some people use the numbers, or the pattern that the questions appear, to cheat)

Now you know what I want to do and why. This is basically what I have so far:

I generate a random number, then $read -l <number>, remove the question number from the string (and use that number to $write -dl <number>), but then I realised if my next random string is lower down in the file, my question number is not going to respond to the line where it is, and I'm deleting the wrong line!

I thought of the $fread, but as far as I could see, there is no way to remove the line with $fwrite.

The reason speed is so important, I want my bot to shuffle when it gets to the end of the file (and then the game has to stop for such a long time).

I suppose I can copy the file and let the bot continue and only stop when the shuffling is complete, and start with the newly generated file. But now I'm back to square one with the removing of the line again. *sigh*

Thanks for the replies so far...

The following works instantly, no hassle on a file i have here, 40000 lines.

//var %r = $rand(1,$lines([color:red]test.txt)) | var %l = $read(test.txt,%r) | write $+(-dl,%r) test.txt | write newtest.txt %l
[/color]

%r is the randomized line.
%l is the line it reads from the randomized line.
It then deletes the random line from the file (test.txt)
And add its to the end of newtest.txt.

Edit; Just re-read.


i.e. Your deleting line 2, and as a result line 3 is now becoming line 2?, if so, simply replace line two with a blank line and check the $read isnt null when outputting the question.

Eamonn.

A better way to avoid parsing that entire questions file is to do it one question at a time, when you actually ask that question:

-> Keep 2 files one with 'unasked' questions, one with used questions.
-> Each time you ask a question (chosen at random or in sequence), read (and delete) it from first file and add it to the used questions file in a random position.
-> when the unasked questions file is empty, make the used questions file the unasked questions file and make an empty file used questions...

Only thing left is the sequence number of the questions: maybe just have a counter in a %var and increase it by one for each question. Do not put any question numbers in the files. You could decide to reset it to one when you switch the files, but it's not really necessary. (Use $base(%counter,10,10,5) to always get a 5 digit number padded with zeroes. Do reset at 99999 )