mIRC Home    About    Download    Register    News    Help

Print Thread
Large file handling #94155 14/08/04 11:28 AM
Joined: Feb 2004
Posts: 3
A
alcoholic Offline OP
Self-satisified door
OP Offline
Self-satisified door
A
Joined: Feb 2004
Posts: 3
Hi all, I need to read a random string from a 40000 line file, do some "work" on it, then write it to another file, and delete the line in the original file, then do it all over again, until the original file is empty.

My script works perfectly, but it takes AGES.

If I can just scramble all the lines in the original (to another file, or in itself) would help as well, and I will find another way to manipulate the lines after it is scrambled.

Thanks...

Re: Large file handling #94156 14/08/04 01:22 PM
Joined: Dec 2002
Posts: 774
T
theRat Offline
Hoopy frood
Offline
Hoopy frood
T
Joined: Dec 2002
Posts: 774
loadbuf the file to @window, do the stuff and /savebuf


Code:
//if ( khaled isgod ) echo yes | else echo no
Re: Large file handling #94157 14/08/04 01:40 PM
Joined: Apr 2003
Posts: 701
K
Kelder Offline
Hoopy frood
Offline
Hoopy frood
K
Joined: Apr 2003
Posts: 701
Maybe you can speed it up with /fopen and $fread and reading it into a hash file and shuffle that one, if possible while writing out...

But 40k lines seems like a lot for mIRC scripts, so if you don't want your connections to timeout, you might have to do it in 10 times of 4000 lines each or something like that.

Otherwise use dll's or external shuffle programs for this task.

Re: Large file handling #94158 14/08/04 07:12 PM
Joined: Feb 2004
Posts: 2,019
FiberOPtics Offline
Hoopy frood
Offline
Hoopy frood
Joined: Feb 2004
Posts: 2,019
There is no way in mIRC to do this with decent speed, unless if the shuffling is done, as mentioned by Kelder, by an external program or code.

Could you tell us what exactly you want to change? Perhaps there are better ways of achieving what you want, instead of reading a random line and then deleting that same random line.

What's the purpose? What do you want changed in that random line? And why do you have a text file with 40k lines? There are better means to manage large amonts of data, than a text file. A hash table for example.

Greets

Last edited by FiberOPtics; 14/08/04 07:14 PM.

Gone.
Re: Large file handling #94159 15/08/04 09:34 AM
Joined: Feb 2004
Posts: 3
A
alcoholic Offline OP
Self-satisified door
OP Offline
Self-satisified door
A
Joined: Feb 2004
Posts: 3
OK, first of all, when I said that it works perfectly and it is just slow - this is not the case (too much beer smile).

It is for a trivia-game, what I want to do is read a random line, remove the question number from the string, then write it into a new file with question number 1, then 2, 3 etc.

This way the questions get shuffled, and has different numbers. (because some people use the numbers, or the pattern that the questions appear, to cheat)

Now you know what I want to do and why. This is basically what I have so far:

I generate a random number, then $read -l <number>, remove the question number from the string (and use that number to $write -dl <number>), but then I realised if my next random string is lower down in the file, my question number is not going to respond to the line where it is, and I'm deleting the wrong line!

I thought of the $fread, but as far as I could see, there is no way to remove the line with $fwrite.

The reason speed is so important, I want my bot to shuffle when it gets to the end of the file (and then the game has to stop for such a long time).

I suppose I can copy the file and let the bot continue and only stop when the shuffling is complete, and start with the newly generated file. But now I'm back to square one with the removing of the line again. *sigh*

Thanks for the replies so far...

Re: Large file handling #94160 15/08/04 11:08 AM
Joined: Dec 2002
Posts: 788
C
Coolkill Offline
Hoopy frood
Offline
Hoopy frood
C
Joined: Dec 2002
Posts: 788
The following works instantly, no hassle on a file i have here, 40000 lines.

//var %r = $rand(1,$lines([color:red]test.txt
)) | var %l = $read(test.txt,%r) | write $+(-dl,%r) test.txt | write newtest.txt %l
[/color]

%r is the randomized line.
%l is the line it reads from the randomized line.
It then deletes the random line from the file (test.txt)
And add its to the end of newtest.txt.

Edit; Just re-read.

Quote:

but then I realised if my next random string is lower down in the file, my question number is not going to respond to the line where it is, and I'm deleting the wrong line!

i.e. Your deleting line 2, and as a result line 3 is now becoming line 2?, if so, simply replace line two with a blank line and check the $read isnt null when outputting the question.

Eamonn.

Re: Large file handling #94161 15/08/04 09:59 PM
Joined: Apr 2003
Posts: 701
K
Kelder Offline
Hoopy frood
Offline
Hoopy frood
K
Joined: Apr 2003
Posts: 701
A better way to avoid parsing that entire questions file is to do it one question at a time, when you actually ask that question:

-> Keep 2 files one with 'unasked' questions, one with used questions.
-> Each time you ask a question (chosen at random or in sequence), read (and delete) it from first file and add it to the used questions file in a random position.
-> when the unasked questions file is empty, make the used questions file the unasked questions file and make an empty file used questions...

Only thing left is the sequence number of the questions: maybe just have a counter in a %var and increase it by one for each question. Do not put any question numbers in the files. You could decide to reset it to one when you switch the files, but it's not really necessary. (Use $base(%counter,10,10,5) to always get a 5 digit number padded with zeroes. Do reset at 99999 grin )