Forums Bug Reports Problem with huge textfiles

When I insert a line into a textfile which has 50.700 lines (3,08 MB [3.238.962 Bytes]), mIRC deletes the whole file.

Please don't tell me to split my files. I do it atm. But that's no real solution.
_______________

Try to use the File Handling Commands.. That must solve your problem..
Example of using..

Okay.. thanks.. I will check that out. Fseek is the command to set the line-position I guess.

hmm.. next problem

I cannot use fwrite to insert stuff somewhere. It overwrites the file at the position I set via fseek. So I first would have to shift that stuff and get space. But how to do this?

I just made a 51,000 line file each line being 62 bytes long + cr lf, so 64 bytes each line, total bytes 3,264,000.
then i did this...

/write -il2 myfile.txt blah line 2

it added it fine.

So i suggest you scan your file for EOF markers byte value 0x1A (eof) which well terminate the file.

Also be aware that any byte value 0x00 (null) well be removed. (this appears to be mirc related maybe),
also a byte value 0x0A (lf) not preceded by a 0x0D (cr) well be replaced with 0x0D 0x0A (cr lf)

I well say tho there is a known bug with inserting lines.
I kept running into it, its that you only get to do 65535 inserts before mirc just wont do anymore untill you restart.

I uses this now.

window -c @realbigfilebuffer
window -hn @realbigfilebuffer
filter -cfw myfile.txt @realbigfilebuffer *
;
iline @realbigfilebuffer 2 Blah line 2
;
filter -cwf @realbigfilebuffer myfile.txt *
window -c @realbigfilebuffer

Hmm. I haven't found the need to use the new file handling commands, but I do believe iLLuMiNaT is correct from what I know about the API these commands use. /fseek simply lets you begin over-writing at a specific byte offset, not insert. You would have to fseek, read to a binvar, write your line, then write your binvar. If you wanted to delete a line, you would have to /btrunc before at the point you wish to delete and then write your binvar.

Edit: Sorry, I didn't notice you were talking about plain-old /write in your most recent post, so this only applies to your earlier post.

Yeah, inserting/overwriting lines is possible with FH routine but its a bit of screwing around.

Why don't you use /loadbuf|/savebuf there instead of /filter?

/loadbuf and /savebuf both create hard wrapping, dependant on the window's size at the time of the load or save. /filter does not pay attention to the window's width, and most accurately preserves the information to file (unwrapped) and to window (dynamically wrapped).

In My Humble Opinion... /loadbuf and /savebuf should be depreciated functions. They are not consistant with the way that mIRC internally logs or displays buffer text, however, /filter is.

I'm at the work atm and I could'nt check your "window-idea" out, but doesn't that mean that I have to read 3Megs or more later into a window and save from there? How about the RAM and system performance of my system then?

Anyways, I will check your idea out at home. Thanks.

"/loadbuf and /savebuf both create hard wrapping, dependant on the window's size at the time of the load or save."
Hmm thats odd, if i do this......both files appear to have only 1 line of 900 chars long.

Giving the window a size and not hiding it doesn't affect it either
( window @@ 1 1 300 60)

Try these:

This is how a /savebuf'd file appears in Notepad. (hard wrapping)
[color:000099]//window -e @@ | echo @@ $str(abc,200) | savebuf @@ foo.txt | run notepad foo.txt[/color]

This is how a /savebuf appears after being /loadbuf'd in a larger window.
[color:000099]//window -edo @@ 100 100 500 500 | echo -i @@ $str(abc,200) | savebuf @@ zig.txt | window @@ 100 100 700 500 | loadbuf @@ zig.txt[/color]

This is how a /loadbuf appears when loading a standard log with long lines. (hard non-wrapping)
[color:000099]//write yarf.txt $str(abc,200) | window -edo @@ 100 100 500 500 | loadbuf @@ yarf.txt[/color]

* Raccoon lix Iori's nose.

Wooohooo dicing with distaster there dude.

What about just set the pointer, to the line your removing then write the binvar out, then trunc at the end of it. much safer than storing part of the file in memory even for a few milliseconds , you never know when some yahoos gonna kick the power cord outa the wall.

Personall prefrence is my only cause I used filter once, it worked I never used the other one, that and sometimes i want the files sorted if the line order makes no difference so ill add a "-t c s" on the way in or out depending on the need at the time.

I dont know if the reported (here) problems with load/savebuf are there or not, as i never used it.

I also just dont like reading files, I have had some wierd things happening when a file has by chance a number on the first line (due to it defualting to representing the number of lines in the file) and its not that its just the data of the file.
From memory it was something like i would do a "$read(file.txt,tw,*whatever*)" then a "write -il $+ $calc($readln + 1) file.txt This Line Inserted After One Matched" and if that $readln was the last line of the file I would find the line inserted before the last line instead of after it. (dont quote me on thats what it is exactly, but it was some abnormal thing)

True, perhaps... but the file never (typically) gets physically written until the file is closed. Of course, this all depends on your hardware capabilities and available cache. Chances are though such a senario would be tragic in either case... but I agree that a btrunc off the end may be faster too.

(1) yep thats what your doing,
(2) 3megs well be needed (come on who hasnt got 3 megs of ram lying around the garden shed)
(3) performance wise, its a toss up, maybe better maybe worse depends where the files your accessing are

A normal "write -i" reads the file in and line by line remakes it in a temp file adding the inserted line at the right time, then carrying on tell the end of the file, it then deletes the original file, and either renames the temp file or copies it to the original filename (if its not on the same drive as the mircdir which is where the temp file is made)

So just from that if the file your writting to is no another drive your doing double the disk reads & writes already.

personally i like the hiden window buffer, becuase i dont always close it after doing an update, I keep it there with essentially an in mircwindow copy of the file, i do the iline dline etc etc and just keep sending the file to disk with a filter command, then your never rereading the file from disk. of course thats only practical if your writting all the scripts.

LOL IM BACK!

Bummer : Just did a test on something to do with what i said to make sure, and decided to do a dos "TYPE TESTFILE.TXT" anyone ever heard the PC beep 200,000 times , i couldnt get the dos window to respond, couldnt close it, couldnt reset the pc due to other things running i needed to let finish, cracked the top in the end and pulled the speaker cable out!

That's a diifferent kettle of fish, you are echoing the lines to @Window and then using /savebuf.

What Dave is doing is /filter in - /filter out. For that /loadbuf - /savebuf works the same except slightly quicker.

BTW: Try those same examples with /aline replacing /echo.

This is how I /loadbuf a standard log with long lines. (Wrapped and indented )
//window @Logs | loadbuf -pi @Logs logfile.log

"Personall prefrence is my only cause" Oh yeah OK, fair enough then.

I had a sudden thought and was going to say a /btrunc couldnt be used actually while the file is open with a handle, but it looks like it can.

What i well say is a trunc well write the file size to the directory info, so is likely to flush out to the hdd.

hey would be a big binvar if say you needed to delete line 14 of a 60,000,000 line file <grin>

I looked at the disk reads and writes for a /filter in and out, and a /loadbuf /savebuf , and not to supprising there identical, i would have a feeling it uses the same routines, maybe /loadbuf & /savebuf bypass the filter section with a internal flag.

Not that im doubting you, but how did u manage to show one was faster than the other, i keep getting results within a few ticks of each other doing a 250,000 line file 15meg in size 5 times, then getting ticks diff from start to end