mIRC Home    About    Download    Register    News    Help

Print Thread
/hsave problem #220363 12/04/10 11:55 AM
Joined: Jan 2010
Posts: 10
B
bebegood Offline OP
Pikka bird
OP Offline
Pikka bird
B
Joined: Jan 2010
Posts: 10
Hello, i found a error probably, when my mirc hsave my hashtables, it's gonna be lagging, you can see screenshot under
(yes i check it, lagging only when hsave)

Quote:




first settings:
Quote:

timerSAVE_DataBase.1 -o 0 300 hsave -si DataBase.1 SystemFile\Storage.ini [DataBase.1]
timerSAVE_DataBase.2 -o 0 300 hsave -si DataBase.2 SystemFile\Storage.ini [DataBase.2]
timerSAVE_DataBase.3 -o 0 300 hsave -si DataBase.3 SystemFile\Storage.ini [DataBase.3]
timerSAVE_DataBase.4 -o 0 300 hsave -si DataBase.4 SystemFile\Storage.ini [DataBase.4]


2nd:

Quote:

timerSAVE_DataBase.1 -o 0 300 hsave -s DataBase.1 SystemFile\database1.txt
timerSAVE_DataBase.2 -o 0 300 hsave -s DataBase.2 SystemFile\database2.txt
timerSAVE_DataBase.3 -o 0 300 hsave -s DataBase.3 SystemFile\database3.txt
timerSAVE_DataBase.4 -o 0 300 hsave -s DataBase.4 SystemFile\database4.txt



with 2nd Settings, there no lags more, so probably it's about hsave to .Ini file, i hope u will fix it in next version.

Regards.
sry for my bad english.

Last edited by bebegood; 12/04/10 11:55 AM.
Re: /hsave problem [Re: bebegood] #220395 13/04/10 10:05 AM
Joined: Jan 2003
Posts: 1,063
D
Doqnach Offline
Hoopy frood
Offline
Hoopy frood
D
Joined: Jan 2003
Posts: 1,063
if you have a really big data set it can be expected, because saving it are disc writes.

what you are trying to do is write 4 sets of data to files at approximately the same time, which means your harddisc will have to switch around 4 different locations, trying to write all the data. if your harddisc is really heavy on usage, it can cause lag on your whole system.

one tip would be, that when it's not critical that it has to be done at the same time, to delay some of the saves so they are more sequential instead of simultanius.

ini files will take more processing aswell since it depends on windows API's which are most likely slower compared to normal writes.

this doesn't seem like a bug at all, bit is simply a hardware/OS limitation. but it highly depends also on what you are writing and what the actual difference is between the two attempted methods.


If it ain't broken, don't fix it!
Re: /hsave problem [Re: Doqnach] #220438 14/04/10 05:54 PM
Joined: Feb 2006
Posts: 32
Daveoh Offline
Ameglian cow
Offline
Ameglian cow
Joined: Feb 2006
Posts: 32
mIRC isn't a multi-threaded application (Apparently it is) mIRC scripts aren't multi-threaded anyway, so it would save them in sequence, one after the other.

mIRC 7 does seem to take considerably longer to save/load hash tables though. Here's a measurement I took of loading a hash file (36315 items):
v7.0 - 3500ms~
v6.35 - 1600ms~

However, v7 seems to save hash files differently. After saving the same data to a file, it gains 72630 bytes compared to v6.35.
Edit: This appears to be due to v7 appending CR as well as LF at new lines when saving, even without using the -b switch.

If I load the file saved by v7 on v6.35, it takes about 5100ms~ to load, but after saving it again on v6.35, it reduces file size and loads at 1600ms~ again. The load time is barely changed using either file on v7.0.
So what's going on here?

Edit: v6.35 also seems to be up to 20x faster at saving, 100ms compared to 2000ms (not averages)

Re: /hsave problem [Re: Daveoh] #220454 15/04/10 07:33 AM
Joined: Jan 2003
Posts: 1,063
D
Doqnach Offline
Hoopy frood
Offline
Hoopy frood
D
Joined: Jan 2003
Posts: 1,063
khaled did state he made a change to the INI file handling. maybe this has negatively impacted /HSAVE more than expected.

it's good to maybe have some more tests of this on different systems? (XP, Vista, 7)


If it ain't broken, don't fix it!
Re: /hsave problem [Re: Doqnach] #220802 29/04/10 06:08 PM
Joined: Feb 2006
Posts: 32
Daveoh Offline
Ameglian cow
Offline
Ameglian cow
Joined: Feb 2006
Posts: 32
Any update on this? Preferably from Khaled? It takes so long to start mIRC now! (Loading a hash file on load)

Re: /hsave problem [Re: bebegood] #220910 02/05/10 04:30 PM
Joined: Dec 2002
Posts: 5,000
Khaled Offline
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 5,000
I have not been able to reproduce this issue here so far. Can you provide me with a small, short, minimal script that runs under v6.35 and v7.x that reproduces the issue for you?

Re: /hsave problem [Re: Khaled] #220932 02/05/10 10:46 PM
Joined: Feb 2006
Posts: 32
Daveoh Offline
Ameglian cow
Offline
Ameglian cow
Joined: Feb 2006
Posts: 32
(Presuming you asked me, either way:) I made a script available at http://dl.dropbox.com/u/234274/mIRC/htlcmp.mrc (script info in file)
I used the following versions for testing: mIRC 6.35, 7.01 and 7.02
I used the hash table generator (which creates and saves a table) to create the test file. (Side note at bottom of post)
The file I tested with: http://dl.dropbox.com/u/234274/mIRC/test702.hsh
I ran the generated file through each version and obtained these results:
Code:
6.35: 646 ms avg
7.01: 1484 ms avg
7.02: 2552 ms avg

Note: I created a hash table using each version of mIRC because I remember that 7.0 files took forever to load on 6.35 (or possibly vice versa), although I couldn't test 7.0 without getting a "Beta expired" notice. Between the versions I tested with, there was no noticeable difference in load time between files made in different versions. HOWEVER, 7.xx does save files using CRLF instead of just LF as in 6.35 (no switches). The only difference this makes is file size. According to the help file, this should only happen when using the -b switch.

Re: /hsave problem [Re: Daveoh] #220945 03/05/10 03:07 PM
Joined: Dec 2002
Posts: 5,000
Khaled Offline
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 5,000
Thanks for the script. The issue you are describing is different from the original post in this thread (although it might be related), where the user reported delays when saving hash tables as ini files.

I was able to reproduce your issue. It is taking longer because mIRC has to convert each line to/from Unicode/UTF-8 when loading/saving. Older versions of mIRC saved files as single-byte character data and did not require conversion. Unicode uses two bytes to store text, so it needs to be converted to UTF-8 before it is saved to a file. The UTF-8 then needs to be converted back to Unicode when it is loaded.

mIRC v7.02 is taking even longer because of the BOM handling. I have optimized that and it will be back to the same speed as v7.01 in the next version.

Regarding the CRLF difference: mIRC now uses a standard set of file-handling routines that are used consistently throughout mIRC. Older versions of mIRC were not consistent across routines in whether text ended with an LF or CRLF. The /hsave entry in the help file is referring to the difference between saving as text or as binary data, which is not related to this change.

Re: /hsave problem [Re: Khaled] #220952 03/05/10 10:38 PM
Joined: Oct 2003
Posts: 3,918
A
argv0 Offline
Hoopy frood
Offline
Hoopy frood
A
Joined: Oct 2003
Posts: 3,918
What about storing binary hash table dumps (non .ini files) out as UTF-16 from now on? That would make files /hsave'd in 7.x unusable in 6.x but that shouldn't be such a big deal. If you read the BOM you can just grab the contents without any conversion (which I suspect mIRC already does with the new BOM support anyway).


- argv[0] on EFnet #mIRC
- "Life is a pointer to an integer without a cast"
Re: /hsave problem [Re: argv0] #220965 04/05/10 08:01 AM
Joined: Aug 2006
Posts: 183
T
Thrull Offline
Vogon poet
Offline
Vogon poet
T
Joined: Aug 2006
Posts: 183
Or at the very least, give us an option when we save/load them to save/load the as Mirc 6.x has. If unicode is slowing them down a significant amount, allowing us to save them in non-unicode may be helpful.


Yar
Re: /hsave problem [Re: Thrull] #220973 04/05/10 11:28 AM
Joined: Oct 2003
Posts: 3,918
A
argv0 Offline
Hoopy frood
Offline
Hoopy frood
A
Joined: Oct 2003
Posts: 3,918
It's not "unicode" that's slowing it down. It's the fact that the data is non-UTF16 that's slowing it down, so an option wouldn't be helpful besides for compatibility sake. Even if the data was ASCII, mIRC would still need to convert it to UTF-16 when reading it, since mIRC stores all internal data as UTF-16. This is why I suggested storing the data as UTF-16, so mIRC would not need to perform any conversion.


- argv[0] on EFnet #mIRC
- "Life is a pointer to an integer without a cast"