mIRC Home    About    Download    Register    News    Help

Print Thread
#271437 06/03/23 08:00 AM
Joined: Jan 2004
Posts: 2,127
maroon Offline OP
Hoopy frood
OP Offline
Hoopy frood
Joined: Jan 2004
Posts: 2,127
I have found 2 issues related to the buffer for sendmessage replies from mIRC back to a DLL:

1. Even though mIRC is telling the DLL that dBytes is 20480, and I can get a DLL to correctly receive a DATA parameter containing a text string longer than 10300 bytes, the request for a sendmessage reply sends back nothing if the reply string is greater than 8192, and I was expecting the size to match the 100+$maxlenl of the DATA buffer.

2. There is a significant slowdown when the sendmessage reply is divided into fewer large replies than when divided into many small replies.

I thought it might have been related to the fix of the issue reported in
https://forums.mirc.com/ubbthreads.php/topics/265770/large-number-of-sendmessages-unset-binvars

...but I'm seeing the same rate of slowdown in v7.56 prior to that fix, though I had to use smaller &binvar to avoid the fixed glitch.

--

After making some modifications to Saturn's sha2.dll, I benchmarked my speed vs the original, and was surprised to find that mine was nearly twice as slow when handling large &binvar.

After eliminating other possible causes, I finally tracked it down to the size of the sendmessage reply coming from mIRC back to the DLL. I had assumed that handling the data as fewer larger chunks would be faster than the overhead of making a larger quantity of smaller messages, so based on the increased value for $maxlenl I had increased the number of bytes requested that $bvar(&var,offset,chunk-size) send back to the DLL.

This alias shows the effect of the $bvar string length on the speed. Depending on the string used in /breplace this is creating a &binvar containing either 1/2/3 digit numbers, and the text length of the $bvar(&var,offset,bytes) return sendmessage for a string containing D-digits would be 1037*(D+1)-1. Times are all close to 10x faster when changing 10^7 to 10^6

//var %digits 1 | while (%digits isnum 1-3) { bunset &v | bset &v $calc(10^7) 0 | var %byteval $str(1,%digits) | breplace &v 0 %byteval | var %i 1, %t $ticks | var %a $dll(sha2.dll,sha256,1 &v) | echo -a bytes $bvar(&v,0) byte val: $bvar(&v,1) time: $calc($ticks - %t) | inc %digits } | dll -u sha2.dll

result:
bytes 10000000 byte val: 1 time: 18049
bytes 10000000 byte val: 11 time: 27925
bytes 10000000 byte val: 111 time: 34352

The slowdown is directly related to the length of the sendmessages, so it was nearly twice those times when requesting the same total data by requesting twice as many 518 byte chunks, and was nearly double these benchmark times by requesting half as many 2048 byte chunks.

I cannot see that the code itself is handling the received data in a way that would be causing the slowdown. I had made a similar change to how the DLL reads diskfiles, and was able to get a significant percentage speed boost by increasing the disk read buffer from 4kb to 16kb, so I had assumed I could get the same thing when handling &binvar by requesting strings as long as $maxlenl

The slowdown is also not affected by whether or not I unload the DLL prior to each of the 3 calls.

I encountered issue#1 by trying to edit the DLL to benchmark chunks of 1037*2=2074 bytes. I was getting the expected nearly-double time for 2074-byte chunks when the bytes had 1 or 2 digits each, but was getting just 16 ticks for chunks having 2074 3-digit bytes. The largest size of a chunk that wasn't doing this was 2048, so because the $bvar(&var,offset,2048) length for byte values 111's is 4*2048-1, this means I was getting a blank reply when the $bvar reply string was longer than 8192, which fooled the DLL into thinking the &binvar was zero length.

Joined: Dec 2002
Posts: 5,411
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 5,411
Thanks for your bug report. Neither of these are bugs as far as I can tell. The maximum length is 8192 and has been for as long as I can remember. As for the speed, there is quite literally nothing that can be changed in the sendmessage routine to speed it up. The slowness is likely due to the calls to MapViewOfFile() and VirtualQuery(), both of which are related to shared memory, not to mention calls to the alias/identifier routines.

Joined: Jan 2004
Posts: 2,127
maroon Offline OP
Hoopy frood
OP Offline
Hoopy frood
Joined: Jan 2004
Posts: 2,127
Updated bvar_bench3

For sure it looks like the $bvar(&var,offset,M) is much slower when showing the byte numbers than when it shows the .text output. This updated alias is extremely faster than bvar_bench2 by making 1 slight change. Instead of requesting a list of byte value integers from $bvar(&var,offset,M) directly, it instead /bcopy's the M range to a different &binvar, then uses $encode to encode it to mime, then outputs the mime string using the .text property.

The combined time for the /bopyc + $encode +$bvar().text now 10x faster than $bvar() for the M=2560 length, and instead of seeing many-shorter-strings being faster, I'm now seeing the expected behavior of M=100 being slower due to the overhead of making many more calls to $bvar(). And now since the text mime string is now 4/3rd as long as M instead of being a list of integers that's 4x as long, it's now safe to use M=7680, and the M=7680 output from bvar_bench3 is now around 20x faster than the M=2560 output from bvar_bench3, even though those are both receiving identical string lengths and the M=2560 is making only 3x as many calls to $bvar.

Code
time M = 2560 loops 390 bvar string length: 3416 ticks: 89 vs baseline 100%
time M = 2048 loops 488 bvar string length: 2732 ticks: 116 vs baseline 130.3%
time M = 1037 loops 964 bvar string length: 1384 ticks: 197 vs baseline 221.3%
time M = 1000 loops 1000 bvar string length: 1336 ticks: 169 vs baseline 189.8%
time M = 500 loops 2000 bvar string length: 668 ticks: 263 vs baseline 295.5%
time M = 400 loops 2500 bvar string length: 536 ticks: 274 vs baseline 307.8%
time M = 300 loops 3333 bvar string length: 400 ticks: 345 vs baseline 387.6%
time M = 250 loops 4000 bvar string length: 336 ticks: 414 vs baseline 465.1%
time M = 240 loops 4166 bvar string length: 320 ticks: 418 vs baseline 469.6%
time M = 220 loops 4545 bvar string length: 296 ticks: 471 vs baseline 529.2%
time M = 200 loops 5000 bvar string length: 268 ticks: 503 vs baseline 565.1%
time M = 180 loops 5555 bvar string length: 240 ticks: 546 vs baseline 613.4%
time M = 160 loops 6250 bvar string length: 216 ticks: 611 vs baseline 686.5%
time M = 140 loops 7142 bvar string length: 188 ticks: 693 vs baseline 778.6%
time M = 120 loops 8333 bvar string length: 160 ticks: 836 vs baseline 939.3%
time M = 110 loops 9090 bvar string length: 148 ticks: 908 vs baseline 1020.2%
time M = 100 loops 10000 bvar string length: 136 ticks: 998 vs baseline 1121.3%
time M = 90 loops 11111 bvar string length: 120 ticks: 1096 vs baseline 1231.4%
time M = 80 loops 12500 bvar string length: 108 ticks: 1235 vs baseline 1387.6%
time M = 7680 loops 130 bvar string length: 10240 ticks: 47 vs baseline 52.8%

alias bvar_bench3 {
  var %digits 3
  var -s %list 2560 2048 1037 1000 500 400 300 250 240 220 200 180 160 140 120 110 100 90 80 7680, %first $gettok(%list,1,32)
  while ($gettok(%list,1,32) != $null) {
    var %M $v1, %number $str(1,%digits) , %size 10 ^ 6, %loops $ceil($calc(%size // %M)) , %j %loops,%offset 1
    bunset &v | bset &v %size 0 | breplace &v 0 %number | var %t $ticksqpc
    while (%loops) {
      bcopy -c &v2 1 &v 1 %M | noop $encode(&v2,bm)
      var %a $bvar(&v2,1-).text
      inc %offset %M
      dec %loops
    }
    var %time $calc($ticksqpc - %t) | if (%M == %first) var %baseline %time
    echo -a time M = %M loops %J bvar string length: $len(%a) ticks: %time vs baseline $calc(1000*%time //%baseline /10) $+ %
    var %list $gettok(%list,2-,32)
  }
}

Joined: Dec 2002
Posts: 5,411
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 5,411
Okay, so this is related to $bvar() when N,M is used. Looking at the code, there is an strlen() in the loop which would likely cause an issue for long strings. This has been changed for the next beta.

Joined: Jan 2004
Posts: 2,127
maroon Offline OP
Hoopy frood
OP Offline
Hoopy frood
Joined: Jan 2004
Posts: 2,127
Wow, a big difference in speed in the new beta. $bvar(&var,offset,length) is now about 25x faster in bvar_bench2a than it used to be for the max string length for the list of integers, and it has the expected behavior of becoming slower if you chop it up into many short strings, so the 'sweet spot' for the fastest speed is now fewer $maxlenl length strings instead of many strings having only 100 integers in them.

It's now faster than the bvar_bench3a method that did bcopy-then-mime-then-bvar(&var,1-).text it, as it should be, though that alternative is also now faster, taking only around 2/3rds of the time that it used to take.


Link Copied to Clipboard