Forums General Discussion Free Bots

There's really no such thing as "behind the scenes" with threading. Once you add threads to your environment you need a way for the script to synchronize them, it's far too difficult if not impossible to do automatically, especially for a crude interpreter such as mIRC.

Consider a script that relies on the results of a $findfile for a subsequent event-- a setup that in your system would run on two separate threads:



If a user types !count and !report immediately after, mIRC will still be in a $findfile loop on another thread when it returns the total value. It will therefore report a wrong or undefined value. It's impossible for mIRC to synchronize this automatically, because mIRC has no way of knowing what data is being shared across threads. Only the programmer can possibly know the answer to that (and often even the programmer doesn't know, hence the complexity involving threaded code). It is therefore necessary for the user to give the user a way to synchronize this code. Setting %total to $null is not sufficient, because there would still be a race condition that could lead to corruption or data, or worse, crashing of mIRC.

If every command mIRC ran was immutable and stateless, it would be feasible to run (almost) everything in a thread. It is of course impossible to guarantee this with mIRC's language, since data is neither immutable nor stateless.

Reading this thread raised some questions for me. I can guess the answers to most of my own questions, but I would like to see them verified.

First, it's mentioned that mIRC script on default runs as a single thread. Does this also imply that if you run a mIRC bot on a Quadcore, it only uses at most a quarter of the processor's raw processing power?

Second, if you use external .dll files like mthreads.dll to create additional threads, would these threads also run on the same part of the processor, or would they be able to trigger other parts of the processor, causing a gain in speed on Dual- and Quadcores?

Third, does anyone have any experience with mthreads.dll? The documentation is very limited, though the accompanying mIRC code is clear enough. What about version compatibility?

Fourth, if you run separate threads, you of course have to make sure that you're not working with the same variables, since that would screw up your own data results. But what about addressing the same hash table. If I'm using a "$hget(MyTable, ValueA)" in one thread, and a "hadd MyTable ValueB 100" in a parallel thread, does a chance exist that while the $hget of the first thread is executing, the hadd of the second thread is being triggered?

Perhaps the chance of that would be kind of small, but possibly a more practical example: I'm running a bot that stores quite a bit of data in hash tables. To prevent the data from disappearing due to a power loss, I hsave all the hash tables every 10 minutes. This causes mIRC to "freeze" for a good 3-4 seconds every 10 minutes.

Now, if I would use mthreads.dll for saving the data, I shouldn't notice the "freeze" effect, correct? But could there be a problem if another script would be writing data to one of those hash files during that time? I wouldn't mind if that single entry wouldn't be stored (after all, it wouldn't be stored during serial processing), but could it corrupt the data being saved to the harddisk?

In that case I would like to see $findfilecall() and $finddircall(), just like $dllcall() and $comcall() - there has been a relatively few number of bug reports related to these identifiers since their addition to the language and I haven't seen any hideous crash reports.

And perhaps a built-in method of doing what WhileFix.dll does, which I believe is a simple call of TranslateMessage() and DispatchMessage() if memory serves me correctly. This allows the UI to update and new input to be processed even whilst in the middle of a loop.

Yes.



Any threads spawned from a DLL would be eligible for scheduling on all 4 cores. So yes, you can get speed increase from using a dll-- HOWEVER,

This speed increase only applies to the processing done inside your dll (outside of mIRC). Once you start touching mIRC (SendMessage, etc.) you are thrown back into mIRC's single thread. You also have to take care, when using such a threaded DLL, to make sure that you properly synchronize threads. Manipulating mIRC in multiple threads (without proper locking) can easily cause mIRC to crash.



I cannot answer this. I would guess that the dll is a very low level API over pthread or Windows' threading API, and you can probably find much more documentation on MSDN.



Again, it depends how you do it. You would need to create semaphores or mutexes around all of your data accesses, though. This means if your dll does nothing but interact with mIRC, you will basically see no speed increase, because you will constantly be locked in a single thread (you might even see a speed decrease from that). With SendMessage, your thread is going to block until mIRC returns the data anyway (see the remarks on SendMessage threading), so theres no synchronization needed there-- but of course there's no threading done there either (it's essentially going to lock you into one thread again).



I don't know enough about mthreads.dll to comment. It looks like it just separates the script engine from the UI. mIRC will continue to look active but the script layer will be blocked. This may have unwanted consequences, though I'm not sure exactly what. I wouldn't recommend using it, in general.

If it's taking 3-4seconds to save, you've probably outgrown hash tables, or at least outgrown using a single hash table. As I mentioned earlier in the thread, you probably shouldn't be loading all of your data directly into memory if this is the case. You should consider using sqlite or some data store with the ability to index your data and optimize lookups without requiring everything in memory. This would also guarantee persistence, too (no 10 minute timer needed for saving data).

Alternatively you can use multiple hash tables-- one for persistent read-only data storing the bulk of the data but only saved on exit, and one hash table for new and modified entries that were added in your current session only. That table would be much smaller and would be the one that was saved every 5-10 minutes. When you close mIRC (or force a flush to disk) you would move the data from the "current session" table to your persistence table and store that.

$comcall and $dllcall are not (strictly) multi-threaded, which is likely why, if you were expecting to see thread related "hideous crash reports", you saw none. When they do involve threads, they're completely synchronized on mIRC's end, which isn't really the same issue at all. Functionally they are simply asynchronous callback functions that make use of the SINGLE THREADED event loop-- mIRC always responds in the same thread. There's no reason why mIRC can't do the same thing with $findfile, except that it will make lookups much slower-- not in terms of responsiveness, but in terms of time to complete a search.

Note that you can achieve the exact same effect by simply writing a /timer loop over $findfile(dir,*,%i)

Also, if you're really concerned with speed, you can also write your own DLL that does directory traversal with an asynchronous callback by just passing your directory and callback method to $dllcall

It's not so much speed that's an issue, it's the fact that there's no way to avoid the synchronous method of these two identifiers at the moment (without using a messy hack like the timer method you suggested).

It's like how the while ($sockbr) sockread method in a single sockread is faster than using multiple asynchronous sockread events. People use the latter even though it's slower because mIRC can continue to do other things.

You didn't comment on my WhileFix.dll suggestion, do you think that would have any adverse effects?

Thanks argv0, you've been a great help!



Actually, mthreads.dll is a dll specifically written for mIRC to allow multiple scripts to run next to each other. I doubt I'll find anything about a mIRC specific dll in the MSDN library. That or I totally mistook your advice.



I'm already using multiple hash tables, some read-only and some dynamic. The read-only hash tables aren't saved of course. As for the dynamic hash tables, one is over 12mb and will keep growing fast, while the others are under half an mb together. I can probably think of an alternative to the single one large hash table, just haven't come around doing it yet.

I'll leave mthreads.dll and sqlite for possible future plans. Lots of work left to do in the near future. All my calls from and sends to hash tables are aliases, so reworking, while it would be a large project, wouldn't require me to dig through the entire code.

Besides making the script slightly slower, not really. Event processing might occur out of order, but that's about it. WhileFix doesn't thread, it basically forces mIRC to process message events between each loop iteration (or whenever you call on the whilefix dll file). Technically you can use this inside your $findfile call:



Perhaps mIRC could do with a /updateui command that would basically force a processing of the message loop (exactly what whilefix does) and people could just insert those into their long running tasks to make mIRC seem more active. That wouldn't be a bad idea at all.

I know that it's meant for scripts, but it's likely just a thin wrapper over the CreateThread API call. Again, I'm not positive, but I'd bet that would be the case. If it is, MSDN has a LOT of resources on threaded programming in windows-- what can go wrong, what the architecture is like, what your expectations should be, etc. It's a good starting point regardless of mthreads' API.



It's the 12mb hash file that you should be splitting up. I'm sure there's a way to further isolate the data you use. Even if you just split them up randomly (choose table based on $r(1,2)) you could schedule your saves independently and reduce the wait-time. For instance, you could have 2 timers saving each table, each running 5 minutes apart (but at 10 minute intervals)-- then, instead of 3-4 seconds of delay every 10 minutes you would only have 1-2 seconds of delay every 5 minutes. You could continue this division as necessary until you have negligible delays (1/4 second every 2 minutes).

It's not really a maintainable solution if your data is growing fast, but it's yet another optimization you can consider if you've tried other options.

I've thought about splitting it up, but since one table is the vast bulk of the data, that would be pointless. The data stored in that table is pretty much log data, for which retrieving speed is not an issue. Since the bot synchronizes channels on different networks and PMs with different people into one, mIRC's default logging system doesn't cut it.

I could just export the data to logfiles as I go, of course. Just gotta check in on how to do that in a nice matter. I don't want mIRC to access my HD a bunch of times every second, just to write one line.

Then why not flush every 1min when the data is small enough to not delay mIRC? A write every 1min is pretty reasonable. You wouldn't even need to redesign anything, just bump your timer down to 1 minute and be sure to flush your table after the write.

You mean with appending to the hash file? The only problem is that the order would be lost. Per "room", I store 1 hash entry containing the number of lines, and 2 hash entries per line, one storing the $gmt value at which the line was generated, and one storing the actual line.

Actually, ignore that. I could just keep the total number of lines in the hash file and keep on counting.

However, I should probably come around and change it so that I append the individual log entries to the end of the file(s), instead of just dumping the hash table into a file.

TBH, it's on my to-do list, and I should come around doing it one of these days. I just needed some quick and dirty way to start logging back when I started, and was planning on converting them to decent logs at a later time.

What's a proper way to write data to files, assuming I'll probably write more than one line at a time? Would the following example code be decent code, or would you recommend to do it different.



Would savebuf be a faster alternative, if I would display the logs in a @window, instead of writing them to a hash file? If so, is there a way to "mark" until where the window was saved during the last time, or would I have to use a counter per window?

I would love to see an /updateui. Most complaints about mIRC being single threaded seem to be related to UI freezes etc.

Whilst the script would definitely be slower, that could be mitigated by updating the UI after every N iterations, like this:



Would definitely be handy to have

All I can say is benchmark the alternatives if you want to optimize properly. Intuition is often wrong, and making "educational guesses" about implementation details (like for /savebuf, for instance) doesn't always work out properly.