The memory allocation is Windows based, not anything to do with mIRC. (ie: Blame Microsoft (again..LOL))
As to using a hash table, I honestly don't know and I don't have sufficient files/directories to be able to test a code to see if it would work or not, but as you noted, it probably would be slower than using the /filter (too bad you couldn't combine the two)..
Good luck, and if you decide to try the hash table idea, let us know how (and if) it works, others might need/want the information later.