mIRC only supports 1.1.1 versions of OpenSSL.
Quote from the change log:
06/07/2019 - mIRC v7.56
...
4.Updated OpenSSL library to v1.1.1b. Required code changes due to to
changes in OpenSSL 1.1.x. Older OpenSSL versions are no longer
supported. Compiled to work on XP. Dynamic loading of OpenSSL DLLs
still supported as libcrypto-1_1.dll and libssl-1_1.dll.
You will need to wait for Khaled to add support for 3.x.
Side note: Your filenames include "x64" which implies 64bit file. mIRC is 32bit and trying to load a 64bit DLL into a 32bit EXE will not work anyway.
