I'm looking into how to make the /CHALLENGE command work in mIRC, and while it'd be neat if it was just directly supported, it's such a niche feature that I won't hold my breath. /CHALLENGE is a means of obtaining IRC operator status on a handful of networks that uses an RSA key-based challenge/response method instead of a typical /OPER password. I'll detail the exact protocol below:

  1. The client issues /CHALLENGE opername to the server
  2. The server responds with one or more lines of the 740 numeric (which contains the base64-encoded challenge text), for example :irc.example.com 740 opername :Abd84Dd0d8s14dddd98492
  3. The server sends a 741 numeric to indicate that it is done sending base64-encoded challenge text
  4. The client takes the base64-encoded challenge text from all of the 740 lines, combines them together, and then base64-decodes them to obtain the encrypted challenge
  5. The client decrypts the encrypted challenge using its RSA private key and PKCS#1 OAEP padding to obtain the decrypted challenge
  6. The client hashes the decrypted challenge using SHA-1 and base64-encodes the binary result of that hash to obtain the encodedHash
  7. The client responds to the server with /CHALLENGE +encodedHash (note the leading + character)

If an error occurs somewhere in the process, either a 464 or 491 numeric will be issued.

Supporting this command natively poses a handful of challenges regarding the generation and storage of the RSA keys, however this would be achievable in mIRC scripting if it supported the following additional identifiers (which should be a lot easier to support as they can be effectively wrappers around functions present in OpenSSL):
  • Open a potentially password-protected RSA key and store it for later use--this could read directly from the file the key is stored in, or it could be passed a string that was obtained via successive calls to $read()
  • RSA decryption
  • PKCS#1 OAEP padding removal
  • base64 encoding/decoding