mIRC Home    About    Download    Register    News    Help

Topic Options
#261892 - 07/12/17 07:37 PM $encode(data,<c|cr|e|er>,key,string) bug
maroon Offline
Hoopy frood

Registered: 12/01/04
Posts: 966
$encode is not consistent with how it handles the presence of an unused 4th parameter without the presence of either the 's' or 'i' switches which make use of it.

The behavior changes depending on the composition of the 4th parameter, but doesn't seem to be affected by whether or not the length is 8 or by the length of the text parameter. Most values for the 1st character return the 1 digit, but a few of them return $null, and there appears to always be at least one 1st digit character for each rest-of-string which allows the 4th parameter to be ignored, with encryption returning the same value as if only 3 parameters were used.

This example has several first-byte-values that return the normal value by ignoring the not-used 4th parameter, but many last-7 patterns return a normal value only for the hyphen or for only strings beginning with '1'.

Code:
//clear | var %i 1 | while (%i isnum 1-255) { echo -a when 1st char of param4 is $ $+ chr( %i ) returns -> $encode(text,cr,key, $+($chr(%i),1a34567)) | inc %i }



The behavior for binary variables is similar, except the cases which returned length '1' in the above example leave &binvar unchanged instead of being the number of characters written to it. Perhaps 45 being the length of encoding 1 block using a salt/iv and '1' being a common return value is related to why encoding is done as normal for many 4th parameters which begin with $chr(45) hyphen or a '1'.

Code:
//clear | var %i 1 | while (%i isnum 1-255) { bset -tc &binvar 1 BLOWFISH123 | var %result $encode(&binvar,bcr,key, $+($chr(%i),1a34567)) | echo -a when 1st char of param4 is $ $+ chr( %i ) returns -> result: %result output: $bvar(&binvar,1-).text  | inc %i }


Top
#261896 - 07/12/17 08:08 PM Re: $encode(data,<c|cr|e|er>,key,string) bug [Re: maroon]
Khaled Offline


Planetary brain

Registered: 04/12/02
Posts: 4295
Loc: London, UK
Thanks for your bug report. This is by design. $encode() originally only supported three parameters, with the last two parameters being optional. It was then extended to handle encryption that required different parameter types and different numbers of parameters. However, it was requested that support for N be maintained as the last parameter in all cases. By not specifying the s or i switches, you are telling encode that you are not providing a fourth parameter for a salt or iv. In this case, the fourth parameter becomes N.

Top
#264449 - 30/11/18 10:27 AM Re: $encode(data,<c|cr|e|er>,key,string) bug [Re: Khaled]
maroon Offline
Hoopy frood

Registered: 12/01/04
Posts: 966
Your reply indicated that the encryption switches were supposed to support the N parameter, and it appears that $encode is trying to support that parameter.

In addition to the other bug report where $decode is always ignoring the chunk N parameter - when using encryption switches, the $encode support for the chunk N parameter is not working correctly because:

* the 16-byte "Salted__" or "RandomIV" counts as part of the 45 input bytes to $encode encrypted N=1 chunk.
* Each chunk represents 45 input bytes, which is not a multiple of the 64-bit 8-bytes of the Blowfish cipher's encryption block.

Chunk#1 is shortened and garbled due to an incomplete 8-byte Blowfish block at end of the first 45 bytes, which consists of the 16-byte header + 3 Blowfish blocks @ 8 bytes + 5 bytes of the next Blowfish block being split across 2 chunks. The 3 Blowfish blocks are the only portion of CBC Chunk#1 which decodes correctly:

Code:
//var %chunk 1 , %switch mc , %a $encode($regsubex($str(x,150),/x/g,$base(\n,10,10,3)),%switch,key,%chunk) | echo -a %a -> $chr(22) $decode(%a,%switch,key)



CBC Chunk#1 is shortened and garbled due to incomplete block at end of the first 45 bytes, which is the 16-byte header + 3 Blowfish blocks @ 8 bytes + 5 bytes of the next Blowfish divided across 2 chunks.

Chunk#2 is garbled because the chunk begins at an offset that's not aligned with the 8-byte Blowfish blocks.

Chunk#9 is the first chunk beyond Chunk#1 which does align with the 8-byte Blowfish blocks. While ECB Chunk#9 is not garbled except for the partial Blowfish block at the end, the CBC Chunk#9 is completely garbled. This possibly indicates that later CBC chunks are either ignoring the Salt/IV, or are trying to use the beginning IV for decoding later chunks.

For the non-encryption switches, the chunk#N parameter needs to be a length which encodes an input which is a multiple of 3 for Base64 and UUencode, and a multiple of 5 for Base32. And this is accomplished by the current chunk lengths of 60 and 72.

For the N chunk parameter to be supported by encryption, the chunk's input length needs to be a multiple of 8. It would also need to include a header which could be length zero, 16, or 32. It would need the Salted__ header if a random Salt is being used, and when using CBC encryption it would need to include the IV used to encrypt the Blowfish block at the beginning of that chunk, in a header as if that value is a RandomIV. When the Salt/IV is used to encrypt the data, unless that parameter would be ignored when using the Chunk parameter, there would need to be some way of making the header string dependent on using the correct Salt/IV parameter used.

The Nth chunk would also need to have an indicator that avoids confusion between legitimate padding vs having a portion of the input data at the end of a chunk which happens to match the type of padding. i.e. using the 'p' switch where the last portion of a block decodes to 1 or more spaces but is not near the end of file.

--

Question:

Some of this I was only able to make guesses because I can't find where the CBC encryption is compatible with OpenSSL. Can you give an example of output from $encode CBC encryption which can be decrypted with OpenSSL, or OpenSSL CBC encryption which can be decrypted with $decode?

Top
#264452 - 30/11/18 11:23 AM Re: $encode(data,<c|cr|e|er>,key,string) bug [Re: maroon]
Khaled Offline


Planetary brain

Registered: 04/12/02
Posts: 4295
Loc: London, UK
How are you testing to compare $encode() with OpenSSL?

Also, are you saying that there is an issue with $decode() ie. that it is not decoding to the original $encode()ed text correctly?

Top
#264455 - 30/11/18 06:37 PM Re: $encode(data,<c|cr|e|er>,key,string) bug [Re: Khaled]
maroon Offline
Hoopy frood

Registered: 12/01/04
Posts: 966
Update: Since $decode isn't supposed to recognize the chunks parameter anymore, the 1st issue for $decode() not recognizing chunks parameter is no longer an issue. And probably isn't worth the grief to have $encode support chunks in the context of encrypting strings either.

Originally Posted By: Khaled
Also, are you saying that there is an issue with $decode() ie. that it is not decoding to the original $encode()ed text correctly?


Yes $decode can decode the original text, but only if $encode did not use a chunk=N value. And in that situation, $decode ignores the chunk number, and decodes the entire string. The only changed behavior is when I change chunk to 0, where it returns 1 character that changes depending on the beginning of the original string.

Code:
//var %chunk 2 , %switch mc , %a $encode($regsubex($str(x,150),/x/g,$base(\n,10,10,3)), %switch,key) | echo -a %a -> $chr(22) $decode(%a,%switch,key,%chunk)



Originally Posted By: Khaled
How are you testing to compare $encode() with OpenSSL?


I'm trying to match the ECB encryption first, assuming there could be fewer things to go wrong, before moving on to matching CBC, but I can't get ECB to match either.
Code:
alias OpenSSL_test {
  var %opensslexe C:\path\openssl-1.1.0h-win32-mingw\openssl.exe
  var %testfile test.dat
  var %testout test.out
  var %password gryffczmj
  var %inputstring $str(3,16)
  echo -a ===== | if ($isfile(%testout)) remove %testout
  write -cn $qt(%testfile) %inputstring
  echo -a filesize %testfile should be $len(%inputstring) : $file(%testfile).size

  var %a enc -bf-ecb -in %testfile -out %testout -nosalt -p -pass pass: $+ %password
  echo -a commandline seen by OpenSSL: %a
  run cmd /k %opensslexe %a
  echo -a openssl window shows password is: $upper($left($sha256(%password),32))

  echo -a waiting 2 seconds for OpenSSL to diskwrite
  var %ticks $ticks + 2000 | while ($ticks < %ticks) noop

  echo -a bread %testout 0 $file(%testout).size &v1
  bread         %testout 0 $file(%testout).size &v1
  echo 3 -a size created by OpenSSL => $bvar(&v1,0)

  bset -tc &v2 1 %inputstring
  noop $encode(&v2,bme,%password)
  echo -a remove only the base64 from &v2 => $decode(&v2,bm)

  echo 3 -a $bvar(&v1,0) data created by OpenSSL: $bvar(&v1,1-)
  echo 4 -a $bvar(&v2,0) data created by $ $+ encode: $bvar(&v2,1-)
  return
}



Edited by maroon (30/11/18 10:25 PM)

Top
#264458 - 01/12/18 03:37 AM Re: $encode(data,<c|cr|e|er>,key,string) bug [Re: maroon]
maroon Offline
Hoopy frood

Registered: 12/01/04
Posts: 966
... and this alias tries to find compatibility between $encode's CBC vs OpenSSL.

I noticed from versions.txt that mIRC is using the 1.0.series and I had been using the 1.1.series. When I changed to 1_0_2q OpenSSL's ECB showed a different key, which turned out to be $md5(key) instead of first 16 bytes of $sha256(key), but that still encrypted the file differently than mIRC did.
Code:
alias OpenSSL_test-cbc {
  ;var %opensslexe C:\path\openssl-1.1.0h-win32-mingw\openssl.exe
  ;var %opensslexe C:\path\OpenSSL-Win32\bin\openssl.exe
  var %opensslexe C:\path\Win32OpenSSL-1_0_2q\bin\openssl.exe
  var %testfile test.dat
  var %testout test.out
  var %password gryffczmj
  var %inputstring $str(3,16)
  echo -a ===== | if ($isfile(%testout)) remove %testout
  write -cn $qt(%testfile) %inputstring
  echo -a filesize %testfile should be $len(%inputstring) : $file(%testfile).size

  var %a enc -bf-cbc -in %testfile -out %testout -salt   -p -pass pass: $+ %password

  echo -a commandline seen by OpenSSL: %a
  run cmd /k %opensslexe %a

  echo -a waiting 2 seconds for OpenSSL to diskwrite
  var %ticks $ticks + 2000 | while ($ticks < %ticks) noop 

  bread         %testout 0 $file(%testout).size &v1
  bread $qt(%testout) 0 $file(%testout).size &v2
  ; add base64 layer so $decode can remove it
  noop $encode(&v2,bm) | noop $decode(&v2,bmc,%password)
  echo 3 -a $bvar(&v1,0) data created by OpenSSL: $bvar(&v1,1-)
  echo 3 -a $bvar(&v1,0) data created by OpenSSL: $bvar(&v1,1-).text
  echo 4 -a $bvar(&v2,0) data created by $ $+ encode: $bvar(&v2,1-)
  echo 4 -a $bvar(&v2,0) data created by $ $+ encode: $bvar(&v2,1-).text
  return
}

Top
#264460 - 01/12/18 09:46 AM Re: $encode(data,<c|cr|e|er>,key,string) bug [Re: maroon]
Khaled Offline


Planetary brain

Registered: 04/12/02
Posts: 4295
Loc: London, UK
Quote:
Yes $decode can decode the original text, but only if $encode did not use a chunk=N value.

Thanks this has been fixed in the next version. Since $decode() no longer returns chunks, N = 0 should only return the number 1 indicating a single decoded line.

Top
#264461 - 01/12/18 10:59 AM Re: $encode(data,<c|cr|e|er>,key,string) bug [Re: maroon]
Khaled Offline


Planetary brain

Registered: 04/12/02
Posts: 4295
Loc: London, UK
Quote:
... and this alias tries to find compatibility between $encode's CBC vs OpenSSL.

I have been unable to find a combination of $encode()/$decode() that matches OpenSSL either.

As you noticed, older versions of openssl.exe use md5. At the time of implementation, I was using an even older version of openssl.exe. Unfortunately, I no longer have the test mIRC/perl/PHP/bat scripts and openssl.exe version that I was using in 2013 to confirm compatibility.

It does match Perl's blowfish CBC encrypt/decrypt though.

Unless there is a certain combination of switches and parameters that we're missing, I am going to assume that it does not match OpenSSL and will be removing this description from the help file.

I may look into adding a new switch to enable OpenSSL compatibility. I have added this to my to-do list. That said, I am not sure how useful this would be.

Top
#264470 - 03/12/18 10:03 PM Re: $encode(data,<c|cr|e|er>,key,string) bug [Re: Khaled]
maroon Offline
Hoopy frood

Registered: 12/01/04
Posts: 966
I found a 32-bit install of Perl, but it didn't have any of the CBC/Blowfish modules part of it, and I've not yet figured out how to add these modules. But, based on descriptions of how Perl did things, I've replicated behavior when using the 'i' and 's' switches. The methods used by Perl heavily relies on MD5.

In summary, when not using the literal 'l' switch, the 'c' switch requires you use 1 of 2 hashing methods for generating a 56-byte binary literal key used to encrypt the input string.

When using 'i' or 'r' (or 'ir' to put user-defined IV into the header), users should provide a key string which they have already salted, such as using the 'i' user-defined IV to HMAC the real key and using that hash as the key parameter. Otherwise, the literal 56-byte key is a simplistic expansion of $md5(key parameter) where the 1st 16 bytes are the literal MD5 hash of it, and the same encryption is generated from all key parameter strings having identical MD5 hashes.

When using the 's' switch, the Perl method repeats a MD5 hash 4 times to create the 64 bytes needed to create the literal 56-byte key and the 8-byte salt. The 1st 16 bytes of this is the MD5 hash of the (key $+ salt) parameters.

--

Below is just detail of how I confirmed that $encode's c/s/i/r switches are doing the same thing as Perl, and describing exactly what that is doing. Both examples use the same key parameter foobar, and use deadbeef as either the salt or IV.

The Perl method of turning -nosalt password into the encryption key is mentioned here:

http://www.drdobbs.com/web-development/encryption-using-cryptcbc/184416083

Based on that, I created an alias which produces a 56-byte binary string.

Code:
alias openssl_nosalt_keygen {
  bunset &key | var %pass foobar | noop $nosalt_digest_to_binary($md5(%pass) )
  while ($bvar(&key,0) < 56) { noop $nosalt_digest_to_binary($md5(&key,1)) }
  echo 4 -a $bvar(&key,0) literal key in hex: $regsubex($bvar(&key,1-),/(\d+)/g,$base(\1,10,16,2) )
}
alias nosalt_digest_to_binary {
  bset &key -1 $regsubex($1,/(..)/g,$base(\1,16,10) $chr(32)) | if ($bvar(&key,0) > 56) bcopy -c &key 56 &key 56 1
  echo 3 -a $bvar(&key,0) of 56 generated: $regsubex($bvar(&key,1-),/(\d+)/g, $base(\t,10,16,2) $chr(32))
}
alias use_foobar_iv {
  bset -t &v 1 BLOWFISH
  noop $encode(&v,bmcir,foobar,deadbeef) | noop $decode(&v,bm)
  echo 4 -a $regsubex($bvar(&v,1-),/(\d+)/g,$base(\1,10,16,2) $iif(!$calc(\n % 8),.)) $bvar(&v,1-).text
}


I was able to feed the 56 bytes output from /openssl_nosalt_keygen to an external program which accepted a 56-byte binary string as the encryption key, without UTF-8 encoding byte values 128-255 or excluding 0x00's. This alias produces matching encryption to that of $encode() where it used the 'cir' switches but didn't use the 'l' switch. This key is not affected by the content of the IV.

For example, "/use_foobar_iv" uses key=foobar and iv=deadbeef. Using the 'ir' switches together causes the user-defined IV to be listed in the header as if it's a randomly created IV, which is why this alias shows the header contains an IV of 'deadbeef'. The presence of the IV forces $encode to NOT use a salt and to use the above alias's method to generate the actual key. The encoded output of the "BLOWFISH" string immediately follows the 'deadbeef', and in hex format it's:

"39 17 4D 71 8B 9D 20 82".

"/openssl_nosalt_keygen" shows that using password "foobar" generates a 56 byte binary key string as hex:

38 58 F6 22 30 AC 3C 91 5F 30 0C 66 43 12 C6 3F C4 52 9D C8 56 43 BB 0C 5A 96 E4 65 87 37 77 77 B6 65 DE EB 42 29 7C FF FA B0 E1 4E 7E 78 0E 76 80 46 2D 01 3E 52 94 70

... where the first 16 bytes are the literal MD5 digest for 'foobar', and the remaining 40 of 56 bytes are entirely calculated using only these 16 bytes. i.e. there are only 2^128 possible 56-byte strings. The other program knowing nothing about 'foobar', using these 56 binary bytes as the literal key and using the IV of 'deadbeef', also encrypts the "BLOWFISH" text string to those same "39 17 4D 71 8B 9D 20 82" bytes.

Because /openssl_nosalt_keygen used the key parameter once, as the input to $md5, this limits the password strength to no more than 128 bits, and all password strings having the same MD5 hash generate equivalent encryption.

Code:
alias md5_collision {
  bset -t &v1 1 Tclo/w7jXCCVctR3e3IVh9Nvp7Ib3Fa3Sj3AeD57lRivv6IAqChL826OS1WzX0J1k9hJZ22g0VVdg2D7Xwf+og==
  noop $decode(&v1,bm) | bcopy -c &v2 1 &v1 1 -1
  bset &v2 36 $xor($bvar(&v2,36),2)
  bset &v2 56 $xor($bvar(&v2,56),128)
  echo 3 -a $md5(&v1,1) $bvar(&v1,36) $bvar(&v1,56)
  echo 4 -a $md5(&v2,1) $bvar(&v2,36) $bvar(&v2,56)
}

--

The switch 's' method for combining the key and salt parameters to generate the salted-key and IV binary strings is described at:

http://www.herongyang.com/Blowfish/Perl-Crypt-CBC-Secret-Key-and-IV-Algorithm.html

Based on this description, I created another alias to duplicate the way Perl generates the key and salt when using the 's' switch:

Code:
alias openssl_salted_keygen {
  bunset &raw | var %pass foobar , %salt deadbeef
  bset -tc &pass+salt 1 %pass $+ %salt
  while ($bvar(&key,0) < $calc(56+8)) { noop $salted_digest_to_binary }
  bcopy -c &pass 1 &key 1 56 | bcopy -c &iv  1 &key 57 8
  echo 4 -a literal key in hex: $regsubex($bvar(&key,1-56) ,/(\d+)/g,$base(\t,10,16,2) $chr(32))
  echo 4 -a literal  iv in hex: $regsubex($bvar(&key,57- ) ,/(\d+)/g,$base(\t,10,16,2) $chr(32))
}
alias salted_digest_to_binary {
  if ($bvar(&hash,0)) bcopy -c &raw 1 &hash 1 -1 | bcopy &raw -1 &pass+salt 1 -1
  bset -c &hash 1 $regsubex($md5(&raw,1),/(..)/g,$base(\1,16,10) $chr(32)) | bcopy &key -1 &hash 1 -1
  if ($bvar(&key,0) > $calc(56+8)) bcopy -c &key 64 &key 64 1
  echo 3 -a $bvar(&key,0) of 64 generated: $regsubex($bvar(&key,1-) ,/(\d+)/g,$base(\t,10,16,2) $chr(32))
}
alias use_foobar_salted {
  bset -t &v 1 BLOWFISH
  noop $encode(&v,bmcs,foobar,deadbeef) | echo noop $decode(&v,bm)
  echo 4 -a $regsubex($bvar(&v,1-),/(\d+)/g,$base(\1,10,16,2) $iif(!$calc(\n % 8),.)) $bvar(&v,1-).text
}


Using /use_foobar_salted shows that, when using password=foobar and a user-provided salt=deadbeef, the encryption of the text "BLOWFISH" follows the 'deadbeef' salt in the header, and in hex is "41 E1 92 D9 AA 51 90 E9".

Using password=foobar and salt=deadbeef as the inputs to /openssl_salted_keygen generates binary values for a 56-byte literal key and an IV, both dependent on the password and salt:

key: 38 58 F6 22 30 AC 3C 91 5F 30 0C 66 43 12 C6 3F 56 83 78 52 96 14 D2 2D DB 49 23 7D 2F 60 BF DF 0E BF 58 78 E8 2A F7 DA 61 8E D5 6F C6 7D 4A B7 AD FF 94 C3 8F B4 45 7A
iv: D4 9F 7A 4F AE 8E DA DE

Using these binary strings as literal input key and iv, the other program encrypted the "BLOWFISH" string to the same 8 bytes "41 E1 92 D9 AA 51 90 E9".

Top