Forums Bug Reports $hotp() base-32 decoding

In summary results of my 'research', the /help reference to "base32 format of 16/24/32 chars" and "lower case with spaces" both appear to be references to Google Authenticator format, but $hotp implements them both incorrectly. The length should be strings of 16/26/32 Base32 characters, either as a continuous string or as separated by spaces into groups of 4 for easier readability. $hotp/$totp are also measuring those lengths before removing spaces but instead should measure the length after removing spaces.

The specified lengths of hex strings appears to be designed to support applications that use binary hash digests of lengths 20/32/64 for sha1/sha256/sha512. Any such application would have no reason to strip 0x00's and UTF-8 encode the remaining bytes as if they're text.

All issues affect $hotp and $totp equally, as $totp uses a count parameter derived from $ctime instead of being a purely sequential value.

- -

The closest I can find to identifying $hotp identifier's key parameter's parsing is a quote I find on the Wikipedia page for "Google Authenticator":



The /help says "base32 format of 16/24/32 chars", but the 24 appears to be trying to support Google Auth but the wrong number is printed and also incorrectly implemented into actual string handling. At first I saw the 26 as if it were a typo, but the 3 bit-lengths referenced in the quote are 80 128 and 160 bits. When excluding the '=' padding, 16 26 and 32 are the Base32 lengths which encode binary key lengths of 80 128 160:



Instead, $hotp is only decoding base32 strings whose length are an exact multiple of 8 greater than length 8, without allowing them to have any '=' padding. This means $hotp is not supporting 128-bit keys encoded as 26 characters. It appears Google Auth doesn't pad their 26-character strings with ='s.

The closest I could find where Google Authenticator is associated with base32 encoding to every multiple of 8 is the source code at: https://github.com/google/google-authent...authenticator.c

... where the line:



... has the key length hard-coded as 80 bits, but the comment implies it can be edited to be any value that's a multiple of 8. But this is a multiple of 8 bits for the binary key, not the byte length of the base32 encoding.

Each group of 8 base32 characters can encode 40 bits, so having the base32 strings be multiples of 8 without padding assumes the binary key is always going to be a multiple of 40, which so far is true only for 80 and 160 bit keys.

The references to spaces and lower-case being part of Google Authenticator are at places like:

https://soeithelp.stanford.edu/hc/en-us/...d-or-iPod-Touch
https://garbagecollected.org/2014/09/14/how-google-authenticator-works/

The 1st link describes the key being a 26-character base32 string which can be typed as upper or lower case, and with-or-without spaces. The 2nd link shows a 160-bit key being presented as 8 groups of XXXX place-holders separated by spaces. I can't find a reference to how Google Auth handles the fact that 26 isn't a multiple of 4, but I'm guessing that there's either a couple groups of 5 or a final group of 2 characters. The keys are presented in small groups of digits to be user friendly.

I can't find reference to how rigid Google Auth is when someone enters their code using spaces, but I suspect that it allows as many spaces as the user wants, then simply removes the spaces to check if the remaining string is base32 of the appropriate length. But that can be too 'grabby' in this context where the identifier is trying to discern between literal plaintext and base32 encoded strings.

--

In addition to not supporting the 128-bit keys encoded as 26 base32 digits, $hotp is incorrectly supporting the 'lower-case-and-spaces' method, because it is measuring the 16/24/32 (instead of 16/26/32) length while they include spaces instead of verifying those lengths after the spaces are deleted. The base32 encoding of 12345678901 is the 18 character string GEZDGNBVGY3TQOJQGE. $hotp uses this string as a literal text key because the length isn't a multiple of 8. But when padded internally with 6 spaces to make the length be 24, $hotp then deletes the spaces, then it base32 decodes the remaining 18-char string into the underlying binary contents which in this example happen to also be bytes in the printable ASCII range. In this example, 3 different strings return the same password:



As far as $hotp's checking is concerned, it doesn't matter where the spaces are inserted or how many non-space characters are in the string, as long as the spaces+alphanumeric string is a total length of 16/24/32. In the above example, the 2nd command returns identical passwords because the insert of 8 additional spaces brought the spaced-padded length to 32, causing the space-padded string to again be handled as base32.

Even if $hotp is fixed to evaluate the correct lengths after the spaces are removed, I'm not sure it's desirable to support space padding of alphanumeric strings in non 'official' groups of characters, or even supporting mixed case. This example shows the password is the same in upper/lower/mixed case due to the actual password being the base32 decoding of the 21 non-spaces into the underlying non-utf8-encoded binary string inside:



Adding 8 additional non-consecutive spaces results in the key length increasing from 24 to 32, causing the same 21 non-space characters to be base32-decoded as the same key.

--

I haven't been able to track down any Google Auth references related to hex lengths of 40/64/128 chars needing to be decoded before being used as the key. Every reference of Google Auth keys being encoded has them being encoded as base32 not hex. But I can't imagine $hotp's current handling of hex strings matching any test vectors containing hex encoding of ASCII 00 or 128-255.

The 40/64/128 lengths seem obviously intended to be the hex-text display of 160/256/512 -bit key lengths where an application is wanting to decode the hex-text digests for sha1/sha256/sha512 into binary strings of length 20/32/64. This seems like the kind of thing OpenSSL would do, but I've been finding references to it encoding things as Mime or Base32 and not so much as Hex. It would not be desirable for an application to assume that the underlying contents needs to be UTF8-encoded after the 0x00's are stripped. A pair of hash digests which are identical except for location of 0x00 bytes would generate matching keys if all 0x00's were stripped.

Even when the decoded hex contents appears to already be UTF-8 encoded, the string is being encoded again, as shown by these matching passwords, where the hex encoded key already contains the UTF-8 encoding of $chr(10004):



--

This shows what I was trying to say in an earlier post, that Base32 and Hex16 encoded strings are not being handled the same way. The underlying decoded contents of Base32 strings are being used as their un-modified binary contents. Even though the above hex key is already a UTF8-encoded string, it is re-encoded again, so the "E2 9C 94" hex bytes are each encoded so the binary key for both usages becomes "C3 A2 C2 9C C2 94".

When $hotp recognizes a key as being a Base32 string, the key used is the binary contents that's 5/8ths as long. It does not have 0x00's stripped from the binary key nor does it have the remaining bytes UTF8-encoded. On the other hand, the underlying decoded contents of Hex16 strings are being UTF8-encoded after having 0x00's stripped.

In this example, the base32-decoded binary string is not altered, causing it to match the different literal text key and the hex-encoded key, where the hex digits are handled as if they're the encoding of non-UTF8-encoded text instead of encoding a binary hash digest. The latter 2 identical keys are obtained by UTF-8 encoding the 2nd and 3rd different strings into having the same password output for all 3 strings:



--

To fix these issues, it seems like the hierarchy of rules for handling the key parameter needs to change. Even though that will break backwards compatibility, it would restore support for 128-bit keys encoded by Google Authenticator into 26-character base-32 strings. It would also restore compatibility with applications that expect hex digests of length 40/64/128 to be binary keys of 20/32/64 bytes. It should also avoid false-matches of language passphrases containing spaces and no punctuation simply because their space-padded lengths happened to be 16/24/32.

1st rule:
Old: If key is a length 40/64/128 case-insensitive hex string, it is decoded to become a text string that is then UTF-8 encoded, and any 0x00's stripped. If the string is entirely 0x00's, the key is $null.
New: These hex strings should instead be decoded to binary keys of length 20/32/64 the same way base32 strings are being decoded to their binary contents.

2nd rule:
Old: If key length is 16 or greater and a multiple of 8 (except for lengths 40/64/128 containing only 0-9a-f), and if it's a valid case-insensitive Base32 encoded string without spaces or '=' padding, the key is the binary decoded contents whose length is 5/8ths the length of the Base32 string, with no 0x00's stripped and no UTF-8 encoding as if the contents are text.
New: no change

3rd rule:
Old: If the key length is 16/24/32 and contains only spaces or case-insensitive Base-32 characters, the spaces are stripped and the remaining Base-32 characters of arbitrary length are decoded into a binary key.
New: The target key lengths should instead be 16/26/32, and should be compared against the string length only after the spaces are removed. The spaces should be used only to group the characters into the same pattern presented by Google Authenticator, such as groups of 4 non-spaces, and valid strings should probably not include mixed-case letters.

4th rule:
Old: Any remaining strings not matching the 1st 3 patterns are considered literal text keys, and the input is assumed to already be UTF-8 encoded where necessary.
New: no change