Forums Bug Reports $hotp() base-32 decoding

For reference when you look at this to-do list item, below summarizes my research and gives a solution to fix the issues I mentioned in the past, without needing to create a new parameter to define the key format.

+ eliminates the vast majority of incorrect guesses between text and base32 by skipping rare magic base32 key lengths

+ supports the most common usage of Google Authenticator base32 format, which is a length 39 string having 32 text + 7 spaces

+ offers a way for users to have 100% of all key lengths supported if using hex format

The proposed modified design:


1. LEN is the length of the key-parameter string excluding all spaces
2. if LEN is any of 40 64 128 256 && key is valid hex and spaces, then Keyformat = hex
3. elseif LEN is any of 16 26 32 (possibly also 103 205) && key is valid base32 and spaces, then KeyFormat = base32
4. else KeyFormat = text
5. If KeyFormat = hex then KEY = decoded from hex to byte leng LEN/2 without UTF8 encoding to text
6. If KeyFormat = base32 then KEY = decoded from base32 to binary, as is currently done
7. If KeyFormat = text then KEY = unaltered key parameter as currently done
8. If byte_length(KEY) > block_length(hashname), then KEY = binary hashname_digest(KEY)


Key points...

• $hotp and $totp already correctly handle steps #6-8.
  .
• Avoiding UTF8 encoding of hex keys makes hex keys compatible with everyone else.
  .
  
  Originally Posted by Khaled

  I based the design of $hotp()/$totp() on many of the real-world C/C++ examples, discussions, and examples I found. So, for example, UTF-8 encoding all of the formats was something that was common to the implementations I saw, so that is what mIRC does. UTF-8 encoding obviously breaks keys that include null bytes. So the question is, are null bytes actually allowed?
  
  UTF8 encoding breaks all keys having bytes outside the 1-127 range, except for the bytes in the trailing position which just get put right back in by HMAC padding short keys with 0x00's. Yes null are allowed, and in keys randomly generated by Google Authenticator and other programs trying to be compatible with it, null bytes are no more/less common than any other byte, and by random chance approximately 1 per 14 160-bit Google authenticator keys has at least 1 null byte.
  
  I have never found any implementation out there who takes a hex/base32 encoded binary key who then modifies it by UTF8 encoding it or altering it in any way.
  
  The ambiguity is caused by the unfortunate decision by the RFP author to use 20 text numerics as the 1 and only test vector, which made it less obvious that the key is a binary string where there are 2^N possible N-bit strings.
  
  You can google for 'totp generator online' and can find several sites that will return a 6 digit code for whatever base32 key you feed it, including xanxys.net/totp totp.danhersam.com or totp.app
  
  At xanxys they randomly generate keys, and you'll find one having a null byte after refreshing a few times. They present the binary byte string encoded as both hex and as the equivalent base32 encoding.
  
  If you input that base32 string into the danhersam template it returns the same digits whether the 'key' is with/without optional spaces, and if $totp is fed the same base32 string with the spaces removed, it also matches the danhersam results assuming clocks are synched. The base32 result is currently generated correctly by handling all the bytes including the null bytes as a binary string without modifying them into UTF8 text, so the hex equivalent key should be handled compatibly with the way base32 keys are.
  
  I cannot find any such discussions, code, or examples which involves hex or base32 keys containing bytes outside the 1-127 range then modifying them into UTF8 text. The closest I could find was discussions for programs like OpenSSL in discussions unrelated to TOTP, which were about the correct way to handle normal text when used as passphrases or how to interpret name or other text identifier fields. And there the alternative to UTF8 was the old way of everyone interpreting the text in their own local codepage.
  
  Here's something that's not just an online generator, but it's a real world implementation of TOTP which does not transform the key into UTF8 text nore strip 0x00 bytes. You can make an account at cservice.undernet.org which registers you with 'X', their roughly equivalent of nickserv/chanserv.
  
  In your account area in their website, you can enable an option where you'd need to use TOTP in order to login at their website and into your IRC account. Instead of scanning the QR image into your phone, you can click on 'enter your secret key manually' which gives a 160-bit Google Authenticator key that's been encoded as base32 then padded with 7 spaces to a length 39 string. If the 1st base32 key they give you doesn't contain an embedded 0x00 byte, you can halt the registration process and retry asking to setup TOTP a few more times until you're given such a key. With the following command in the editbox ready to press <enter>, it makes it easy to know when the clipboard contains the key containing the 0x00 byte. The spaces don't need to be stripped, because $decode base32 conveniently ignores spaces.
  
  //var %a $cb | bset -t &v 1 %a | echo -a $bvar(&v,0) : $decode(&v,ba) : $regsubex($bvar(&v,1-),/\b(0)\b/g,$chr(22) \t $chr(22))
  
  When you are given a key containing the null byte, you can take the length 39 string they give you, remove the spaces to make the length be 32, and feeding that to $totp will give the correct 6 digits if your clock is in synch.
  
  They won't enable TOTP mode until you can prove that you're able to answer with the correct 6-digit codes. And the only way to make Undernet happy is for the 6 digit code to be created the same way base32 is already handled, where the bytes outside the 1-127 range are not changed or stripped..
  
  And the hex keys should be handled in an equivalent manner to the base32 keys.
  .
• By adding 256 as a magic hex length, the 128 and 256 hex-digit lengths can provide support for 100% of keys and keylengths for all hashnames including the longer block lengths belonging to SHA384/512.
  
  Because HMAC pads 0x00 bytes to keys shorter than the 64/128 block length used by the underlying hash, all hex lengths shorter than 128 hex digits which are being handled correctly at the 40 or 64 lengths, would also be handled identically when '0' digits are appended to make them be 128 digit hex. And for sha384/sha512, the same would be true for all hex keys for all hex lengths shorter than 256.
  
  Almost all hex keys created randomly or from a hash digest are going to have at least 1 byte outside the 1-127 range, so it's extremely unlikely that any of those random keys were not broken, and would have been compatible only between clients interpreting them the same way.
  .
• By excluding spaces when checking for the 'magic lengths', this cuts down on the key lengths where text strings can be incorrectly guessed as base32. There's 3 Google Authenticator lengths, 16,26,32. And when they have the normal amount of optional spaces padding, that would add the lengths 19 and 37. Instead of having extra magic lengths that could provide additional collisions with text strings, excluding spaces from that count would only need to support the 16/26/32, and it wouldn't matter whether they used some/all/none/extra optional spaces.
  
  From what I can tell, the length 39 text string used by Undernet to present the base32 key is by far the most common way of presenting Google Authenticator keys.
  .
• This elimination of all the undocumented multiple-of-8 base32 lengths greatly reduces the case-sensitive text+spaces passphrases that could be seen as if case-insensitive base32.
  
  [+/quote=Khaled]
  That is correct. The comments in my code state that it should check for >=16 and multiples of 8 instead of just 16/24/32. I cannot remember why as I implemented this feature three years ago and researched it at that time. You will need to research this, and the Google Authenticator format, yourself I'm afraid. [/quote]
  
  The person who you found stating it needs to be lengths 16/24/32 plus other multiple of 8 bytes was not correct on either. Yes, the lengths are >= 16 because that's the shortest base32 length used by Google Authenticator. But the G.A. lengths are 16/26/32 not 16/24/32, Because the 1st trio are the 3 lengths for the base32 encodings of bit lengths 80,128,160 which have been common key lengths in the past and present. On the other hand, 24 would be the encoding of 24*5=120 bits. While that's a valid key length as all byte lengths are, it wouldn't be something chosen by Google.
  
  Also, they should have been talking about multiple of 8 Bits not Bytes, and would have not even mentioned this topic except for the fact that 128 bit length encodes as a length 26 base32 string which has the capacity to hold 26*5=130 bits. But what they were tryig to say is nothing special, as that's simply the normal behavior that $decode already has when handling mime and base32 string, where the decoded string contains only bytes where all 8 bits had been encoded into the mime/base32/uuencode string, and does not extract any bytes where only some of their bytes were encoded.
  
  So, what they should have been saying was that the Google Authenticator handling of length 26 base32 strings should be the same way that $decode and all other decoders handle it, which is to see it as the longest byte string it can hold completely, 16*8=128 bits, not the longest bit string it can hold of 26*5=130 bits.
  .
• Most of the undocumented multiple-of-8-byte base32 lengths would be rarely if ever used. Those 8x lengths would be the encoding of bit-lengths that are multiples of 40 bits, of which the 40x lengths most likely to be encoded as base32 are the 16-bytes=80-bits and 32-bytes=160-bits already used by Google Authenticator.
  
  Base32 is used little except at the G.A. lengths. The main reason they use base32 is because they wanted to help people hand-typing the key from 1 device to another, which would be less likely to be done interactively unless they know copypaste is available. They chose base32 because it's shorter than hex, and didn't use mime because base32 is case-insensitive. If not for that, they would have used hex which is built into compiler languages, and avoids the need to create tricky base32 decoders.
  
  If you wish to offer 100% cover for all possible keys encoded in the base32 format, you can still trim the magic base32 lengths down to just the 3 G.A. lengths 16 26 32, and from there you can optionally accept 103 and 205 as base32 lengths, which decode to the same binary lengths as above hex lengths 128 and 256. Users would then be able to take any base32 string that's already a multiple of 8 'characters' and that's shorter than 103, and all they'd need to do is just append enough "A" digits to reach 103, and for sha384/sha512 they can also do the same "A" exending of strings to length 205. Something like:
  
  //var %base32key DEADBEEF | if ((8 // $len(%base32key)) && ($len(%base32key) < 103)) var -s %base32key %base32key $+ $str(A,$calc($v2 -$v1))
  
  The only application I've seen using base32 instead of hex for keys longer than the Google-Authenticator strings having 32 digits - is at verifyr.com/en/otp/check where they have a template that generates random TOTP keys then lets you test them against the current time. They generate random keys that occasionally contain null bytes, and their base32 string has the 103 length above, because they decided to have the key length be the same binary length as hex 128 strings have.
  
  The only drawback I see from adding 103 and 205 as magic base32 lengths is that these might create collisions with word+spaces passphrases, but as long as /help warns of these lengths, they can either avoid them or make sure their passphrases use at least 1 character that's not base32.

You had previously posted...


I have looked at current/prior OpenSSL version source code, and can find no evidence that HOTP or TOTP have ever been part of OpenSSL. People may be taking advantage of the HMAC built into OpenSSL to avoid writing their own HMAC function, but any OpenSSL involvement is being just a subroutine that takes whatever binary strings are passed to it then spitting back a hash digest, and the applications do their own truncating to create the digits.


Summary of a few other items I'd mentioned previously along with these issues

• Digits being limited to 9
  
  I'm guessing the max of 9 was from some application deciding that 9 was enough. The RFC doesn't say 10 digits is invalid, it just warns that going from 9 -> 10 digits just doubles the strength instead of getting the 10x strength from adding the other digits.
  .
• Timestep limited to 3600
  
  If the decision is to continue limiting to 3600, that's fine, but would be great if it were documented in /help and treated as if an invalid parameter. While for most cases the timestep window is kept at the default 30, there can be situations where the interval would be longer.
  
  RFC 6287 is OCRA authentication, and that uses HOTP/TOTP in a way which expects it to support as many as 10 digits and to support timestep as large as 48 hours They even have a digits=0 mode that returns the entire hex hash from HMAC instead of truncating it to a few numbers. But if $hmac is able to accept binary keys directly, it would be simple to get that full digest directly, as the complicated part is making the 6 digits.
  
  If users know about the 3600 limit, they can do an easy workaround by setting timestep=1 then do something similar to this one that changes the code at 9am each day.
  
  $totp(key,$calc( ($ctime -$timezone -3600*9)//86400),sha1,6,1)