This is based on personal observation not on any "official" rfc or other docs.
Key seems to be limited to 23 characters. Some IRCd's truncate longer keys. IRCu responds with numeric 525 (Key is not well formed)
Key cannot start with colon ":" It will be stripped from the actual key that is set.
Key cannot start with space. It will be stripped from the actual key that is set.
Key cannot contain space. Everything after the space is ignored.
Colon being prefix character and space being separator character per rfc1459
Everything else looks to be fair game. However different IRCd's seem to just make up their own rules.
Bahamut seems to allow entire ASCII range even non printable range except few obvious ones like char10 (Line Feed) char13 (Carriage return) and maybe few more.
IRCu throws numeric 525 when you try to use ASCII non printable range.
Solanum seems to silently strip everything but prinatble range and converting unicode.
Unicode characters also take two character spaces. Therefore key that is only unicode range can be only 11 characters long.
Pretty sure there are more quirks down the road but those are things I have discovered while writing my own scripts that deal with channel keys.