There's really no problem with it.
Just use the existing ipv4 support for connecting to ipv4 addresses, and ipv6 support for connecting to ipv6 adresses. They really can't be mistaken or each other.