I have the solution for you :
echo 'net.ipv6.ip_nonlocal_bind = 1' | tee /etc/sysctl.d/99-nonlocal_bind_ipv6.conf
You can do the same for IPv4
This allow daemons to bind to addresses before these addresses have been attributed to the kernel.
I’m using it on all my services because systemd start all of them to early, even if i ask to wait for network-online.
So, even for ssh, i do limit binding and it allow me to have 2 or 3 ssh running of the same machine but with different configurations (allow password or not, chroot or not…).