IPv6 can be tricky indeed, as it heavily favours auto configuration.
Very likely, your router will continue to advertise your ISP's IPv6 DNS server addresses on your network. Any client joining your network will see those advertisements in addition to those of Pi-hole.
It would then depend entirely upon a client's implementation which DNS server it would prefer, and that mechanism is a black box from the user perspective.
Add to this that many clients, especially Smartphones, will prefer IPv6 over IPv4, and that'd explain why you may see Pi-hole being by-passed regularly.
Now, even if we can't tell exactly how a specific device will pick a DNS server (this is also largely true for IPv4, btw), we can safely assume that any device will ignore a DNS server address that's not available.
And this would be the case if your Pi-hole's IPv6 address would be changing.
With IPv6, this could happen if your ISP would change your network's IPv6 prefix, and some ISP do this on a regular basis.
If that's the case for you, you should avoid using a public IPv6 address (from 2000::/3 range). Try to Use IPv6 ULA addresses for Pi-hole instead.
Another way to tackle this would be to change your device's preferences to prefer IPv4 over IPv6. But I am unaware of any Smartphone OS that would allow you to do so, so this is likely not going to help you.
And finally, you could try to disable IPv6 altogether on your network (unless you are dependent on IPv6 for any reason).
If your devices would still list a link-local adress (in fe80::/10 range), that should be ok. As long as none of your devices has a public IPv6 address (aka GUA, 2000::/3 range), your IPv6 DNS traffic will stay in your network or even cease completely.