If you only shadow the public A record, then clients would be expected to receive the public DNS records as publically available, including AAAA ones.
Your my-domain.com is a public domain, and quite obviously, its authoritative DNS server has AAAA records for it.
If you cannot or do not want to provide local IPv6 addresses, you may want to regex block AAAA replies for that domain, see Pi-hole extensions - Pi-hole documentation.