Pi-hole returning a different/wrong ip for registry-1.docker.io

[supermamon]

tl;dr
Overview:

• Pi-hole configured as DNS server on each client, not on router
• Pi-hole not configured as DHCP server

I'm expecting Pi-hole to give me

pi@dazzler:~/apps/plex $ nslookup registry-1.docker.io 1.1.1.1
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
Name:	registry-1.docker.io
Address: 34.238.187.50
Name:	registry-1.docker.io
Address: 52.1.121.53
Name:	registry-1.docker.io
Address: 54.236.131.166
Name:	registry-1.docker.io
Address: 52.72.232.213
Name:	registry-1.docker.io
Address: 54.236.165.68
Name:	registry-1.docker.io
Address: 52.54.232.21
Name:	registry-1.docker.io
Address: 35.174.73.84
Name:	registry-1.docker.io
Address: 3.218.162.19

but otherwise, gives me

pi@dazzler:~/apps/plex $ nslookup registry-1.docker.io
Server:		192.168.18.2
Address:	192.168.18.2#53

Name:	registry-1.docker.io
Address: 52.4.20.24

what could be wrong?

Long version
My problem started with me trying to pull the docker image for plex one of my Pi's.

pi@dazzler:~/apps/plex $ docker-compose pull
Pulling plex ... error

ERROR: for plex  b'Get https://registry-1.docker.io/v2/: x509: certificate is valid for *.env.meliopayments.com, not registry-1.docker.io'
ERROR: Get https://registry-1.docker.io/v2/: x509: certificate is valid for *.env.meliopayments.com, not registry-1.docker.io

Uhh, was docker hijacked?
So tried curl https://registry-1.docker.io and yes, invalid cert. Then I tried on another Pi. Same thing. Tried it on my mobile (wifi off), and it works properly. So the problem is in my network.

Next I tried nslookup:

pi@dazzler:~/apps/plex $ nslookup registry-1.docker.io
Server:		192.168.18.2
Address:	192.168.18.2#53

Name:	registry-1.docker.io
Address: 52.4.20.24

pi@dazzler:~/apps/plex $ nslookup registry-1.docker.io 1.1.1.1
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
Name:	registry-1.docker.io
Address: 34.238.187.50
Name:	registry-1.docker.io
Address: 52.1.121.53
Name:	registry-1.docker.io
Address: 54.236.131.166
Name:	registry-1.docker.io
Address: 52.72.232.213
Name:	registry-1.docker.io
Address: 54.236.165.68
Name:	registry-1.docker.io
Address: 52.54.232.21
Name:	registry-1.docker.io
Address: 35.174.73.84
Name:	registry-1.docker.io
Address: 3.218.162.19

So it seems that Pi-hole is providing a different ip.
Current setup:

• Pi-hole configured as DNS server on each client, not on router
• Pi-hole not configured as DHCP server
• Cloudflare as upstream DNS
• Only local machines configured on the "Local DNS" section on the admin console.

Any ideas on what to check?

[DanSchaper]

What are you seeing that indicates the IP returned is wrong?

Edit:

Please provide the debug token as requested.

dig registry-1.docker.io @192.168.88.5

; <<>> DiG 9.11.5-P4-5.1+deb10u2-Raspbian <<>> registry-1.docker.io @192.168.88.5
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1315
;; flags: qr rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;registry-1.docker.io.          IN      A

;; ANSWER SECTION:
registry-1.docker.io.   40      IN      A       52.5.11.128
registry-1.docker.io.   40      IN      A       54.236.165.68
registry-1.docker.io.   40      IN      A       52.1.121.53
registry-1.docker.io.   40      IN      A       18.213.137.78
registry-1.docker.io.   40      IN      A       23.22.155.84
registry-1.docker.io.   40      IN      A       3.218.162.19
registry-1.docker.io.   40      IN      A       52.55.168.20
registry-1.docker.io.   40      IN      A       52.54.232.21

;; Query time: 5 msec
;; SERVER: 192.168.88.5#53(192.168.88.5)
;; WHEN: Sun Jan 24 08:45:29 PST 2021
;; MSG SIZE  rcvd: 177

[supermamon]

debug token: ofl94worsm

I'm trying to find out where 52.4.20.24 is coming from

❯ dig registry-1.docker.io @192.168.18.2

; <<>> DiG 9.10.6 <<>> registry-1.docker.io @192.168.18.2
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1212
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;registry-1.docker.io.          IN      A

;; ANSWER SECTION:
registry-1.docker.io.   2       IN      A       52.4.20.24

;; Query time: 13 msec
;; SERVER: 192.168.18.2#53(192.168.18.2)
;; WHEN: Mon Jan 25 01:34:05 +08 2021
;; MSG SIZE  rcvd: 65

[DanSchaper]

First, disable IPv6 on the Pi-hole to get rid of a hard to diagnose variable. I say this because you are forwarding to an IPv6 address that is not in your upstream configuration:

   server=1.1.1.1
   server=1.0.0.1


   Jan 25 00:00:03 dnsmasq[2238]: forwarded configuration.apple.com to 2606:4700:4700::1111
   Jan 25 00:00:03 dnsmasq[2238]: query[AAAA] configuration.apple.com from 192.168.18.5
   Jan 25 00:00:03 dnsmasq[2238]: forwarded configuration.apple.com to 2606:4700:4700::1111
   Jan 25 00:00:03 dnsmasq[2238]: query[A] configuration.apple.com from 192.168.18.5
   Jan 25 00:00:03 dnsmasq[2238]: forwarded configuration.apple.com to 2606:4700:4700::1111

*** [ DIAGNOSING ]: Name resolution (IPv6) using a random blocked domain and a known ad-serving domain
[✓] millennialmedia.com is :: via localhost (::1)
[✓] millennialmedia.com is :: via Pi-hole (<<PUBLIC IPV6 ADDRESS REDACTED)
[✗] Failed to resolve doubleclick.com via a remote, public DNS server (2001:4860:4860::8888)

Second, when you re-enable IPv6, don't use GUA (public) IP addresses.

[DanSchaper]

Also, try running

grep registry-1.docker.io /var/log/pihole.log

[supermamon]

grep registry-1.docker.io /var/log/pihole.log

Jan 25 00:06:56 dnsmasq[5772]: query[A] registry-1.docker.io from 192.168.18.9
Jan 25 00:06:56 dnsmasq[5772]: /etc/hosts registry-1.docker.io is 52.4.20.24

Frak! Of course, first obvious thing I did not check. I removed the ip from the hosts file and this looks solved now.
Just wondering now how it got there.