Getting NXDOMAIN in clearly existing sites with pihole and Unbound

Hello everyone.

I have setup a server with Pi-hole and Unbound and I am not using any external DNS resolvers like Cloudfare, etc.
It's working fine for quite some time now, but I've ran into problems with some specific websites.
I noticed this in Aliexpress and Docker Hub.

For example:
Digging hub.docker.com using Quad9 enabled in the DNS options section of pihole

$ dig @9.9.9.9 hub.docker.com

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> @9.9.9.9 hub.docker.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: **NOERROR**, id: 10227
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;hub.docker.com.                        IN      A

;; ANSWER SECTION:
hub.docker.com.         300     IN      CNAME   elb-default.us-east-1.aws.dckr.io.
elb-default.us-east-1.aws.dckr.io. 60 IN CNAME  prodextdefblue-1cc5ls33lft-b42d79a68e9f190c.elb.us-east-1.amazonaws.com.
prodextdefblue-1cc5ls33lft-b42d79a68e9f190c.elb.us-east-1.amazonaws.com. 60 IN A 18.210.197.188
prodextdefblue-1cc5ls33lft-b42d79a68e9f190c.elb.us-east-1.amazonaws.com. 60 IN A 18.206.20.10
prodextdefblue-1cc5ls33lft-b42d79a68e9f190c.elb.us-east-1.amazonaws.com. 60 IN A 3.228.146.75

;; Query time: 199 msec
;; SERVER: 9.9.9.9#53(9.9.9.9) (UDP)
;; WHEN: Mon Oct 17 16:15:45 UTC 2022
;; MSG SIZE  rcvd: 220

Now if I disable any external DNS and use Unbound upstream:

$ dig @127.0.0.1 hub.docker.com

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> @127.0.0.1 hub.docker.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: **NXDOMAIN**, id: 55260
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;hub.docker.com.                        IN      A

;; ANSWER SECTION:
hub.docker.com.         52      IN      CNAME   elb-default.us-east-1.aws.dckr.io.

;; AUTHORITY SECTION:
aws.dckr.io.            900     IN      SOA     ns-1870.awsdns-41.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

;; Query time: 167 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Mon Oct 17 16:19:53 UTC 2022
;; MSG SIZE  rcvd: 174

I get a NXDOMAIN error. I only noticed this error in aliexpress.com and hub.docker.com for now, any other website works fine.

Its not a blocking problem, it happens even with blocking completely disabled.

I am using:
Pi-hole v5.13 FTL v5.18.2 Web Interface v5.16

Thanks.

Who is listening on 127.0.0.1#53 ?

sudo ss -nltup sport = 53

And if its pihole-FTL thats listening, whats the configured upstream DNS server(s)?

dig +short @127.0.0.1 servers.bind chaos txt

Please upload a debug log and post just the token URL that is generated after the log is uploaded by running the following command from the Pi-hole host terminal:

pihole -d

or do it through the Web interface:

Tools > Generate Debug Log

Thank you for the reply.

@deHakkelaar
Yes, it is pihole-FTL listening, this is what I got from your dig command:

"127.0.0.1#5335 88 5"

@Bucking_Horn
That's the token URL

https://tricorder.pi-hole.net/ntA845zW/

Could you check ik below file exists and post its content here pls?

cat /etc/unbound/unbound.conf.d/resolvconf_resolvers.conf

If exists, try below:

The file does not exist.
Also, openresolv is not installed.

Could you post output for below pls?

sudo grep -v '^\s*#\|^\s*$' -R /etc/unbound/unbound.conf*

I cannot recreate your observation:
hub.docker.com resolves correctly with my Pi-hole/unbound configuration.

Your debug log shows you are using Pi-hole's default BLOCKINGMODE, which would return 0.0.0.0 for blocked domains.
This indeed confirms that it is not Pi-hole providing that NXDOMAIN answer by blocking hub.docker.com or any of its CNAMEs.

This would suggest that Pi-hole is using the reply as provided by unbound.
You could check that by running your dig directly through unbound:

dig -p 5335 @127.0.0.1 hub.docker.com
Some observations from your debug log, unrelated to your issue (click for details)
*** [ DIAGNOSING ]: Domainlist (0/1 = exact white-/blacklist, 2/3 = regex white-/blacklist)
 id     type  enabled  group_ids  domain           date_added           date_modified        comment
 -----  ----  -------  ---------  ---------------  -------------------  -------------------  ---------------------
 52     0           1  0          104.112.149.216  2022-10-14 00:11:10  2022-10-14 00:11:10  Whitelist(aliexpress)

Pi-hole is DNS filter - it is allowing or blocking domains, not IPs.
If you want to allow or block access to some or all of Aliexpress sites, you have to use the respective domain names.

You should remove that 104.112.149.216 entry via Pi-hole's Domain Management.

*** [ DIAGNOSING ]: contents of /etc

lrwxrwxrwx 1 root root 29 out  4 20:31 /etc/resolv.conf -> ../run/resolvconf/resolv.conf
   search 172.16.0.1 172.16.0.10

Rather than IP addresses,
By default, resolv.conf's search option would expect a single local search domain name, rather than IPs.

Your local search domain name should appear here (e.g. lan, home.arpa, fritz.box,...).

@deHakkelaar here it goes:
sudo grep -v '^\s*#|^\s*` -R /etc/unbound/unbound.conf*

/etc/unbound/unbound.conf:include-toplevel: "/etc/unbound/unbound.conf.d/*.conf"
/etc/unbound/unbound.conf.d/pi-hole.conf:server:
/etc/unbound/unbound.conf.d/pi-hole.conf:    verbosity: 0
/etc/unbound/unbound.conf.d/pi-hole.conf:    interface: 127.0.0.1
/etc/unbound/unbound.conf.d/pi-hole.conf:    port: 5335
/etc/unbound/unbound.conf.d/pi-hole.conf:    do-ip4: yes
/etc/unbound/unbound.conf.d/pi-hole.conf:    do-udp: yes
/etc/unbound/unbound.conf.d/pi-hole.conf:    do-tcp: yes
/etc/unbound/unbound.conf.d/pi-hole.conf:    do-ip6: no
/etc/unbound/unbound.conf.d/pi-hole.conf:    prefer-ip6: no
/etc/unbound/unbound.conf.d/pi-hole.conf:    harden-glue: yes
/etc/unbound/unbound.conf.d/pi-hole.conf:    harden-dnssec-stripped: yes
/etc/unbound/unbound.conf.d/pi-hole.conf:    use-caps-for-id: no
/etc/unbound/unbound.conf.d/pi-hole.conf:    edns-buffer-size: 1232
/etc/unbound/unbound.conf.d/pi-hole.conf:    prefetch: yes
/etc/unbound/unbound.conf.d/pi-hole.conf:    num-threads: 1
/etc/unbound/unbound.conf.d/pi-hole.conf:    so-rcvbuf: 1m
/etc/unbound/unbound.conf.d/pi-hole.conf:    private-address: 192.168.0.0/16
/etc/unbound/unbound.conf.d/pi-hole.conf:    private-address: 169.254.0.0/16
/etc/unbound/unbound.conf.d/pi-hole.conf:    private-address: 172.16.0.0/12
/etc/unbound/unbound.conf.d/pi-hole.conf:    private-address: 10.0.0.0/8
/etc/unbound/unbound.conf.d/pi-hole.conf:    private-address: fd00::/8
/etc/unbound/unbound.conf.d/pi-hole.conf:    private-address: fe80::/10
/etc/unbound/unbound.conf.d/root-auto-trust-anchor-file.conf:server:
/etc/unbound/unbound.conf.d/root-auto-trust-anchor-file.conf:    auto-trust-anchor-file: "/var/lib/unbound/root.key"

Thank you @Bucking_Horn for your extra tips.
Well, that's awkward, but hub.docker.com is now working. The only thing I changed was the multiple IPs in the resolv.conf file as you pointed out. Now it has just my pihole IP address.
But aliexpress still won't resolve so i used:

dig -p 5335 @127.0.0.1 www.aliexpress.com

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> -p 5335 @127.0.0.1 www.aliexpress.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 30555
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;www.aliexpress.com.            IN      A

;; ANSWER SECTION:
www.aliexpress.com.     100     IN      CNAME   global.aliexpress.com.
global.aliexpress.com.  49      IN      CNAME   global.aliexpress.com.gds.alibabadns.com.

;; AUTHORITY SECTION:
gds.alibabadns.com.     143     IN      SOA     gdsns1.alibabadns.com. none. 2018122017 1800 600 3600 360

;; Query time: 3 msec
;; SERVER: 127.0.0.1#5335(127.0.0.1) (UDP)
;; WHEN: Tue Oct 18 23:19:18 UTC 2022
;; MSG SIZE  rcvd: 166

OK, I can buy my chinese stuff somewhere else, but I couldn't live without docker hub :grin:

Update
This whatsapp media server also get a NXDOMAIN 'media.fldb5-1.fna.whatsapp.net'.

That dig result shows that your observation is related to unbound rather than Pi-hole.

We currently have another report that looks very similar to yours, see Unbound DNSSEC Failing to Docker.

That wouldn't affect your issue.
As mentioned, a resolv.conf search option expects a domain name (presumably as defined by your router) rather than an IP address. Your host machine will use that domain to append to DNS queries (e.g. a lookup for mylaptop would also request resolution of mylaptop.lan).

Looks good and is same as mine.

It might be interesting to take a domain that's giving this incorrect NXDOMAIN result and try it again after first clearing unbound's cache of all results relating to that domain. Eg for that whatsapp domain

Check if incorrectly showing NXDOMAIN

dig @127.0.0.1 -p 5335 media.fldb5-1.fna.whatsapp.net

Check the cache and see it listed

sudo unbound-control dump_cache | grep whatsapp.net

Flush the entire zone for it

sudo unbound-control flush_zone whatsapp.net

Check the cache again and confirm gone

sudo unbound-control dump_cache | grep whatsapp.net

Run dig again and see what result comes back now

dig @127.0.0.1 -p 5335 media.fldb5-1.fna.whatsapp.net

I flushed it as you suggested, still won't work.
I have enabled some logs, probably the problem is within unbound. I'll report back.

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.