Both my piholes stopped working, can't figure out why

The issue I am facing:

pihole is not resolving dns anymore
I have two pis that did a great job in the past and now both failed simultaneously.
I am running unbound on both. pi1 is running vaultwarden in a docker container, therefore the webserver's port has been moved to 8000. I can reach the webservers without problems (on both pis)

Now, when I run a dig whatever.tld @pihole1 or @pihole2 I do get a SERVFAIL

Any hint or help would be very much appreciated.

Debug tokens are
pihole1: https://tricorder.pi-hole.net/5NLajf0e/
pihole2: https://tricorder.pi-hole.net/F0W3SHvb/

Both debug logs show your router is using itself as a DNS server.

You need to set Pi-hole IP as the DNS on your router DHCP settings.
You can add both pi-holes (10.7.8.8 and 10.7.8.9) as DNS servers on the router.

Since both piholes were not resolving, I had to switch my network back and make the router distribute the providers DNS. This is just a temp workaround for the time I’m trying to fix local DNS resolver.

From my Linux machine …

dig pihole.net @10.7.8.8
dig bbc.com @10.7.8.9

return status SERVFAIL.

Consistent SERVFAILs for arbitrary domains usually hint at an upstream issue.
From your debug log, you are using unbound as upstream.

As that is using DNSSEC, and DNSSEC in turn requires acurate time, you should verify that your Pi-hole host's local time and time zone information is correct.

In addition, if you'd upgraded to an OS of the Debian Bullseye variety recently, you may want to check for potential DNS loops caused by an unlucky combination of unbound's and openresolv's package defaults.

Check your unbound configuration for potentially unwanted references to resolvconf_resolvers.conf

sudo grep -v '#\|^$' -R /etc/unbound/unbound.conf*
1 Like

Thanks for your reply. I'll check on unbound and time settings when I get back home.

Yes, I am using unbound as local resolver. This used to work just fine but stopped working all of a sudden (not triggered by package updates or dist-upgrade). The only thing I can image is that a sudden power outage corrupted files on the SD card. On the other hand, it would be quite random that a power cut corrupts unbound on two devices and everything else continues to work. Let me check the items you mentioned and I'll get back.

/etc/unbound/unbound.conf:include-toplevel: "/etc/unbound/unbound.conf.d/*.conf"
/etc/unbound/unbound.conf.d/resolvconf_resolvers.conf:forward-zone:
/etc/unbound/unbound.conf.d/resolvconf_resolvers.conf:  name: "fritz.box"
/etc/unbound/unbound.conf.d/resolvconf_resolvers.conf:  forward-addr: 10.7.8.1
/etc/unbound/unbound.conf.d/resolvconf_resolvers.conf:forward-zone:
/etc/unbound/unbound.conf.d/resolvconf_resolvers.conf:  name: "."
/etc/unbound/unbound.conf.d/resolvconf_resolvers.conf:  forward-addr: 10.7.8.1
/etc/unbound/unbound.conf.d/root-auto-trust-anchor-file.conf:server:
/etc/unbound/unbound.conf.d/root-auto-trust-anchor-file.conf:    auto-trust-anchor-file: "/var/lib/unbound/root.key"
/etc/unbound/unbound.conf.d/pi-hole.conf:server:
/etc/unbound/unbound.conf.d/pi-hole.conf:    logfile: "/var/log/unbound/unbound.log"
/etc/unbound/unbound.conf.d/pi-hole.conf:    verbosity: 1
/etc/unbound/unbound.conf.d/pi-hole.conf:    interface: 127.0.0.1
/etc/unbound/unbound.conf.d/pi-hole.conf:    port: 5335
/etc/unbound/unbound.conf.d/pi-hole.conf:    do-ip4: yes
/etc/unbound/unbound.conf.d/pi-hole.conf:    do-udp: yes
/etc/unbound/unbound.conf.d/pi-hole.conf:    do-tcp: yes
/etc/unbound/unbound.conf.d/pi-hole.conf:    do-ip6: no
/etc/unbound/unbound.conf.d/pi-hole.conf:    prefer-ip6: no
/etc/unbound/unbound.conf.d/pi-hole.conf:    harden-glue: yes
/etc/unbound/unbound.conf.d/pi-hole.conf:    harden-dnssec-stripped: yes
/etc/unbound/unbound.conf.d/pi-hole.conf:    use-caps-for-id: no
/etc/unbound/unbound.conf.d/pi-hole.conf:    edns-buffer-size: 1232
/etc/unbound/unbound.conf.d/pi-hole.conf:    prefetch: yes
/etc/unbound/unbound.conf.d/pi-hole.conf:    num-threads: 1
/etc/unbound/unbound.conf.d/pi-hole.conf:    so-rcvbuf: 1m
/etc/unbound/unbound.conf.d/pi-hole.conf:    private-address: 192.168.0.0/16
/etc/unbound/unbound.conf.d/pi-hole.conf:    private-address: 169.254.0.0/16
/etc/unbound/unbound.conf.d/pi-hole.conf:    private-address: 172.16.0.0/12
/etc/unbound/unbound.conf.d/pi-hole.conf:    private-address: 10.0.0.0/8
/etc/unbound/unbound.conf.d/pi-hole.conf:    private-address: fd00::/8
/etc/unbound/unbound.conf.d/pi-hole.conf:    private-address: fe80::/10

Output of cat /etc/unbound/unbound.conf.d/resolvconf_resolvers.conf

# Generated by resolvconf

forward-zone:
        name: "fritz.box"
        forward-addr: 10.7.8.1

forward-zone:
        name: "."
        forward-addr: 10.7.8.1

Here is the output of

cat /var/log/unbound/unbound.log
[1676315233] unbound[368:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
[1676315233] unbound[368:0] info: generate keytag query _ta-4f66. NULL IN
[1676315233] unbound[368:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
[1676315339] unbound[368:0] info: generate keytag query _ta-4f66. NULL IN
[1676315339] unbound[368:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
[1676315339] unbound[368:0] info: generate keytag query _ta-4f66. NULL IN
[1676315339] unbound[368:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
[1676315339] unbound[368:0] info: generate keytag query _ta-4f66. NULL IN
[1676315339] unbound[368:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
[1676315339] unbound[368:0] info: generate keytag query _ta-4f66. NULL IN
[1676315339] unbound[368:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
[1676315339] unbound[368:0] info: generate keytag query _ta-4f66. NULL IN
[1676315339] unbound[368:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
[1676315339] unbound[368:0] info: generate keytag query _ta-4f66. NULL IN
[1676315339] unbound[368:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
[1676323217] unbound[368:0] info: generate keytag query _ta-4f66. NULL IN
[1676328661] unbound[368:0] info: service stopped (unbound 1.13.1).
[1676328662] unbound[368:0] info: server stats for thread 0: 5 queries, 1 answers from cache, 4 recursions, 0 prefetch, 0 rejected by ip ratelimiting
[1676328662] unbound[368:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0
[1676328662] unbound[368:0] info: average recursion processing time 0.056727 sec
[1676328662] unbound[368:0] info: histogram of recursion processing times
[1676328662] unbound[368:0] info: [25%]=0.032768 median[50%]=0.065536 [75%]=0.098304
[1676328662] unbound[368:0] info: lower(secs) upper(secs) recursions
[1676328662] unbound[368:0] info:    0.016384    0.032768 1
[1676328662] unbound[368:0] info:    0.032768    0.065536 1
[1676328662] unbound[368:0] info:    0.065536    0.131072 2
[1676328695] unbound[372:0] notice: init module 0: subnet
[1676328695] unbound[372:0] notice: init module 1: validator
[1676328695] unbound[372:0] notice: init module 2: iterator
[1676328696] unbound[372:0] info: start of service (unbound 1.13.1).
[1676405719] unbound[372:0] info: generate keytag query _ta-4f66. NULL IN
[1676405719] unbound[372:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
[1676405719] unbound[372:0] info: generate keytag query _ta-4f66. NULL IN
[1676405719] unbound[372:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
[1676405719] unbound[372:0] info: generate keytag query _ta-4f66. NULL IN
[1676405719] unbound[372:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
[1676405719] unbound[372:0] info: generate keytag query _ta-4f66. NULL IN
[1676405719] unbound[372:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
[1676405719] unbound[372:0] info: generate keytag query _ta-4f66. NULL IN
[1676405719] unbound[372:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
[1676405719] unbound[372:0] info: generate keytag query _ta-4f66. NULL IN
[1676405719] unbound[372:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN

Please try the following:

  1. Edit file /etc/resolvconf.conf and comment out the last line which should then read:

#unbound_conf=/etc/unbound/unbound.conf.d/resolvconf_resolvers.conf

  1. Delete the unwanted unbound configuration file:

sudo rm /etc/unbound/unbound.conf.d/resolvconf_resolvers.conf

  1. Restart unbound:

sudo service unbound restart

1 Like

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35311

I am wondering what introduced this mal config ... package update?

Most likely- we've been seeing related reports since Bullseye was released.
If you're interested in details, take a look at WARNING: Raspbian October 2021 release bullseye + unbound.

1 Like

thank you so much for your help.

For the second pihole I am running, I removed resolvconf_resolvers.conf and all is working fine again!

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.