Pi-Hole "DNS service not running" eventually

eddieparker · January 11, 2020, 5:50am

Please follow the below template, it will help us to help you!

Expected Behaviour:

The DNS service should stay up.

Actual Behaviour:

I run pi-hole and all is well for a while until it eventually stops serving requests. The status page ends up saying:

DNS service not running FTL offline
Load: 0.11 0.04 0.01
Memory usage: 9.5 %

Debug Token:

I tried to run pihole -d and run the web interface version, but it yields:

[✗] There was an error uploading your debug log.

Please try again or contact the Pi-hole team for assistance.
A local copy of the debug log can be found at: /var/log/pihole_debug.log

Then if I restart DNS I can upload this:

https://tricorder.pi-hole.net/lcelzt6dd1

technicalpyro · January 11, 2020, 7:21am

run pihole -d while it is down.
do not upload the log yet
once resolution is avail again use the following command to upload the debug log from the failed state
cat /var/log/pihole_debug.log | pihole tricorder

jfb · January 11, 2020, 12:23pm

The log outputs in your debug log show the problem. You have a huge number of DNS requests, and they eventually overwhelm the capabilities of your Pi (appears to be a Zero?).

   [2020-01-11 05:38:16.147 19788] Resizing "/FTL-queries" from 140967936 to 141164544
   [2020-01-11 05:38:16.194 19788] Imported 2938661 queries from the long-term database
   [2020-01-11 05:38:16.198 19788]  -> Total DNS queries: 2938661
   [2020-01-11 05:38:16.199 19788]  -> Cached DNS queries: 10759
   [2020-01-11 05:38:16.199 19788]  -> Forwarded DNS queries: 2926812
   [2020-01-11 05:38:16.200 19788]  -> Exactly blocked DNS queries: 1090
   [2020-01-11 05:38:16.200 19788]  -> Unknown DNS queries: 0
   [2020-01-11 05:38:16.200 19788]  -> Unique domains: 498
   [2020-01-11 05:38:16.200 19788]  -> Unique clients: 25
   [2020-01-11 05:38:16.201 19788]  -> Known forward destinations: 3

You have almost 3 million DNS requests in 24 hours, which is likely circular traffic due to conditional forwarding. Please post the outputs of the following commands from the Pi terminal:

echo ">top-clients" | nc localhost 4711

echo ">top-domains" | nc localhost 4711

tail -n25 /var/log/pihole.log

eddieparker · January 14, 2020, 1:35am

Thanks for the response. This might be due to what I'm trying to do on my network - please let me know if it's crazy and there's a better way to do it.

I have a Ubiquiti USG (10.0.0.2) that handles IP assignments since it's got a nice UI for static IPs and the like. I figured I'd have it point it's clients to my pi, (10.0.0.3) and all would be merry.

Here's the information you asked for:

root@pihole:/etc/dnsmasq.d# echo ">top-clients" | nc localhost 4711

tail -n25 /var/log/pihole.log0 2254 10.0.20.2
1 1480 10.0.20.3
2 38 127.0.0.1 localhost
3 6 10.0.0.2 unifi
4 2 10.0.110.6
5 2 10.0.0.157
6 2 10.0.0.90 optimus.home.mydomain.com
7 2 10.0.111.6
8 1 10.0.0.190
9 1 10.0.0.165
---EOM---



root@pihole:/etc/dnsmasq.d# echo ">top-domains" | nc localhost 4711
0 2752 panda-rancher-01.home.mydomain.com
1 2742 panda-rancher-01.home.mydomain.com.home.mydomain.com
2 22 panda-rancher-03.home.mydomain.com
3 10 panda-rancher-02.home.mydomain.com
4 8 www.gstatic.com
5 6 rancher.home.mydomain.com
6 6 s3.amazonaws.com
7 6 api.github.com
8 4 ping.ui.com
9 3 3.20.0.10.in-addr.arpa
---EOM---


root@pihole:/etc/dnsmasq.d# tail -n25 /var/log/pihole.log
Jan 14 01:32:44 dnsmasq[15262]: forwarded panda-rancher-01.home.mydomain.com to 10.0.0.1
Jan 14 01:32:44 dnsmasq[15262]: query[A] panda-rancher-01.home.mydomain.com from 10.0.20.2
Jan 14 01:32:44 dnsmasq[15262]: forwarded panda-rancher-01.home.mydomain.com to 10.0.0.1
Jan 14 01:32:44 dnsmasq[15262]: query[AAAA] panda-rancher-01.home.mydomain.com from 10.0.20.2
Jan 14 01:32:44 dnsmasq[15262]: forwarded panda-rancher-01.home.mydomain.com to 10.0.0.1
Jan 14 01:32:44 dnsmasq[15262]: query[AAAA] panda-rancher-01.home.mydomain.com.home.mydomain.com from 10.0.
20.3
Jan 14 01:32:44 dnsmasq[15262]: forwarded panda-rancher-01.home.mydomain.com.home.mydomain.com to 10.0.0.1
Jan 14 01:32:44 dnsmasq[15262]: query[A] panda-rancher-01.home.mydomain.com.home.mydomain.com from 10.0.20.
3
Jan 14 01:32:44 dnsmasq[15262]: forwarded panda-rancher-01.home.mydomain.com.home.mydomain.com to 10.0.0.1
Jan 14 01:32:44 dnsmasq[15262]: query[A] panda-rancher-01.home.mydomain.com.home.mydomain.com from 10.0.20.
2
Jan 14 01:32:44 dnsmasq[15262]: forwarded panda-rancher-01.home.mydomain.com.home.mydomain.com to 10.0.0.1
Jan 14 01:32:44 dnsmasq[15262]: query[AAAA] panda-rancher-01.home.mydomain.com.home.mydomain.com from 10.0.
20.2
Jan 14 01:32:44 dnsmasq[15262]: forwarded panda-rancher-01.home.mydomain.com.home.mydomain.com to 10.0.0.1
Jan 14 01:32:44 dnsmasq[15262]: query[A] panda-rancher-01.home.mydomain.com from 10.0.20.2
Jan 14 01:32:44 dnsmasq[15262]: forwarded panda-rancher-01.home.mydomain.com to 10.0.0.1
Jan 14 01:32:44 dnsmasq[15262]: query[AAAA] panda-rancher-01.home.mydomain.com from 10.0.20.2
Jan 14 01:32:44 dnsmasq[15262]: forwarded panda-rancher-01.home.mydomain.com to 10.0.0.1
Jan 14 01:32:44 dnsmasq[15262]: query[AAAA] panda-rancher-01.home.mydomain.com.home.mydomain.com from 10.0.
20.2
Jan 14 01:32:44 dnsmasq[15262]: forwarded panda-rancher-01.home.mydomain.com.home.mydomain.com to 10.0.0.1
Jan 14 01:32:44 dnsmasq[15262]: query[A] panda-rancher-01.home.mydomain.com.home.mydomain.com from 10.0.20.
2
Jan 14 01:32:44 dnsmasq[15262]: forwarded panda-rancher-01.home.mydomain.com.home.mydomain.com to 10.0.0.1
Jan 14 01:32:44 dnsmasq[15262]: query[AAAA] panda-rancher-01.home.mydomain.com from 10.0.20.3
Jan 14 01:32:45 dnsmasq[15262]: forwarded panda-rancher-01.home.mydomain.com to 10.0.0.1
Jan 14 01:32:45 dnsmasq[15262]: query[A] panda-rancher-01.home.mydomain.com from 10.0.20.3
Jan 14 01:32:45 dnsmasq[15262]: forwarded panda-rancher-01.home.mydomain.com to 10.0.0.1
root@pihole:/etc/dnsmasq.d#

My dnsmsq.d has a 02-custom.conf that might be of interest as well:

root@pihole:/etc/dnsmasq.d# cat 02-custom.conf
# Could use a LB in front of it?
address=/internal.home.mydomain.com/10.0.110.3
# Could use a LB in front of it?
address=/external.home.mydomain.com/10.0.111.3
address=/rancher.home.mydomain.com/10.0.10.0
address=/pihole.home.mydomain.com/10.0.0.3

# metallb assigned
address=/unitycache.home.mydomain.com/10.0.110.202

domain=home.mydomain.com,10.0.0.0/16,local

jfb · January 14, 2020, 3:36am

Please generate a new debug log and post the token. The old one is expired and the outputs you show don't match the traffic volume reported in the earlier debug log.

eddieparker · January 14, 2020, 3:45am

New debug token: https://tricorder.pi-hole.net/y9ybdzki7k. I'm assuming this has the logs you need?

I should mention that in the intervening time I noticed that one of the hosts it was looking for repeatedly was down. I since rebooted that machine and now my dns traffic seems to have stabilized. I'm guessing the error is still there, but it only shows itself when a host is down?

jfb · January 14, 2020, 3:54am

The volume of traffic appears to have decreased significantly, to 325K requests in the last 24 hours. You may have resolved the problem.

eddieparker · January 14, 2020, 4:19am

Oh; so you think the issue was just a set of clients being naughty? I was wondering if I somehow had pihole stuck in a loop when this error condition happened.

jfb · January 14, 2020, 5:01am

Tale a look at your query volume in 12 hours. The data shown is a rolling 24 hour average and you want to look 24 hours after you fixed that client.

system · February 4, 2020, 5:01am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.