DNS not running, FTL down

kidspi · September 5, 2020, 8:38am

Please follow the below template, it will help us to help you!

If you are Experiencing issues with a Pi-hole install that has non-standard elements (e.g you are using `nginx` instead of `lighttpd`, or there is some other aspect of your install that is customised) - please use the Community Help category.

Expected Behaviour:

_[Replace this text with what you think should be happening. Please include as much detail as possible including, but not limited to:
-Buster
-Rpi3

Actual Behaviour:

DNS service not running. FTL on web interface is down.

Debug Token:

There was an error uploading your debug log.

Bucking_Horn · September 5, 2020, 9:02am

Your description is a bit brief.
Does that relate to first time install, an upgrade or an issue manifesting for a long running Pi-hole installation?

You may retry uploading a debug log by explictly providing a DNS server.

Temporarily reset the nameserver on the Pi to bypass Pi-Hole DNS:

sudo nano /etc/resolv.conf

Edit the nameserver line to nameserver 9.9.9.9 or your preferred third party DNS service, save and exit.

Run

pihole -d

and upload the debug log.

kidspi · September 5, 2020, 9:20am

It is an issue running in a pihole for the last year.

I have uploaded the debug log.

https://tricorder.pi-hole.net/4j1fh7lj14

Thank you.

Bucking_Horn · September 5, 2020, 9:50am

From your debug log, it would seem your Pi-hole has IPv4 connectivity, but fails to bind the DNS port 53. In consequence, it fails to receive and resolve any DNS queries: (click for log excerpt details)

*** [ DIAGNOSING ]: Networking
[✓] IPv4 address(es) bound to the eth0 interface:
   192.168.1.6/24 matches the IP found in /etc/pihole/setupVars.conf

[i] Default IPv4 gateway: 192.168.1.1
   * Pinging 192.168.1.1...
[✓] Gateway responded.

*** [ DIAGNOSING ]: Ports in use
127.0.0.1:45975 cloudflare (IPv4)
127.0.0.1:5053 cloudflare (IPv4)
*:5900 vncserver- (IPv6)
*:5900 vncserver- (IPv4)
*:22 sshd (IPv4)
*:22 sshd (IPv6)
[80] is in use by lighttpd
[80] is in use by lighttpd

*** [ DIAGNOSING ]: Name resolution (IPv4) using a random blocked domain and a known ad-serving domain
[✗] Failed to resolve hasznaltgumivetel.hu via localhost (127.0.0.1)
[✗] Failed to resolve hasznaltgumivetel.hu via Pi-hole (192.168.1.6)
[✓] doubleclick.com is 216.58.192.206 via a remote, public DNS server (8.8.8.8)

Let's see what processes are grabbbing ports on your system.
Run on your Pi-hole machine, what's the output of:

sudo ss -tulpn

kidspi · September 5, 2020, 10:10am

Thank you.

Here is the output -

$ sudo ss -tulpn
Netid  State   Recv-Q  Send-Q     Local Address:Port      Peer Address:Port                                                                               
udp    UNCONN  0       0                0.0.0.0:55965          0.0.0.0:*      users:(("avahi-daemon",pid=357,fd=14))                                      
udp    UNCONN  0       0                0.0.0.0:5353           0.0.0.0:*      users:(("avahi-daemon",pid=357,fd=12))                                      
udp    UNCONN  0       0              127.0.0.1:5053           0.0.0.0:*      users:(("cloudflared",pid=465,fd=5))                                        
udp    UNCONN  0       0                0.0.0.0:68             0.0.0.0:*      users:(("dhcpcd",pid=428,fd=10))                                            
udp    UNCONN  0       0                      *:5353                 *:*      users:(("avahi-daemon",pid=357,fd=13))                                      
udp    UNCONN  0       0                      *:52118                *:*      users:(("avahi-daemon",pid=357,fd=15))                                      
tcp    LISTEN  0       4096           127.0.0.1:5053           0.0.0.0:*      users:(("cloudflared",pid=465,fd=6))                                        
tcp    LISTEN  0       5                0.0.0.0:5900           0.0.0.0:*      users:(("vncserver-x11-c",pid=501,fd=11))                                   
tcp    LISTEN  0       1024             0.0.0.0:80             0.0.0.0:*      users:(("lighttpd",pid=559,fd=4))                                           
tcp    LISTEN  0       128              0.0.0.0:22             0.0.0.0:*      users:(("sshd",pid=526,fd=3))                                               
tcp    LISTEN  0       4096           127.0.0.1:45975          0.0.0.0:*      users:(("cloudflared",pid=465,fd=3))                                        
tcp    LISTEN  0       5                   [::]:5900              [::]:*      users:(("vncserver-x11-c",pid=501,fd=10))                                   
tcp    LISTEN  0       1024                [::]:80                [::]:*      users:(("lighttpd",pid=559,fd=5))                                           
tcp    LISTEN  0       128                 [::]:22                [::]:*      users:(("sshd",pid=526,fd=4))

Bucking_Horn · September 5, 2020, 10:42am

(You can format your output for readability by using the </> Preformatted text menu option. I've just done that for your above command output. )

Port 53 is not taken, neither by Pi-hole nor any other process.

Try restarting Pi-hole:

pihole restartdns

Check for any startup errors afterwards:

sudo systemctl status --full --no-pager pihole-FTL.service

kidspi · September 5, 2020, 10:49am

Here is output after restarting pi-hole


sudo systemctl status --full --no-pager pihole-FTL.service
● pihole-FTL.service - LSB: pihole-FTL daemon
   Loaded: loaded (/etc/init.d/pihole-FTL; generated)
   Active: active (exited) since Sat 2020-09-05 05:46:51 CDT; 1min 47s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 5215 ExecStart=/etc/init.d/pihole-FTL start (code=exited, status=0/SUCCESS)

Sep 05 05:45:50 kidsraspberrypi systemd[1]: Starting LSB: pihole-FTL daemon...
Sep 05 05:45:50 kidsraspberrypi pihole-FTL[5215]: Not running
Sep 05 05:45:51 kidsraspberrypi su[5233]: (to pihole) root on none
Sep 05 05:45:51 kidsraspberrypi su[5233]: pam_unix(su:session): session opened for user pihole by (uid=0)
Sep 05 05:46:49 kidsraspberrypi pihole-FTL[5215]: FTL started!
Sep 05 05:46:51 kidsraspberrypi systemd[1]: Started LSB: pihole-FTL daemon.

BTW per the web interface FTL is still down.

Bucking_Horn · September 5, 2020, 11:16am

That looks like a normal startup without any errors.

Did Pi-hole bind to port 53 this time?

kidspi · September 5, 2020, 11:21am

here is the output


sudo ss -tulpn
Netid  State   Recv-Q  Send-Q     Local Address:Port      Peer Address:Port                                                                               
udp    UNCONN  0       0                0.0.0.0:55965          0.0.0.0:*      users:(("avahi-daemon",pid=357,fd=14))                                      
udp    UNCONN  0       0                0.0.0.0:5353           0.0.0.0:*      users:(("avahi-daemon",pid=357,fd=12))                                      
udp    UNCONN  0       0              127.0.0.1:5053           0.0.0.0:*      users:(("cloudflared",pid=465,fd=5))                                        
udp    UNCONN  0       0                0.0.0.0:68             0.0.0.0:*      users:(("dhcpcd",pid=428,fd=10))                                            
udp    UNCONN  0       0                      *:5353                 *:*      users:(("avahi-daemon",pid=357,fd=13))                                      
udp    UNCONN  0       0                      *:52118                *:*      users:(("avahi-daemon",pid=357,fd=15))                                      
tcp    LISTEN  0       4096           127.0.0.1:5053           0.0.0.0:*      users:(("cloudflared",pid=465,fd=6))                                        
tcp    LISTEN  0       5                0.0.0.0:5900           0.0.0.0:*      users:(("vncserver-x11-c",pid=501,fd=11))                                   
tcp    LISTEN  0       1024             0.0.0.0:80             0.0.0.0:*      users:(("lighttpd",pid=559,fd=4))                                           
tcp    LISTEN  0       128              0.0.0.0:22             0.0.0.0:*      users:(("sshd",pid=526,fd=3))                                               
tcp    LISTEN  0       4096           127.0.0.1:45975          0.0.0.0:*      users:(("cloudflared",pid=465,fd=3))                                        
tcp    LISTEN  0       5                   [::]:5900              [::]:*      users:(("vncserver-x11-c",pid=501,fd=10))                                   
tcp    LISTEN  0       1024                [::]:80                [::]:*      users:(("lighttpd",pid=559,fd=5))                                           
tcp    LISTEN  0       128                 [::]:22                [::]:*      users:(("sshd",pid=526,fd=4))

kidspi · September 5, 2020, 11:24am

I have uploaded another debug log as well.

https://tricorder.pi-hole.net/1b6pd3cacw

The Web interface says DNS not running, FTL still down

Bucking_Horn · September 5, 2020, 11:45am

The new debug log just looks the same as your first.
It's probably worth noting that content of pihole-FTL.log has changed, while pihole.log just stayed the same.
This is coherent with Pi-hole starting up and logging normally, but not processing and logging any queries.

Let's check if pihole-FTL is running at all:

ps -e | grep "PID\|pihole"

And also check dnsmasq configuration for possible syntax errors:

pihole-FTL dnsmasq-test

kidspi · September 5, 2020, 11:52am

~ $ ps -e | grep "PID|pihole"
PID TTY TIME CMD

pihole-FTL dnsmasq-test
dnsmasq: syntax check OK.

Bucking_Horn · September 5, 2020, 12:09pm

It seems Pi-hole is indeed not running anymore.

Your recent pihole-FTL.log shows it to have started with the following PID:

   [2020-09-05 05:46:49.217 5260M] PID of FTL process: 5260

sudo lsof -p 5260 likely won't return anything, as no Pi-hole process seems to be active.

Try checking the service again:

sudo systemctl status --full --no-pager pihole-FTL.service

Also, check the current end of /var/log/pihole-FTL.log for any errors that would imply a shutdown of pihole-FTL.

And check for under-voltage situations also:

dmesg -T | grep -i voltage

kidspi · September 5, 2020, 12:13pm

Before my communication with you, I noticed that I can restart the RPI and every thing would work fine for about 10 min then DNS would stop working and FTL was down.

sudo lsof -p 5260
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.

$ sudo systemctl status --full --no-pager pihole-FTL.service
● pihole-FTL.service - LSB: pihole-FTL daemon
   Loaded: loaded (/etc/init.d/pihole-FTL; generated)
   Active: active (exited) since Sat 2020-09-05 05:46:51 CDT; 1h 25min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 5215 ExecStart=/etc/init.d/pihole-FTL start (code=exited, status=0/SUCCESS)

Sep 05 05:45:50 kidsraspberrypi systemd[1]: Starting LSB: pihole-FTL daemon...
Sep 05 05:45:50 kidsraspberrypi pihole-FTL[5215]: Not running
Sep 05 05:45:51 kidsraspberrypi su[5233]: (to pihole) root on none
Sep 05 05:45:51 kidsraspberrypi su[5233]: pam_unix(su:session): session opened for user pihole by (uid=0)
Sep 05 05:46:49 kidsraspberrypi pihole-FTL[5215]: FTL started!
Sep 05 05:46:51 kidsraspberrypi systemd[1]: Started LSB: pihole-FTL daemon.
pi@kidsraspberrypi:~ $

This line of code does not return anything.

pi@kidsraspberrypi:~ $ dmesg -T | grep -i voltage

deHakkelaar · September 5, 2020, 12:40pm

Try running pihole-FTL in the foreground -f with below and post the output when it stops (or not):

sudo setcap CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_NET_ADMIN,CAP_SYS_NICE+eip "$(which pihole-FTL)"; sudo su -s /bin/sh -c "/usr/bin/pihole-FTL -f"

You would need to kill it again in another SSH session if still running and when finished diagnosing with:

sudo kill $(pidof pihole-FTL)

jfb · September 5, 2020, 1:09pm

Likely out of memory:

   [2020-09-05 03:48:46.017 1422M] Resizing "/FTL-queries" from 484671488 to 484900864
   [2020-09-05 03:48:46.036 1422M] Imported 8658199 queries from the long-term database
   [2020-09-05 03:48:46.037 1422M]  -> Total DNS queries: 8658199
   [2020-09-05 03:48:46.037 1422M]  -> Cached DNS queries: 47
   [2020-09-05 03:48:46.037 1422M]  -> Forwarded DNS queries: 8658151
   [2020-09-05 03:48:46.037 1422M]  -> Blocked DNS queries: 1
   [2020-09-05 03:48:46.037 1422M]  -> Unknown DNS queries: 0
   [2020-09-05 03:48:46.037 1422M]  -> Unique domains: 51
   [2020-09-05 03:48:46.037 1422M]  -> Unique clients: 10
   [2020-09-05 03:48:46.038 1422M]  -> Known forward destinations: 3

kidspi · September 5, 2020, 1:12pm

How do I fix the memory issue. This is a dedicated rpi running just the stock buster version and Pihole.

jfb · September 5, 2020, 1:14pm

First, take a look at where 8.6 million DNS requests are coming from. What is the output of these commands from the Pi terminal:

echo ">top-clients >quit" | nc localhost 4711

echo ">top-domains >quit" | nc localhost 4711

echo ">top-ads >quit" | nc localhost 4711

kidspi · September 5, 2020, 1:17pm

Not sure I am doing this right.

I typed your suggestions into terminal and there appears to be no output.

pi@kidsraspberrypi:~ $ echo ">top-clients >quit" | nc localhost 4711
pi@kidsraspberrypi:~ $ echo ">top-domains >quit" | nc localhost 4711
pi@kidsraspberrypi:~ $ echo ">top-ads >quit" | nc localhost 4711

jfb · September 5, 2020, 1:25pm

Restart FTL and run them again.