Pihole keeps going offline

nanos · September 26, 2018, 11:12am

So, this has been going on for a while now: I have a Ubuntu server that I use for VPN server when travelling. I run pihole on that server, and a few websites (using apache, lighttpd isn't active).

Every now and then (sometimes every few minutes, sometimes every few hours) dns resolution just stops working:

Expected Behaviour:

I should be able to resolve DNS.
The Web interface should say "Active" and load data.

Actual Behaviour:

DNS resolution doesn't work.
Web interface says "DNS service not running" " FTL offline" "Lost connection to API"

When I do systemctl status pihole-FTL I get this output, showing FTL is definitely running:

$ sudo systemctl     status pihole-FTL
● pihole-FTL.service - LSB: pihole-FTL daemon
   Loaded: loaded (/etc/init.d/pihole-FTL; bad; vendor preset: enabled)
   Active: active (exited) since Wed 2018-09-26 11:53:01 BST; 9min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 21304 ExecStop=/etc/init.d/pihole-FTL stop (code=exited, status=0/SUCCESS)
  Process: 21312 ExecStart=/etc/init.d/pihole-FTL start (code=exited, status=0/SUCCESS)

Sep 26 11:53:01 do1 systemd[1]: Stopped LSB: pihole-FTL daemon.
Sep 26 11:53:01 do1 systemd[1]: Starting LSB: pihole-FTL daemon...
Sep 26 11:53:01 do1 pihole-FTL[21312]: Not running
Sep 26 11:53:01 do1 pihole-FTL[21312]: chown: cannot access '/etc/pihole/dhcp.leases': No such file or directory
Sep 26 11:53:01 do1 su[21338]: Successful su for pihole by root
Sep 26 11:53:01 do1 su[21338]: + ??? root:pihole
Sep 26 11:53:01 do1 su[21338]: pam_unix(su:session): session opened for user pihole by (uid=0)
Sep 26 11:53:01 do1 pihole-FTL[21312]: FTL started!
Sep 26 11:53:01 do1 systemd[1]: Started LSB: pihole-FTL daemon.

Simply doing systemctl restart pihole-FTL fixes the issue - until it recurs a few min/hours later...

Debug Token:

Can't get one, because I cannot upload the log due to broken DNS resolution. Happy to upload and/or send the log somewhere confidential.

jfb · September 26, 2018, 12:11pm

If you reset your name server temporarily, you should be able to upload the debug log:

sudo nano /etc/resolv.conf
Change 127.0.0.1 to 1.1.1.1, save and exit

Then run debug and upload the log.

nanos · September 26, 2018, 1:16pm

Thanks. I'll do that the next time it happens (obviously it's been running smoothly since I posted ...)

crabc4ke · September 26, 2018, 2:37pm

I have this issue too, running on a RPi 3. Have never been able to get to it when its happening but can confirm that the console is completely unavailable. Leave it for a short period of time (minutet(s)) and it returns. All internet based activity fails during an incident - YouTube, web browsing, streaming etc. Raspian and PiHole are fully up to date.

Bit new to debugging - whats required to assist troubleshooting please ?

nanos · September 26, 2018, 3:06pm

I now have a token: 03fzavx7w3

Thanks in advance!

RamSet · September 27, 2018, 4:10pm

when DNS resolution is failing, what is the output of sudo systemctl status pihole-FTL ?

There's a known issue with ubuntu and networkmanager (and it's out-of-the-box dnsmasq requirement).

If FTLDNS is not starting, check to see if you have any active dnsmasq processes.

Do you by any chance have NewtworkManager installed? (which NetworkManager)

nanos · September 28, 2018, 5:57am

Hi,

thanks for your response!

I can't see anything obviously wrong here (Just to be absolutely clear: This is the output while DNS resolution is failing):

$ sudo systemctl status pihole-FTL.service
● pihole-FTL.service - LSB: pihole-FTL daemon
   Loaded: loaded (/etc/init.d/pihole-FTL; bad; vendor preset: enabled)
   Active: active (exited) since Thu 2018-09-27 09:40:06 BST; 21h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 28820 ExecStop=/etc/init.d/pihole-FTL stop (code=exited, status=0/SUCCESS)
  Process: 28828 ExecStart=/etc/init.d/pihole-FTL start (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 0B
      CPU: 0

Sep 27 09:40:06 do1 systemd[1]: Stopped LSB: pihole-FTL daemon.
Sep 27 09:40:06 do1 systemd[1]: Starting LSB: pihole-FTL daemon...
Sep 27 09:40:06 do1 pihole-FTL[28828]: Not running
Sep 27 09:40:06 do1 su[28854]: Successful su for pihole by root
Sep 27 09:40:06 do1 su[28854]: + ??? root:pihole
Sep 27 09:40:06 do1 su[28854]: pam_unix(su:session): session opened for user pihole by (uid=0)
Sep 27 09:40:06 do1 pihole-FTL[28828]: FTL started!
Sep 27 09:40:06 do1 systemd[1]: Started LSB: pihole-FTL daemon.

Nope. I mean, firstly TFLDNS appears to be running just fine, based on the output above, and dnsmasq is not active:

$ sudo systemctl status dnsmasq
● dnsmasq.service - dnsmasq - A lightweight DHCP and caching DNS server
   Loaded: loaded (/lib/systemd/system/dnsmasq.service; disabled; vendor preset: enabled)
  Drop-In: /run/systemd/generator/dnsmasq.service.d
           └─50-dnsmasq-$named.conf, 50-insserv.conf-$named.conf
   Active: inactive (dead)

No

RamSet · September 28, 2018, 1:45pm

That looks like it should work.

Try running nslookup flurry.com on the ubuntu hosting Pi-hole and the same command on a client. Lets see what those outputs looke like ...

nanos · September 28, 2018, 2:10pm

I know ...

On the ubuntu machine:

$ nslookup flurry.com
;; connection timed out; no servers could be reached

On my home raspberry pi:

$ nslookup flurry.com
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   flurry.com
Address: 0.0.0.0

Basically pihole just isn't answering

I might add that doing sudo lsof -i :53 gives an empty output.

The following sequence makes no sense whatsoever to me:

$ sudo lsof -i :53 

$ sudo systemctl status pihole-FTL.service
● pihole-FTL.service - LSB: pihole-FTL daemon
   Loaded: loaded (/etc/init.d/pihole-FTL; bad; vendor preset: enabled)
   Active: active (exited) since Thu 2018-09-27 09:40:06 BST; 1 day 5h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 28820 ExecStop=/etc/init.d/pihole-FTL stop (code=exited, status=0/SUCCESS)
  Process: 28828 ExecStart=/etc/init.d/pihole-FTL start (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 0B
      CPU: 0

Sep 27 09:40:06 do1 systemd[1]: Stopped LSB: pihole-FTL daemon.
Sep 27 09:40:06 do1 systemd[1]: Starting LSB: pihole-FTL daemon...
Sep 27 09:40:06 do1 pihole-FTL[28828]: Not running
Sep 27 09:40:06 do1 su[28854]: Successful su for pihole by root
Sep 27 09:40:06 do1 su[28854]: + ??? root:pihole
Sep 27 09:40:06 do1 su[28854]: pam_unix(su:session): session opened for user pihole by (uid=0)
Sep 27 09:40:06 do1 pihole-FTL[28828]: FTL started!
Sep 27 09:40:06 do1 systemd[1]: Started LSB: pihole-FTL daemon.

$ sudo systemctl restart pihole-FTL.service

$ sudo lsof -i :53
COMMAND     PID   USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
pihole-FT 31942 pihole    4u  IPv4 1922499      0t0  UDP *:domain
pihole-FT 31942 pihole    5u  IPv4 1922500      0t0  TCP *:domain (LISTEN)
pihole-FT 31942 pihole    6u  IPv6 1922501      0t0  UDP *:domain
pihole-FT 31942 pihole    7u  IPv6 1922502      0t0  TCP *:domain (LISTEN)

$ sudo systemctl status pihole-FTL.service
● pihole-FTL.service - LSB: pihole-FTL daemon
   Loaded: loaded (/etc/init.d/pihole-FTL; bad; vendor preset: enabled)
   Active: active (exited) since Fri 2018-09-28 15:14:27 BST; 1s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 31894 ExecStop=/etc/init.d/pihole-FTL stop (code=exited, status=0/SUCCESS)
  Process: 31903 ExecStart=/etc/init.d/pihole-FTL start (code=exited, status=0/SUCCESS)

Sep 28 15:14:27 do1 systemd[1]: Stopped LSB: pihole-FTL daemon.
Sep 28 15:14:27 do1 systemd[1]: Starting LSB: pihole-FTL daemon...
Sep 28 15:14:27 do1 pihole-FTL[31903]: Not running
Sep 28 15:14:27 do1 su[31930]: Successful su for pihole by root
Sep 28 15:14:27 do1 su[31930]: + ??? root:pihole
Sep 28 15:14:27 do1 su[31930]: pam_unix(su:session): session opened for user pihole by (uid=0)
Sep 28 15:14:27 do1 pihole-FTL[31903]: FTL started!
Sep 28 15:14:27 do1 systemd[1]: Started LSB: pihole-FTL daemon.

RamSet · September 28, 2018, 4:15pm

*** [ DIAGNOSING ]: Networking
[✓] IPv4 address(es) bound to the eth0 interface:
   *:80/18 matches the IP found in /etc/pihole/setupVars.conf
   *:5/16 does not match the IP found in /etc/pihole/setupVars.conf (https://discourse.pi-hole.net/t/use-ipv6-ula-addresses-for-pi-hole/2127)

[✓] IPv6 address(es) bound to the eth0 interface:
 *:4001 matches the IP found in /etc/pihole/setupVars.conf
   *:9d64 does not match the IP found in /etc/pihole/setupVars.conf (https://discourse.pi-hole.net/t/use-ipv6-ula-addresses-for-pi-hole/2127)

   ^ Please note that you may have more than one IP address listed.
   As long as one of them is green, and it matches what is in /etc/pihole/setupVars.conf, there is no need for concern.

   The link to the FAQ is for an issue that sometimes occurs when the IPv6 address changes, which is why we check for it.

[i] Default Pv4 gateway: *:.1
   * Pinging *:.1...
[✓] Gateway responded.
[i] Default IPv6 gateway: *::1
   * Pinging *::1...
[✓] Gateway responded.

Something is wonky with your settings ...

That still does not explain why the (at least) local resolution does not work.

This actually worked ...

Do you have an instance of Pi-hole on the raspberry pi ?

nanos · September 28, 2018, 4:22pm

I'm not sure what you mean? It's about what I'd expect? The one adress is the external ip, the other is my VPN IP, I think.

Yes and yes.

Still don't understand why pihole-FTL is running, but not listening on port 53?

RamSet · September 28, 2018, 4:34pm

You're on /18 subnet with that IP. Is it assigned straight from your ISP (as in bridged router ?)

Try nslookup flurry.com 127.0.0.1 on the ubuntu.

run a dig pi-hole.net @127.0.0.1 -p 53 on the same ubuntu and see what it replies

nanos · October 1, 2018, 7:56am

Sorry, been mostly offline over the weekend. Interestingly, pihole has stayed up all weekend, which was somewhat unexpected, and means I can't really test the the queries. Will report back, the next time it turns up.

Out of curiosity:

I'm quite frankly not entirely sure what that means. I'm vaguely familiar with what a subnet is, but what makes /18 special? Is it just the size of it? I did get it assigned from my provider: It's a DigitalOcean droplet, and I didn't really touch the interface config.

RamSet · October 1, 2018, 7:23pm

Correct. It's related to the size. And if not deployed by you, then you don't have access to manage it.

Unrelated though in this case.

nanos · October 2, 2018, 9:52am

@RamSet And it just crashed again. Here are the requested queries:

$ nslookup flurry.com 127.0.0.1
;; connection timed out; no servers could be reached

$ dig pi-hole.net @127.0.0.1 -p 53

; <<>> DiG 9.10.3-P4-Ubuntu <<>> pi-hole.net @127.0.0.1 -p 53
;; global options: +cmd
;; connection timed out; no servers could be reached

systemctl status pihole-FTL still reports the service as active.

nanos · October 5, 2018, 9:04am

Sorry, @RamSet but would you have any other thoughts or suggestions on this?

nanos · October 24, 2018, 11:23am

So, I thought I should provide some update on this.

TL;DR: It appears the machine was running out of memory. Increasing memory solved it.

I noticed about a week ago, that I was still running on one of DigitalOcean's legacy plans with 512 MB of RAM, so upgraded to 1 GB. Pihole has been stable since.

I should note that I did switch to using dnsmasq rather than pihole-FTL for DNS resolution in the mean time, and dnsmasq was running stably on 512 MB of memory (as I'd expect it to do, given it really isn't doing much.)

@RamSet / @jfb According to Hardware/Software Requirements 512 MB should've been sufficient. Just as a data point, that for me it didn't run stably at all on 512 MB memory since upgrading to 4.0.

system · November 14, 2018, 11:23am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.