Unexplained pi-hole fail

Obidose · June 26, 2019, 8:26pm

Please follow the below template, it will help us to help you!

Expected Behaviour:

[PiHole had been running very well (with unbound also acting as upstream DNS) for around a month without any issues.]

Actual Behaviour:

_[Suddenly and without any changes by myself PiHole stopped working. As I had this set as my sole DNS, and I also force all DNS traffic through this using firewall settings in Unifi gateway, this effectively knocked out my internet.
I was able to SSH in to my pi and also to look at the web interface.
The web interface showed that FTL was offline and also the DNS service. I could restrart the DNS briefly before it crashed again. FTL remained offline throughout.

I managed to temporarily deal with this by reverting my router to use an external DNS.

Meanwhile I did the following updated my Pi resolv.conf to an external DNS and then ran pihole -r
This seemed to fix the problem but I am reluctant to leave this without an answer as to why it broke.
While this is only a home network, having my pi setup as my sole DNS makes it fairly mission critical and so I am afraid this will happen again and a very inconvenient time so any thoughts on causes would be great.]_

Debug Token:

_[*** [ INITIALIZING ]
[i] 2019-06-26:19:48:52 debug log has been initialized.

*** [ INITIALIZING ] Sourcing setup variables
[i] Sourcing /etc/pihole/setupVars.conf...

*** [ DIAGNOSING ]: Core version
[i] Core: v4.3 (How do I update Pi-hole?)
[i] Branch: master
[i] Commit: v4.3-0-g1d43c0a

*** [ DIAGNOSING ]: Web version
[i] Web: v4.3 (How do I update Pi-hole?)
[i] Branch: master
[i] Commit: v4.3-0-g44aff72

*** [ DIAGNOSING ]: FTL version
[?] FTL: v4.3 (How do I update Pi-hole?)

*** [ DIAGNOSING ]: lighttpd version
[i] 1.4.45

*** [ DIAGNOSING ]: php version
[i] 7.0.33

*** [ DIAGNOSING ]: Operating system
[?] Raspbian GNU/Linux 9 (stretch)

*** [ DIAGNOSING ]: SELinux
[i] SELinux not detected

*** [ DIAGNOSING ]: Processor
[?] armv7l

*** [ DIAGNOSING ]: Networking
[?] IPv4 address(es) bound to the eth0 interface:
192.168.1.111/24 matches the IP found in /etc/pihole/setupVars.conf

[?] IPv6 address(es) bound to the eth0 interface:
fe80::fb64:adc1:d2d2:e66b does not match the IP found in /etc/pihole/setupVars.conf (Use IPv6 ULA addresses for Pi-hole)

^ Please note that you may have more than one IP address listed.
As long as one of them is green, and it matches what is in /etc/pihole/setupVars.conf, there is no need for concern.

The link to the FAQ is for an issue that sometimes occurs when the IPv6 address changes, which is why we check for it.

[i] Default IPv4 gateway: 192.168.1.1

Pinging 192.168.1.1...
[?] Gateway responded.

*** [ DIAGNOSING ]: Ports in use
:22 sshd (IPv4)
:22 sshd (IPv6)
127.0.0.1:5353 unbound (IPv4)
127.0.0.1:8953 unbound (IPv4)
[:80] is in use by lighttpd
[:80] is in use by lighttpd
*:8080 java (IPv6)
*:8880 java (IPv6)
*:8443 java (IPv6)
*:8843 java (IPv6)
*:6789 java (IPv6)
127.0.0.1:27117 mongod (IPv4)

*** [ DIAGNOSING ]: Name resolution (IPv4) using a random blocked domain and a known ad-serving domain
[?] Failed to resolve pepcriskalertus.club via localhost (127.0.0.1)
[?] Failed to resolve pepcriskalertus.club via Pi-hole (192.168.1.111)
[?] doubleclick.com is 172.217.169.78 via a remote, public DNS server (8.8.8.8)

*** [ DIAGNOSING ]: Pi-hole processes
[?] lighttpd daemon is active
[?] pihole-FTL daemon is active

*** [ DIAGNOSING ]: Setup variables
PIHOLE_INTERFACE=eth0
IPV4_ADDRESS=192.168.1.111/24
IPV6_ADDRESS=
QUERY_LOGGING=true
INSTALL_WEB_SERVER=true
INSTALL_WEB_INTERFACE=true
LIGHTTPD_ENABLED=true
BLOCKING_ENABLED=true
DNSMASQ_LISTENING=single
PIHOLE_DNS_1=208.67.222.222
PIHOLE_DNS_2=208.67.220.220
DNS_FQDN_REQUIRED=true
DNS_BOGUS_PRIV=true
DNSSEC=false
CONDITIONAL_FORWARDING=false

*** [ DIAGNOSING ]: Dashboard and block page
[?] Block page X-Header: X-Pi-hole: A black hole for Internet advertisements.
[?] Web interface X-Header: X-Pi-hole: The Pi-hole Web interface is working!

*** [ DIAGNOSING ]: Gravity list
-rw-r--r-- 1 root root 15600162 Jun 23 03:49 /etc/pihole/gravity.list
-----head of gravity.list------
-rotation.de
-sso.anbtr.com
-traffic.com
0-07.ru

-----tail of gravity.list------
zzzrtrcm2.com
zzzxxxcc.no-ip.biz
zzzyzs.net
zzzzzqp.com

*** [ DIAGNOSING ]: contents of /etc/pihole

-rw-r--r-- 1 root root 48 Jun 23 03:49 /etc/pihole/local.list
192.168.1.111 raspberrypi
192.168.1.111 pi.hole

-rw-r--r-- 1 root root 234 May 25 00:14 /etc/pihole/logrotate
/var/log/pihole.log {
su root root
daily
copytruncate
rotate 5
compress
delaycompress
notifempty
nomail
}
/var/log/pihole-FTL.log {
su root root
weekly
copytruncate
rotate 3
compress
delaycompress
notifempty
nomail
}

*** [ DIAGNOSING ]: contents of /etc/dnsmasq.d

-rw-r--r-- 1 root root 1625 Jun 26 19:33 /etc/dnsmasq.d/01-pihole.conf
addn-hosts=/etc/pihole/gravity.list
addn-hosts=/etc/pihole/black.list
addn-hosts=/etc/pihole/local.list
localise-queries
no-resolv
cache-size=10000
log-queries
log-facility=/var/log/pihole.log
local-ttl=2
log-async
dhcp-name-match=set:wpad-ignore,wpad
dhcp-ignore-names=tag:wpad-ignore
server=208.67.222.222
server=208.67.220.220
domain-needed
bogus-priv
interface=eth0

*** [ DIAGNOSING ]: contents of /etc/lighttpd

-rw-r--r-- 1 root root 3102 May 25 00:14 /etc/lighttpd/lighttpd.conf
server.modules = (
"mod_access",
"mod_accesslog",
"mod_auth",
"mod_expire",
"mod_compress",
"mod_redirect",
"mod_setenv",
"mod_rewrite"
)
server.document-root = "/var/www/html"
server.error-handler-404 = "pihole/index.php"
server.upload-dirs = ( "/var/cache/lighttpd/uploads" )
server.errorlog = "/var/log/lighttpd/error.log"
server.pid-file = "/var/run/lighttpd.pid"
server.username = "www-data"
server.groupname = "www-data"
server.port = 80
accesslog.filename = "/var/log/lighttpd/access.log"
accesslog.format = "%{%s}t|%V|%r|%s|%b"
index-file.names = ( "index.php", "index.html", "index.lighttpd.html" )
url.access-deny = ( "~", ".inc", ".md", ".yml", ".ini" )
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
compress.cache-dir = "/var/cache/lighttpd/compress/"
compress.filetype = ( "application/javascript", "text/css", "text/html", "text/plain" )
include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port
include_shell "/usr/share/lighttpd/create-mime.assign.pl"
include_shell "find /etc/lighttpd/conf-enabled -name '*.conf' -a ! -name 'letsencrypt.conf' -printf 'include "%p"
' 2>/dev/null"
$HTTP["url"] =~ "^/admin/" {

   setenv.add-response-header = (
       "X-Pi-hole" => "The Pi-hole Web interface is working!",
       "X-Frame-Options" => "DENY"
   )
   $HTTP["url"] =~ ".ttf$" {
       
       setenv.add-response-header = ( "Access-Control-Allow-Origin" => "*" )
   }

}
$HTTP["url"] =~ "^/admin/.(.*)" {
url.access-deny = ("")
}
include_shell "cat external.conf 2>/dev/null"

*** [ DIAGNOSING ]: contents of /etc/cron.d

-rw-r--r-- 1 root root 1704 May 25 00:14 /etc/cron.d/pihole
48 3 * * 7 root PATH="$PATH:/usr/local/bin/" pihole updateGravity >/var/log/pihole_updateGravity.log || cat /var/log/pihole_updateGravity.log
00 00 * * * root PATH="$PATH:/usr/local/bin/" pihole flush once quiet
@reboot root /usr/sbin/logrotate /etc/pihole/logrotate
*/10 * * * * root PATH="$PATH:/usr/local/bin/" pihole updatechecker local
58 19 * * * root PATH="$PATH:/usr/local/bin/" pihole updatechecker remote
@reboot root PATH="$PATH:/usr/local/bin/" pihole updatechecker remote reboot

*** [ DIAGNOSING ]: contents of /var/log/lighttpd

-rw-r--r-- 1 www-data www-data 2024 Jun 26 19:49 /var/log/lighttpd/error.log
2019-06-23 06:25:03: (server.c.1534) logfiles cycled UID = 0 PID = 13258
2019-06-25 22:31:42: (log.c.217) server started
2019-06-25 23:07:37: (log.c.217) server started
2019-06-25 23:18:22: (log.c.217) server started
2019-06-25 23:41:25: (log.c.217) server started
2019-06-26 19:44:43: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: unlink(../custom_disable_timer): No such file or directory in /var/www/html/admin/api.php on line 47
2019-06-26 19:45:28: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: dns_get_record(): A temporary server error occurred. in /var/www/html/pihole/index.php on line 188
2019-06-26 19:45:28: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: array_key_exists() expects parameter 2 to be array, null given in /var/www/html/pihole/index.php on line 189
2019-06-26 19:45:28: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: dns_get_record(): A temporary server error occurred. in /var/www/html/pihole/index.php on line 188
2019-06-26 19:45:28: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: array_key_exists() expects parameter 2 to be array, null given in /var/www/html/pihole/index.php on line 189
2019-06-26 19:46:34: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: unlink(../custom_disable_timer): No such file or directory in /var/www/html/admin/api.php on line 47
2019-06-26 19:49:01: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: dns_get_record(): A temporary server error occurred. in /var/www/html/pihole/index.php on line 188
2019-06-26 19:49:01: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: array_key_exists() expects parameter 2 to be array, null given in /var/www/html/pihole/index.php on line 189
2019-06-26 19:49:01: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: dns_get_record(): A temporary server error occurred. in /var/www/html/pihole/index.php on line 188
2019-06-26 19:49:01: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: array_key_exists() expects parameter 2 to be array, null given in /var/www/html/pihole/index.php on line 189

*** [ DIAGNOSING ]: contents of /var/log

-rw-r--r-- 1 pihole pihole 9930 Jun 26 19:34 /var/log/pihole-FTL.log
-----head of pihole-FTL.log------
[2019-06-26 19:30:16.205 670] Shutting down...
[2019-06-26 19:30:16.680 670] Finished final database update
[2019-06-26 19:30:16.684 670] ########## FTL terminated after 71327360.0 ms! ##########
[2019-06-26 19:30:18.109 1817] Using log file /var/log/pihole-FTL.log
[2019-06-26 19:30:18.110 1817] ########## FTL started! ##########
[2019-06-26 19:30:18.110 1817] FTL branch: master
[2019-06-26 19:30:18.110 1817] FTL version: v4.3
[2019-06-26 19:30:18.110 1817] FTL commit: 9e8273b
[2019-06-26 19:30:18.110 1817] FTL date: 2019-05-18 16:00:39 -0400
[2019-06-26 19:30:18.110 1817] FTL user: pihole
[2019-06-26 19:30:18.110 1817] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
[2019-06-26 19:30:18.110 1817] SOCKET_LISTENING: only local
[2019-06-26 19:30:18.111 1817] AAAA_QUERY_ANALYSIS: Show AAAA queries
[2019-06-26 19:30:18.111 1817] MAXDBDAYS: max age for stored queries is 365 days
[2019-06-26 19:30:18.111 1817] RESOLVE_IPV6: Resolve IPv6 addresses
[2019-06-26 19:30:18.111 1817] RESOLVE_IPV4: Resolve IPv4 addresses
[2019-06-26 19:30:18.111 1817] DBINTERVAL: saving to DB file every minute
[2019-06-26 19:30:18.111 1817] DBFILE: Using /etc/pihole/pihole-FTL.db
[2019-06-26 19:30:18.111 1817] MAXLOGAGE: Importing up to 24.0 hours of log data
[2019-06-26 19:30:18.111 1817] PRIVACYLEVEL: Set to 0
[2019-06-26 19:30:18.111 1817] IGNORE_LOCALHOST: Show queries from localhost
[2019-06-26 19:30:18.112 1817] BLOCKINGMODE: Null IPs for blocked domains
[2019-06-26 19:30:18.112 1817] ANALYZE_ONLY_A_AND_AAAA: Disabled. Analyzing all queries
[2019-06-26 19:30:18.112 1817] DBIMPORT: Importing history from database
[2019-06-26 19:30:18.112 1817] PIDFILE: Using /var/run/pihole-FTL.pid
[2019-06-26 19:30:18.112 1817] PORTFILE: Using /var/run/pihole-FTL.port
[2019-06-26 19:30:18.112 1817] SOCKETFILE: Using /var/run/pihole/FTL.sock
[2019-06-26 19:30:18.112 1817] WHITELISTFILE: Using /etc/pihole/whitelist.txt
[2019-06-26 19:30:18.112 1817] BLACKLISTFILE: Using /etc/pihole/black.list
[2019-06-26 19:30:18.112 1817] GRAVITYFILE: Using /etc/pihole/gravity.list
[2019-06-26 19:30:18.113 1817] REGEXLISTFILE: Using /etc/pihole/regex.list
[2019-06-26 19:30:18.113 1817] SETUPVARSFILE: Using /etc/pihole/setupVars.conf
[2019-06-26 19:30:18.113 1817] AUDITLISTFILE: Using /etc/pihole/auditlog.list
[2019-06-26 19:30:18.113 1817] MACVENDORDB: Using /etc/pihole/macvendor.db
[2019-06-26 19:30:18.113 1817] PARSE_ARP_CACHE: Active

-----tail of pihole-FTL.log------
[2019-06-26 19:33:53.088 2062] -> Unique domains: 149
[2019-06-26 19:33:53.088 2062] -> Unique clients: 15
[2019-06-26 19:33:53.088 2062] -> Known forward destinations: 1
[2019-06-26 19:33:53.088 2062] Successfully accessed setupVars.conf
[2019-06-26 19:33:53.097 2064] PID of FTL process: 2064
[2019-06-26 19:33:53.098 2064] Listening on port 4711 for incoming IPv4 telnet connections
[2019-06-26 19:33:53.099 2064] Listening on port 4711 for incoming IPv6 telnet connections
[2019-06-26 19:33:53.099 2064] Listening on Unix socket
[2019-06-26 19:33:53.102 2064] Received SIGHUP, reloading cache
[2019-06-26 19:33:53.102 2064] Blocking status is enabled
[2019-06-26 19:33:53.103 2064] INFO: No whitelist file found
[2019-06-26 19:33:53.103 2064] Compiled 0 Regex filters and 0 whitelisted domains in 0.3 msec (0 errors)
[2019-06-26 19:34:02.749 2064] /etc/pihole/gravity.list: parsed 733611 domains (took 9644.8 ms)
[2019-06-26 19:34:02.751 2064] New forward server: 208.67.220.220 (1/512)
[2019-06-26 19:34:02.757 2064] New forward server: 208.67.222.222 (2/512)
[2019-06-26 19:34:04.096 2064] Resizing "/FTL-strings" from 4096 to 8192
[2019-06-26 19:34:05.159 2064] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2019-06-26 19:34:05.160 2064] ----------------------------> FTL crashed! <----------------------------
[2019-06-26 19:34:05.160 2064] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2019-06-26 19:34:05.160 2064] Please report a bug at Issues · pi-hole/FTL · GitHub
[2019-06-26 19:34:05.160 2064] and include in your report already the following details:
[2019-06-26 19:34:05.160 2064] FTL has been running for 12 seconds
[2019-06-26 19:34:05.160 2064] FTL branch: master
[2019-06-26 19:34:05.160 2064] FTL version: v4.3
[2019-06-26 19:34:05.161 2064] FTL commit: 9e8273b
[2019-06-26 19:34:05.161 2064] FTL date: 2019-05-18 16:00:39 -0400
[2019-06-26 19:34:05.161 2064] FTL user: started as pihole, ended as pihole
[2019-06-26 19:34:05.161 2064] Received signal: Segmentation fault
[2019-06-26 19:34:05.161 2064] at address: 1996062720
[2019-06-26 19:34:05.162 2064] with code: SEGV_MAPERR (Address not mapped to object)
[2019-06-26 19:34:05.162 2064] Backtrace:
[2019-06-26 19:34:05.162 2064] B[0000]: /usr/bin/pihole-FTL(+0x1a25c) [0x46325c]
[2019-06-26 19:34:05.163 2064] B[0001]: /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0x76dd86c0]
[2019-06-26 19:34:05.163 2064] Thank you for helping us to improve our FTL engine!
[2019-06-26 19:34:05.163 2064] FTL terminated!

*** [ DIAGNOSING ]: contents of /dev/shm
-rw------- 1 pihole pihole 323584 Jun 26 19:33 /dev/shm/FTL-clients
-rw------- 1 pihole pihole 108 Jun 26 19:33 /dev/shm/FTL-counters
-rw------- 1 pihole pihole 65536 Jun 26 19:33 /dev/shm/FTL-domains
-rw------- 1 pihole pihole 12288 Jun 26 19:33 /dev/shm/FTL-forwarded
-rw------- 1 pihole pihole 28 Jun 26 19:33 /dev/shm/FTL-lock
-rw------- 1 pihole pihole 53248 Jun 26 19:33 /dev/shm/FTL-overTime
-rw------- 1 pihole pihole 393216 Jun 26 19:34 /dev/shm/FTL-queries
-rw------- 1 pihole pihole 12 Jun 26 19:33 /dev/shm/FTL-settings
-rw------- 1 pihole pihole 8192 Jun 26 19:34 /dev/shm/FTL-strings

*** [ DIAGNOSING ]: Locale
LANG=

*** [ DIAGNOSING ]: Pi-hole log
-rw-r--r-- 1 pihole pihole 1844058 Jun 26 19:34 /var/log/pihole.log
-----head of pihole.log------
Jun 26 19:30:13 dnsmasq[670]: query[A] spectrum.s3.amazonaws.com from 192.168.1.245
Jun 26 19:30:13 dnsmasq[670]: forwarded spectrum.s3.amazonaws.com to 127.0.0.1
Jun 26 19:30:13 dnsmasq[670]: forwarded spectrum.s3.amazonaws.com to 127.0.0.1
Jun 26 19:30:13 dnsmasq[670]: forwarded spectrum.s3.amazonaws.com to 127.0.0.1
Jun 26 19:30:13 dnsmasq[670]: reply error is SERVFAIL
Jun 26 19:30:13 dnsmasq[670]: query[A] spectrum.s3.amazonaws.com from 192.168.1.1
Jun 26 19:30:13 dnsmasq[670]: forwarded spectrum.s3.amazonaws.com to 127.0.0.1
Jun 26 19:30:13 dnsmasq[670]: forwarded spectrum.s3.amazonaws.com to 127.0.0.1
Jun 26 19:30:13 dnsmasq[670]: forwarded spectrum.s3.amazonaws.com to 127.0.0.1
Jun 26 19:30:13 dnsmasq[670]: reply error is SERVFAIL
Jun 26 19:30:13 dnsmasq[670]: query[A] spectrum.s3.amazonaws.com from 192.168.1.245
Jun 26 19:30:13 dnsmasq[670]: forwarded spectrum.s3.amazonaws.com to 127.0.0.1
Jun 26 19:30:13 dnsmasq[670]: forwarded spectrum.s3.amazonaws.com to 127.0.0.1
Jun 26 19:30:13 dnsmasq[670]: forwarded spectrum.s3.amazonaws.com to 127.0.0.1
Jun 26 19:30:13 dnsmasq[670]: reply error is SERVFAIL
Jun 26 19:30:13 dnsmasq[670]: query[A] spectrum.s3.amazonaws.com from 192.168.1.1
Jun 26 19:30:13 dnsmasq[670]: forwarded spectrum.s3.amazonaws.com to 127.0.0.1
Jun 26 19:30:13 dnsmasq[670]: forwarded spectrum.s3.amazonaws.com to 127.0.0.1
Jun 26 19:30:13 dnsmasq[670]: forwarded spectrum.s3.amazonaws.com to 127.0.0.1
Jun 26 19:30:13 dnsmasq[670]: reply error is SERVFAIL

[?] ** FINISHED DEBUGGING! **

* The debug log can be uploaded to tricorder.pi-hole.net for sharing with developers only.
* For more information, see: https://pi-hole.net/2016/11/07/crack-our-medical-tricorder-win-a-raspberry-pi-3/
* If available, we'll use openssl to upload the log, otherwise it will fall back to netcat.

[i] Debug script running in automated mode
* Using curl for transmission.
[?] There was an error uploading your debug log.

Please try again or contact the Pi-hole team for assistance.
A local copy of the debug log can be found at: /var/log/pihole_debug.log]_

DL6ER · June 26, 2019, 8:40pm

You're running an outdated version of Pi-hole. The bug you encountered had been fixed in FTL v4.3.1, which we released on May 25.

Please update your Pi-hole and you should never see this bug again. Sorry for the inconvenience and thanks for using the Pi-hole!

Obidose · June 26, 2019, 8:49pm

That's good to hear. I updated the Pi-hole as part of my troubleshooting process, so hopefully this is solved.

Thanks for the speedy reply.

system · July 17, 2019, 8:49pm

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.