Pi-hole Stopped Responding

Hi Team. Pi-hole stopped responding for me a moment ago.

Tricorder: https://tricorder.pi-hole.net/yzzzsism/

I notice the old restartdns command has gone so tried sudo pihole disable and enable and got the responses below.

I don't know how to restart the service otherwise, so rebooted and now it's working again.

Sorry it's vague, but if you need more details, please let me know.

Kind regards,
Paul.

>  $ sudo pihole disable
> curl: (3) URL using bad/illegal format or missing URL
> curl: (6) Could not resolve host: communicationsauth
> curl: (6) Could not resolve host: errorauth
> curl: (6) Could not resolve host: toauth
> curl: (6) Could not resolve host: timedauth
> curl: (3) URL using bad/illegal format or missing URL
> curl: (6) Could not resolve host: communicationsauth
> curl: (6) Could not resolve host: errorauth
> curl: (6) Could not resolve host: toauth
> curl: (6) Could not resolve host: timedauth
> curl: (3) URL using bad/illegal format or missing URL
> curl: (6) Could not resolve host: communicationsauth
> curl: (6) Could not resolve host: errorauth
> curl: (6) Could not resolve host: toauth
> curl: (6) Could not resolve host: timedauth
> curl: (3) URL using bad/illegal format or missing URL
> curl: (6) Could not resolve host: noauth
> curl: (6) Could not resolve host: serversauth
> curl: (6) Could not resolve host: couldauth
> curl: (6) Could not resolve host: beauth
> curl: (6) Could not resolve host: reachedauth
> curl: (6) Could not resolve host: reachedauth
> No response from FTL server. Please check connectivity
>  $ sudo pihole enable
> curl: (3) URL using bad/illegal format or missing URL
> curl: (6) Could not resolve host: communicationsauth
> curl: (6) Could not resolve host: errorauth
> curl: (6) Could not resolve host: toauth
> curl: (6) Could not resolve host: timedauth
> curl: (3) URL using bad/illegal format or missing URL
> curl: (6) Could not resolve host: communicationsauth
> curl: (6) Could not resolve host: errorauth
> curl: (6) Could not resolve host: toauth
> curl: (6) Could not resolve host: timedauth
> curl: (3) URL using bad/illegal format or missing URL
> curl: (6) Could not resolve host: communicationsauth
> curl: (6) Could not resolve host: errorauth
> curl: (6) Could not resolve host: toauth
> curl: (6) Could not resolve host: timedauth
> curl: (3) URL using bad/illegal format or missing URL
> curl: (6) Could not resolve host: noauth
> curl: (6) Could not resolve host: serversauth
> curl: (6) Could not resolve host: couldauth
> curl: (6) Could not resolve host: beauth
> curl: (6) Could not resolve host: reachedauth
> curl: (6) Could not resolve host: reachedauth
> No response from FTL server. Please check connectivity

This is a real crash.

2024-11-10 18:21:00.052 GMT [9334/T9336] INFO:                  Name: database
   2024-11-10 18:21:00.052 GMT [9334/T9336] INFO: Received signal: Segmentation fault
   2024-11-10 18:21:00.052 GMT [9334/T9336] INFO:      at address: 0x808
   2024-11-10 18:21:00.052 GMT [9334/T9336] INFO:      with code:  SEGV_MAPERR (Address not mapped to object)
   2024-11-10 18:21:00.052 GMT [9334/T9336] INFO: !!! INFO: pihole-FTL has not been compiled with glibc/backtrace support, not generating one !!!
   2024-11-10 18:21:00.052 GMT [9334/T9336] INFO: ------ Listing content of directory /dev/shm ------
   2024-11-10 18:21:00.053 GMT [9334/T9336] INFO: File Mode User:Group      Size  Filename
   2024-11-10 18:21:00.053 GMT [9334/T9336] INFO: rwxrwxrwx root:root       360   .
   2024-11-10 18:21:00.053 GMT [9334/T9336] INFO: rwxr-xr-x root:root         3K  ..
   2024-11-10 18:21:00.054 GMT [9334/T9336] INFO: rw------- pihole:pihole    56   FTL-lock
   2024-11-10 18:21:00.054 GMT [9334/T9336] INFO: rw------- pihole:pihole   328   FTL-counters
   2024-11-10 18:21:00.055 GMT [9334/T9336] INFO: rw------- pihole:pihole   136   FTL-settings
   2024-11-10 18:21:00.055 GMT [9334/T9336] INFO: rw------- pihole:pihole   123K  FTL-strings
   2024-11-10 18:21:00.056 GMT [9334/T9336] INFO: rw------- pihole:pihole    78K  FTL-domains
   2024-11-10 18:21:00.056 GMT [9334/T9336] INFO: rw------- pihole:pihole   340K  FTL-clients
   2024-11-10 18:21:00.057 GMT [9334/T9336] INFO: rw------- pihole:pihole    12K  FTL-upstreams
   2024-11-10 18:21:00.057 GMT [9334/T9336] INFO: rw------- pihole:pihole     2M  FTL-queries
   2024-11-10 18:21:00.058 GMT [9334/T9336] INFO: rw------- pihole:pihole     8K  FTL-overTime
   2024-11-10 18:21:00.058 GMT [9334/T9336] INFO: rw------- pihole:pihole   102K  FTL-dns-cache
   2024-11-10 18:21:00.059 GMT [9334/T9336] INFO: rw------- pihole:pihole     4K  FTL-per-client-regex
   2024-11-10 18:21:00.059 GMT [9334/T9336] INFO: rw------- pihole:pihole   560K  FTL-fifo-log
   2024-11-10 18:21:00.060 GMT [9334/T9336] INFO: rw------- pihole:pihole     4K  FTL-clients-lookup
   2024-11-10 18:21:00.060 GMT [9334/T9336] INFO: rw------- pihole:pihole    20K  FTL-domains-lookup
   2024-11-10 18:21:00.061 GMT [9334/T9336] INFO: rw------- pihole:pihole    20K  FTL-dns-cache-lookup
   2024-11-10 18:21:00.061 GMT [9334/T9336] INFO: rw------- pihole:pihole   786K  FTL-recycler
   2024-11-10 18:21:00.061 GMT [9334/T9336] INFO: ---------------------------------------------------
   2024-11-10 18:21:00.062 GMT [9334/T9336] INFO: Please also include some lines from above the !!!!!!!!! header.
   2024-11-10 18:21:00.062 GMT [9334/T9336] INFO: Thank you for helping us to improve our FTL engine!
   2024-11-10 18:21:00.062 GMT [9334/T9336] INFO: Waiting for threads to join
   2024-11-10 18:21:00.094 GMT [9334/T9337] INFO: Terminating GC thread
   2024-11-10 18:21:00.150 GMT [9334/T9339] INFO: Terminating timer thread
   2024-11-10 18:21:00.291 GMT [9334/T9338] INFO: Terminating resolver thread
   2024-11-10 18:21:02.062 GMT [9334/T9336] INFO: Thread database (0) is still busy, cancelling it.
   2024-11-10 18:21:02.062 GMT [9334M] ERROR: Error when obtaining outer SHM lock: Previous owner died
   2024-11-10 18:21:02.063 GMT [9334M] ERROR: Error when obtaining inner SHM lock: Previous owner died

Could you please post a few lines above the snippet I posted here from /var/log/pihole/FTL.log?

@DL6ER

Hi, thanks for looking, sure.

I know you asked for a few, but it looks like it's preceded with a load of errors relating to unbound:

2024-11-10 18:16:49.278 GMT [14029/F9334] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (Connection prematurely closed by remote server)
2024-11-10 18:16:49.955 GMT [14039/F9334] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (Connection prematurely closed by remote server)
2024-11-10 18:16:50.048 GMT [14040/F9334] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (Connection prematurely closed by remote server)
2024-11-10 18:16:50.050 GMT [14042/F9334] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (Connection prematurely closed by remote server)
2024-11-10 18:16:50.054 GMT [14041/F9334] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (Connection prematurely closed by remote server)
2024-11-10 18:16:50.054 GMT [14044/F9334] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (Connection prematurely closed by remote server)
2024-11-10 18:16:50.143 GMT [14047/F9334] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (Connection prematurely closed by remote server)
2024-11-10 18:17:38.907 GMT [14090/F9334] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (Connection prematurely closed by remote server)
2024-11-10 18:17:38.907 GMT [14091/F9334] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (Connection prematurely closed by remote server)
2024-11-10 18:18:32.493 GMT [14106/F9334] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (Connection prematurely closed by remote server)
2024-11-10 18:19:06.734 GMT [14130/F9334] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (Connection prematurely closed by remote server)
2024-11-10 18:19:06.848 GMT [14134/F9334] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (Connection prematurely closed by remote server)
2024-11-10 18:20:07.580 GMT [14149/F9334] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (Connection prematurely closed by remote server)
2024-11-10 18:21:00.049 GMT [9334/T9336] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-11-10 18:21:00.050 GMT [9334/T9336] INFO: ---------------------------->  FTL crashed!  <----------------------------
2024-11-10 18:21:00.050 GMT [9334/T9336] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-11-10 18:21:00.050 GMT [9334/T9336] INFO: Please report a bug at https://github.com/pi-hole/FTL/issues
2024-11-10 18:21:00.050 GMT [9334/T9336] INFO: and include in your report already the following details:
2024-11-10 18:21:00.050 GMT [9334/T9336] INFO: FTL has been running for 17616 seconds
2024-11-10 18:21:00.050 GMT [9334/T9336] INFO: FTL branch: development
2024-11-10 18:21:00.050 GMT [9334/T9336] INFO: FTL version: vDev-848367f
2024-11-10 18:21:00.051 GMT [9334/T9336] INFO: FTL commit: 848367fa
2024-11-10 18:21:00.051 GMT [9334/T9336] INFO: FTL date: 2024-11-08 19:13:07 +0100
2024-11-10 18:21:00.051 GMT [9334/T9336] INFO: FTL user: started as pihole, ended as pihole
2024-11-10 18:21:00.051 GMT [9334/T9336] INFO: Compiled for linux/arm/v6 (compiled on CI) using cc (Alpine 13.2.1_git20240309) 13.2.1 20240309
2024-11-10 18:21:00.051 GMT [9334/T9336] INFO: Process details: MID: 9334
2024-11-10 18:21:00.051 GMT [9334/T9336] INFO:                  PID: 9334
2024-11-10 18:21:00.052 GMT [9334/T9336] INFO:                  TID: 9336

This is a crash in the database component, we have seen something maybe related recently in FTL v54b4ad93 crashes sometimes (several times today) for unknown reasons · Issue #2112 · pi-hole/FTL · GitHub. Do you have any possibility to reproduce it? If so, I'd like to ask you to do the same attaching of the gdb debugger as in the other post behind my link above.

I'm afraid I don't know how to recreate it, but I'm happy to give debugging a try.

Is it safe to leave debugging on over a couple of days to see if it crashes again?

Sorry for the delay in responding. Sure, debugging can run as long as you like. I, for instance, have it on 24/7 on my own Pi-hole. Attaching the gdb debugger also doesn't affect performance in any noticeable way (except, maybe, if your machine is realllyy slow).