Hi,
As per topic FTL Started Crashing I have opened a new topic.
Network:
2 Raspberry Pi 4B with 4 MB RAM and a 64 MB SDX Memory Card, using same router from 2 different LAN Ports (eth1 and eth eth2).
Pi4-1 = Pi-hole1 on eth1 (wired LAN connections)
Pi4-2 = Pi-hole2 on eth2 (wireless connections)
When both are using the DHCP server from the Router all OK.
Pi-hole2 was last week setup to be the DHCP sever, which after less than an hour results in a crash.
This appeared to be related to the said FTL Started Crashing issue.
After latest update which included the FTL fix, Pi-hole2 was again configured as DHCP server, which again resulted in a crash.
To sort of eliminate it could be the Pi causing this the Pi were swapped, but using same memory card (both memory cards are new). The only other application running on Pi4-2 besides Pi-hole is the unifi network controller.
Whole network dies, no access to Pi4-2 via SSH, no access Pi-hole2 via web interface.
Router re-configured as DHCP server and Pi4-2 rebooted.
The command pihole -d when completed could not be uploaded:
[?] Would you like to upload the log? [y/N] y
* Using curl for transmission.
* curl failed, contact Pi-hole support for assistance.
* Error message: curl: (6) Could not resolve host: tricorder.pi-hole.net
[✗] There was an error uploading your debug log.
* Please try again or contact the Pi-hole team for assistance.
* A local copy of the debug log can be found at: /var/log/pihole/pihole_debug.log
To avoid overwriting I copied the debug file, zipped and attached here - just in case it had something to tell. pihole_debug.zip (11.0 KB)
After Pi4-2 backup I run the debug command again.
https://tricorder.pi-hole.net/198D9loe/
Looking at the FTL.log from within Pi-hole it showed some connecting errors.
Looking at pihole.log from within Pi-hole it showed network unreachable.
Tried with other Memory card, crashed after a few hours.
I got the Pi back to life by removing and then inserting the LAN connection.
I also got a FTL error:
2024-04-22 19:59:59.278 WARNING WARNING in dnsmasq core: DHCP packet received on eth0 which has no address
2024-04-22 19:59:59.466 WARNING WARNING in dnsmasq core: DHCP packet received on eth0 which has no address
2024-04-22 20:00:00.258 INFO !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-04-22 20:00:00.258 INFO ----------------------------> FTL crashed! <----------------------------
2024-04-22 20:00:00.258 INFO !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-04-22 20:00:00.258 INFO Please report a bug at https://github.com/pi-hole/FTL/issues
2024-04-22 20:00:00.258 INFO and include in your report already the following details:
2024-04-22 20:00:00.258 INFO FTL has been running for 9094 seconds
2024-04-22 20:00:00.259 INFO FTL branch: development-v6
2024-04-22 20:00:00.259 INFO FTL version: vDev-1b71a44
2024-04-22 20:00:00.259 INFO FTL commit: 1b71a440
2024-04-22 20:00:00.259 INFO FTL date: 2024-04-21 10:28:14 +0200
2024-04-22 20:00:00.259 INFO FTL user: started as pihole, ended as pihole
2024-04-22 20:00:00.259 INFO Compiled for linux/arm64/v8 (compiled on CI) using cc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924
2024-04-22 20:00:00.259 INFO Process details: MID: 765
2024-04-22 20:00:00.260 INFO PID: 765
2024-04-22 20:00:00.260 INFO TID: 6437
2024-04-22 20:00:00.260 INFO Name: DNS client
2024-04-22 20:00:00.260 INFO Received signal: Segmentation fault
2024-04-22 20:00:00.260 INFO at address: 0x80a9b0844f
2024-04-22 20:00:00.260 WARNING Connection error (1.1.1.1#53): failed to send UDP request (Network unreachable)
2024-04-22 20:00:00.261 INFO with code: SEGV_MAPERR (Address not mapped to object)
2024-04-22 20:00:00.261 INFO !!! INFO: pihole-FTL has not been compiled with glibc/backtrace support, not generating one !!!
2024-04-22 20:00:00.261 INFO ------ Listing content of directory /dev/shm ------
2024-04-22 20:00:00.261 INFO File Mode User:Group Size Filename
2024-04-22 20:00:00.262 INFO rwxrwxrwx root:root 280 .
2024-04-22 20:00:00.262 INFO rwxr-xr-x root:root 4K ..
2024-04-22 20:00:00.262 INFO rw------- pihole:pihole 88 FTL-lock
2024-04-22 20:00:00.263 INFO rw------- pihole:pihole 292 FTL-counters
2024-04-22 20:00:00.263 INFO rw------- pihole:pihole 16 FTL-settings
2024-04-22 20:00:00.263 INFO rw------- pihole:pihole 82K FTL-strings
2024-04-22 20:00:00.263 INFO rw------- pihole:pihole 37K FTL-domains
2024-04-22 20:00:00.264 INFO rw------- pihole:pihole 86K FTL-clients
2024-04-22 20:00:00.264 INFO rw------- pihole:pihole 29K FTL-upstreams
2024-04-22 20:00:00.264 INFO rw------- pihole:pihole 590K FTL-queries
2024-04-22 20:00:00.265 INFO rw------- pihole:pihole 8K FTL-overTime
2024-04-22 20:00:00.265 INFO rw------- pihole:pihole 49K FTL-dns-cache
2024-04-22 20:00:00.265 INFO rw------- pihole:pihole 4K FTL-per-client-regex
2024-04-22 20:00:00.265 INFO rw------- pihole:pihole 560K FTL-fifo-log
2024-04-22 20:00:00.265 INFO ---------------------------------------------------
2024-04-22 20:00:00.266 INFO Please also include some lines from above the !!!!!!!!! header.
2024-04-22 20:00:00.266 INFO Thank you for helping us to improve our FTL engine!
2024-04-22 20:00:00.266 INFO Waiting for threads to join
2024-04-22 20:00:00.266 INFO Thread database (0) is idle, terminating it.
2024-04-22 20:00:00.266 INFO Thread housekeeper (1) is idle, terminating it.
Update 2
Decided to try using the Pi4-1 / Pi-hole 2 as DHCP server, normal the Pi-hole2 is serving as DNS and the Router as the DHCP server.
Around an hour after it crashed.
From the pihole.log I see a few messages like this Apr 23 11:38:03 dnsmasq-dhcp[1207]: DHCP packet received on eth0 which has no address then as time passes the number increases.
Not sure if related at all.
FTL.log shows similar messages
2024-04-23 10:41:44.072 WARNING Could not fopen("/etc/pihole/setupVars.conf", "r") in read_setupVarsconf() (/app/src/config/setupVars.c:641): No such file or directory
2024-04-23 11:21:24.173 WARNING WARNING in dnsmasq core: DHCP packet received on eth0 which has no address
2024-04-23 11:22:35.206 WARNING Connection error (1.1.1.1#53): failed to send UDP request (Network unreachable)
2024-04-23 11:26:49.155 WARNING WARNING in dnsmasq core: DHCP packet received on eth0 which has no address
2024-04-23 11:26:52.127 WARNING WARNING in dnsmasq core: DHCP packet received on eth0 which has no address
Sorry for the extra work, but it seems we need to do further debugging to see what is going on exactly. Please follow the instructions here: gdb - Pi-hole documentation
Once the crash happens again, the debugger will halt the process in the faulty state and you can follow the last step (3.). This will hopefully provide enough information to exactly identify (and subsequently fix) this additional bug.
@DL6ER
Thanks, I shall do my best to follow the guide.
My first, so not too sure if the below is OK?
sudo gdb -p $(pidof pihole-FTL)
GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 848
[New LWP 851]
[New LWP 852]
[New LWP 853]
[New LWP 854]
[New LWP 855]
[New LWP 856]
[New LWP 857]
[New LWP 858]
[New LWP 859]
[New LWP 860]
[New LWP 861]
[New LWP 862]
[New LWP 863]
[New LWP 864]
__syscall_cp_asm () at src/thread/aarch64/syscall_cp.s:28
28 src/thread/aarch64/syscall_cp.s: No such file or directory.
(gdb)
Bit strange, because as soon as I enable the DHCP server on Pi-hole things go bad. This even before I have disabled the DHCP server on the router.
I lose access to Pi-hole web and access from devices to the internet grinds to a halt.
To come back I have to reboot the Pi / Pi-hole from the terminal.
Below the output from the debugging. Hope I done it right.
admin@Pi4-2b:~ $ screen
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 968
[New LWP 1106]
[New LWP 1107]
[New LWP 1108]
[New LWP 1109]
[New LWP 1110]
[New LWP 1111]
[New LWP 1112]
--Type <RET> for more, q to quit, c to continue without paging--c
[New LWP 1113]
[New LWP 1114]
[New LWP 1115]
[New LWP 1116]
[New LWP 1117]
[New LWP 1118]
[New LWP 1119]
__syscall_cp_asm () at src/thread/aarch64/syscall_cp.s:28
28 src/thread/aarch64/syscall_cp.s: No such file or directory.
(gdb) continiue
Undefined command: "continiue". Try "help".
(gdb) continue
Continuing.
Thread 9 "civetweb-worker" received signal SIGPIPE, Broken pipe.
Thread 13 "civetweb-worker" received signal SIGPIPE, Broken pipe.
Thread 14 "civetweb-worker" received signal SIGPIPE, Broken pipe.
Thread 11 "civetweb-worker" received signal SIGPIPE, Broken pipe.
Thread 8 "civetweb-worker" received signal SIGPIPE, Broken pipe.
Thread 7 "civetweb-worker" received signal SIGPIPE, Broken pipe.
Thread 12 "civetweb-worker" received signal SIGPIPE, Broken pipe.
Thread 10 "civetweb-worker" received signal SIGPIPE, Broken pipe.
Thread 13 "civetweb-worker" received signal SIGPIPE, Broken pipe.
Thread 9 "civetweb-worker" received signal SIGPIPE, Broken pipe.
Thread 8 "civetweb-worker" received signal SIGPIPE, Broken pipe.
Thread 11 "civetweb-worker" received signal SIGPIPE, Broken pipe.
[Detaching after vfork from child process 1233]
[Detaching after vfork from child process 1235]
(gdb) [Detaching after vfork from child process 1243]
[Detaching after vfork from child process 1245]
[Detaching after vfork from child process 1251]
[Detaching after vfork from child process 1253]
[Detaching after fork from child process 1255]
Thread 1 "pihole-FTL" received signal SIGTERM, Terminated.
Thread 1 "pihole-FTL" received signal SIG41, Real-time event 41.
__restore_sigs (set=set@entry=0x7fe5846b30) at ./arch/aarch64/syscall_arch.h:48
48 ./arch/aarch64/syscall_arch.h: No such file or directory.
(gdb)
(gdb) backtrace
#0 __restore_sigs (set=set@entry=0x7fe5846b30) at ./arch/aarch64/syscall_arch.h:48
#1 0x0000000000762fb0 in raise (sig=<optimized out>) at src/signal/raise.c:11
#2 0x000000000041eaf4 in SIGTERM_handler (unused=<optimized out>, signum=<optimized out>, si=<optimized out>) at /app/src/signals.c:399
#3 <signal handler called>
#4 __cp_end () at src/thread/aarch64/syscall_cp.s:30
#5 0x000000000076a5b0 in __syscall_cp_c (nr=73, u=<optimized out>, v=<optimized out>, w=<optimized out>, x=<optimized out>, y=<optimized out>,
z=<optimized out>) at src/thread/pthread_cancel.c:33
#6 0x0000000000762d48 in poll (fds=<optimized out>, n=<optimized out>, timeout=timeout@entry=-1) at src/select/poll.c:16
#7 0x00000000004cc2e4 in poll (__s=-1, __n=<optimized out>, __f=<optimized out>) at /usr/include/fortify/poll.h:40
#8 0x00000000004a32dc in main_dnsmasq (argc=<optimized out>, argv=<optimized out>) at /app/src/dnsmasq/dnsmasq.c:1186
#9 0x00000000004003c4 in main (argc=<optimized out>, argv=0x7fe58483a8) at /app/src/main.c:126
(gdb)
This means something has asked pihole-FTL to shut down. This may have been an external call to
pihole restartdns
or maybe
sudo service pihole-FTL restart
or by you changing some settings which requires a restart - but not a crash. When FTL is asked to stop like this from external, the debugger cannot do anything about it.
You cannot change any settings once the debugger is attached. Sorry, I have not been clear about this. Please only attach the debugger once you are in the situation where you are confident that the crash will happen sooner or later.
Hi,
Not sure why the above happened.
I tried to get some date collected today, but after two tries I see the same. After a while the LAN port freezes, at first try I had to reboot the Pi at the second I removed the LAN cable to the PI and changed the router to be DHCP server (kept DHCP enabled on Pi-hole).
I could get back into the screen session, but didn't really show anything, when back in the screen session the command "backtrace" didn't show anything.
Presume - as per your last message - this mean it is not the FTL that crashes...
Not sure if helps - anyway here the Pi-hole debug log https://tricorder.pi-hole.net/p7ByzUMF/
When the DHCP server is enabled on Pi-hole somehow the network connection is broken/not reachable.
Steen
[Detaching after vfork from child process 1679]
Thread 12 "civetweb-worker" received signal SIGPIPE, Broken pipe.
[Detaching after vfork from child process 1692]
[Detaching after vfork from child process 1694]
Thread 11 "civetweb-worker" received signal SIGPIPE, Broken pipe.
Thread 13 "civetweb-worker" received signal SIGPIPE, Broken pipe.
Thread 14 "civetweb-worker" received signal SIGPIPE, Broken pipe.
[Detaching after vfork from child process 1712]
[Detaching after vfork from child process 1714]
[Detaching after vfork from child process 1724]
[Detaching after vfork from child process 1726]
[Detaching after vfork from child process 1736]
[Detaching after vfork from child process 1738]
[Detaching after vfork from child process 1746]
[Detaching after vfork from child process 1748]
[Detaching after vfork from child process 1758]
[Detaching after vfork from child process 1760]
[Detaching after vfork from child process 1770]
[Detaching after vfork from child process 1772]
Thread 9 "civetweb-worker" received signal SIGPIPE, Broken pipe.
[Detaching after fork from child process 1823]
[Detaching after vfork from child process 1826]
[Detaching after vfork from child process 1828]
Thread 8 "civetweb-worker" received signal SIGPIPE, Broken pipe.
Thread 10 "civetweb-worker" received signal SIGPIPE, Broken pipe.
[Detaching after vfork from child process 1842]
[Detaching after vfork from child process 1844]
[Detaching after vfork from child process 1849]
[Detaching after vfork from child process 1851]
[Detaching after vfork from child process 1892]
[Detaching after vfork from child process 1894]
[Detaching after vfork from child process 1902]
[Detaching after vfork from child process 1904]
[Detaching after vfork from child process 1911]
[Detaching after vfork from child process 1913]
2024-04-24 09:57:49.305 INFO Compiled 7 allow and 20 deny regex for 38 clients in 174.6 msec
2024-04-24 09:59:23.510 WARNING API: Not found (/api/hardware)
2024-04-24 10:58:36.299 WARNING WARNING in dnsmasq core: DHCP packet received on eth0 which has no address
2024-04-24 10:58:37.299 WARNING WARNING in dnsmasq core: DHCP packet received on eth0 which has no address
2024-04-24 10:58:39.389 WARNING WARNING in dnsmasq core: DHCP packet received on eth0 which has no address
2024-04-24 10:58:41.307 WARNING WARNING in dnsmasq core: DHCP packet received on eth0 which has no address
2024-04-24 10:58:42.342 WARNING WARNING in dnsmasq core: DHCP packet received on eth0 which has no address
2024-04-24 10:58:44.438 WARNING WARNING in dnsmasq core: DHCP packet received on eth0 which has no address
2024-04-24 10:58:48.775 WARNING WARNING in dnsmasq core: DHCP packet received on eth0 which has no address
2024-04-24 10:58:57.515 WARNING WARNING in dnsmasq core: DHCP packet received on eth0 which has no address
2024-04-24 10:58:58.532 WARNING WARNING in dnsmasq core: DHCP packet received on eth0 which has no address
2024-04-24 10:58:58.781 WARNING Connection error (1.1.1.1#53): failed to send UDP request (Network unreachable)
and would like to ask you to double tripple-check whether your Pi-hole really has a static IP address.
This (and the other bits you wrote) make me think that your Pi-hole is still using DHCP Then, once you switch off your router's DHCP server, your Pi-hole's DHCP lease expires at some point after which the Ethernet port is left in an unconfigured (= unreachable) state. At this point, this seems to by by far the most reasonable (and perfectly matching!) explanation for everything we're seeing here.
I believe my Pi-hole got a static IP address, not sure if the below commands tells if dynamic or static?
For the last command I seen there is away to set-up manually (https://www.abelectronics.co.uk/kb/article/31/set-a-static-ip-address-on-raspberry-pi-os-bookworm , but never done before.
admin@Test2:~ $ sudo nmcli -f ipv4.method con show "Wired connection 1"
ipv4.method: auto
I am clear with the below first two commands:
sudo nmcli c mod "Wired connection 1" ipv4.addresses 192.168.2.254/24 ipv4.method manual
sudo nmcli con mod "Wired connection 1" ipv4.gateway 192.168.2.1
Not sure about the third when it comes to the DNS. Would it be A) Pi-hole IP address, or B) An external DNS server e.g. 1.1.1.1, or The Router IP address? I used B, but not sure if correct?!
sudo nmcli con mod "Wired connection 1" ipv4.dns 1.1.1.1
sudo nmcli -f ipv4.method con show "Wired connection 1"
It depends on what you want and expect. Go for your Pi-hole if you want filtering and logging also on this device. Go for an external provider of you don't want this but want to ensure local DNS resolution still works on the rare case if a crash.
I have mine set to use itself as DNS provider but tastes differ.
