Been busy setting up my spare pogoplug pro v3 with a fresh debian bookworm install and a fresh pihole-v6 install. As you all know, for this to work I need to compile FTL myself. The plug was initially on ip 192.168.1.7 where I did the whole install and testing. All was fine.
This morning I also pulled the latest changes and did a fresh compile for FTL. All was still working fine.
Then changed the IP to 192.168.0.2 (which is the IP from the other plug; pogo01). Shutdown everything, disconnect the old plug, connect the new plug and did a cold boot. FTL did not start and crashed with the following error:
2024-12-22 15:29:21.712 CET [1261M] INFO: ########## FTL started on pogo02! ##########
2024-12-22 15:29:21.714 CET [1261M] INFO: FTL branch: development
2024-12-22 15:29:21.714 CET [1261M] INFO: FTL version: v5.25.2-2422-gc04fa584
2024-12-22 15:29:21.714 CET [1261M] INFO: FTL commit: c04fa584
2024-12-22 15:29:21.714 CET [1261M] INFO: FTL date: 2024-12-22 08:39:41 +0100
2024-12-22 15:29:21.715 CET [1261M] INFO: FTL user: pihole
2024-12-22 15:29:21.715 CET [1261M] INFO: Compiled for armv6l (compiled locally) using cc (Debian 12.2.0-14) 12.2.0
2024-12-22 15:29:21.737 CET [1261M] INFO: Wrote config file:
2024-12-22 15:29:21.737 CET [1261M] INFO: - 150 total entries
2024-12-22 15:29:21.738 CET [1261M] INFO: - 140 entries are default
2024-12-22 15:29:21.738 CET [1261M] INFO: - 10 entries are modified
2024-12-22 15:29:21.738 CET [1261M] INFO: - 0 entries are forced through environment
2024-12-22 15:29:21.752 CET [1261M] INFO: Parsed config file /etc/pihole/pihole.toml successfully
2024-12-22 15:29:21.799 CET [1261M] INFO: PID of FTL process: 1261
2024-12-22 15:29:21.808 CET [1261M] INFO: listening on 0.0.0.0 port 53
2024-12-22 15:29:21.808 CET [1261M] INFO: listening on :: port 53
2024-12-22 15:29:21.830 CET [1263M] INFO: PID of FTL process: 1263
2024-12-22 15:29:21.848 CET [1263M] INFO: Database version is 21
2024-12-22 15:29:21.852 CET [1263M] INFO: Database successfully initialized
2024-12-22 15:29:21.914 CET [1263M] INFO: Imported 38 queries from the on-disk database (it has 59 rows)
2024-12-22 15:29:21.915 CET [1263M] INFO: Parsing queries in database
2024-12-22 15:29:21.948 CET [1263M] INFO: Imported 38 queries from the long-term database
2024-12-22 15:29:21.948 CET [1263M] INFO: -> Total DNS queries: 38
2024-12-22 15:29:21.949 CET [1263M] INFO: -> Cached DNS queries: 25
2024-12-22 15:29:21.949 CET [1263M] INFO: -> Forwarded DNS queries: 8
2024-12-22 15:29:21.949 CET [1263M] INFO: -> Blocked DNS queries: 5
2024-12-22 15:29:21.950 CET [1263M] INFO: -> Unknown DNS queries: 0
2024-12-22 15:29:21.950 CET [1263M] INFO: -> Unique domains: 8
2024-12-22 15:29:21.950 CET [1263M] INFO: -> Unique clients: 3
2024-12-22 15:29:21.950 CET [1263M] INFO: -> DNS cache records: 2
2024-12-22 15:29:21.951 CET [1263M] INFO: -> Known forward destinations: 1
2024-12-22 15:29:23.276 CET [1263M] INFO: FTL is running as user pihole (UID 999)
2024-12-22 15:29:23.280 CET [1263M] INFO: Reading certificate from /etc/pihole/tls.pem ...
2024-12-22 15:29:23.282 CET [1263M] INFO: Using SSL/TLS certificate file /etc/pihole/tls.pem
2024-12-22 15:29:23.288 CET [1263M] INFO: Restored 1 API session from the database
2024-12-22 15:29:23.304 CET [1263M] INFO: Blocking status is enabled
2024-12-22 15:29:23.376 CET [1263/T1268] INFO: Compiled 0 allow and 0 deny regex for 4 clients in 2.3 msec
2024-12-22 15:29:24.730 CET [1263M] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-12-22 15:29:24.730 CET [1263M] INFO: ----------------------------> FTL crashed! <----------------------------
2024-12-22 15:29:24.731 CET [1263M] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-12-22 15:29:24.731 CET [1263M] INFO: Please report a bug at https://github.com/pi-hole/FTL/issues
2024-12-22 15:29:24.732 CET [1263M] INFO: and include in your report already the following details:
2024-12-22 15:29:24.732 CET [1263M] INFO: FTL has been running for 3 seconds
2024-12-22 15:29:24.733 CET [1263M] INFO: FTL branch: development
2024-12-22 15:29:24.733 CET [1263M] INFO: FTL version: v5.25.2-2422-gc04fa584
2024-12-22 15:29:24.734 CET [1263M] INFO: FTL commit: c04fa584
2024-12-22 15:29:24.734 CET [1263M] INFO: FTL date: 2024-12-22 08:39:41 +0100
2024-12-22 15:29:24.735 CET [1263M] INFO: FTL user: started as pihole, ended as pihole
2024-12-22 15:29:24.735 CET [1263M] INFO: Compiled for armv6l (compiled locally) using cc (Debian 12.2.0-14) 12.2.0
2024-12-22 15:29:24.738 CET [1263M] INFO: Process details: MID: 1263
2024-12-22 15:29:24.738 CET [1263M] INFO: PID: 1263
2024-12-22 15:29:24.739 CET [1263M] INFO: TID: 1263
2024-12-22 15:29:24.739 CET [1263M] INFO: Name: pihole-FTL
2024-12-22 15:29:24.739 CET [1263M] INFO: Received signal: Segmentation fault
2024-12-22 15:29:24.740 CET [1263M] INFO: at address: 0x30
2024-12-22 15:29:24.740 CET [1263M] INFO: with code: SEGV_MAPERR (Address not mapped to object)
2024-12-22 15:29:24.742 CET [1263M] INFO: Backtrace:
2024-12-22 15:29:24.746 CET [1263M] INFO: B[0000]: /usr/bin/pihole-FTL(generate_backtrace+0x40) [0x49240c]
2024-12-22 15:29:25.096 CET [1263M] INFO: L[0000]: /root/FTL/src/signals.c:107
2024-12-22 15:29:25.112 CET [1263M] INFO: B[0001]: /usr/bin/pihole-FTL(+0x60a4c) [0x492a4c]
2024-12-22 15:29:25.262 CET [1263M] INFO: L[0001]: /root/FTL/src/signals.c:251
2024-12-22 15:29:25.278 CET [1263M] INFO: B[0002]: /lib/arm-linux-gnueabi/libc.so.6(__default_rt_sa_restorer+0) [0xb6c94930]
2024-12-22 15:29:25.278 CET [1263M] INFO: B[0003]: /usr/bin/pihole-FTL(return_reply+0x4c4) [0x547364]
2024-12-22 15:29:25.459 CET [1263M] INFO: L[0003]: /root/FTL/src/dnsmasq/forward.c:1465
2024-12-22 15:29:25.481 CET [1263M] INFO: B[0004]: /usr/bin/pihole-FTL(reply_query+0x420) [0x5481c8]
2024-12-22 15:29:25.648 CET [1263M] INFO: L[0004]: /root/FTL/src/dnsmasq/forward.c:1307
2024-12-22 15:29:25.669 CET [1263M] INFO: B[0005]: /usr/bin/pihole-FTL(+0x101d94) [0x533d94]
2024-12-22 15:29:25.857 CET [1263M] INFO: L[0005]: /root/FTL/src/dnsmasq/dnsmasq.c:1863
2024-12-22 15:29:25.878 CET [1263M] INFO: B[0006]: /usr/bin/pihole-FTL(main_dnsmasq+0x1600) [0x536218]
2024-12-22 15:29:26.040 CET [1263M] INFO: L[0006]: /root/FTL/src/dnsmasq/dnsmasq.c:1280
2024-12-22 15:29:26.060 CET [1263M] INFO: B[0007]: /usr/bin/pihole-FTL(main+0x2b0) [0x4681b0]
2024-12-22 15:29:26.193 CET [1263M] INFO: L[0007]: /root/FTL/src/main.c:123
2024-12-22 15:29:26.209 CET [1263M] INFO: B[0008]: /lib/arm-linux-gnueabi/libc.so.6(+0x1e434) [0xb6c7c434]
2024-12-22 15:29:26.210 CET [1263M] INFO: B[0009]: /lib/arm-linux-gnueabi/libc.so.6(__libc_start_main+0x98) [0xb6c7c548]
2024-12-22 15:29:26.210 CET [1263M] INFO: ------ Listing content of directory /dev/shm ------
2024-12-22 15:29:26.210 CET [1263M] INFO: File Mode User:Group Size Filename
2024-12-22 15:29:26.212 CET [1263M] INFO: rwxrwxrwx root:root 380 .
2024-12-22 15:29:26.213 CET [1263M] INFO: rwxr-xr-x root:root 3K ..
2024-12-22 15:29:26.214 CET [1263M] INFO: rw------- pihole:pihole 786K FTL-recycler
2024-12-22 15:29:26.215 CET [1263M] INFO: rw------- pihole:pihole 4K FTL-dns-cache-lookup
2024-12-22 15:29:26.216 CET [1263M] INFO: rw------- pihole:pihole 4K FTL-domains-lookup
2024-12-22 15:29:26.217 CET [1263M] INFO: rw------- pihole:pihole 4K FTL-clients-lookup
2024-12-22 15:29:26.218 CET [1263M] INFO: rw------- pihole:pihole 560K FTL-fifo-log
2024-12-22 15:29:26.219 CET [1263M] INFO: rw------- pihole:pihole 4K FTL-per-client-regex
2024-12-22 15:29:26.220 CET [1263M] INFO: rw------- pihole:pihole 4K FTL-dns-cache
2024-12-22 15:29:26.221 CET [1263M] INFO: rw------- pihole:pihole 12K FTL-overTime
2024-12-22 15:29:26.222 CET [1263M] INFO: rw------- pihole:pihole 295K FTL-queries
2024-12-22 15:29:26.223 CET [1263M] INFO: rw------- pihole:pihole 12K FTL-upstreams
2024-12-22 15:29:26.224 CET [1263M] INFO: rw------- pihole:pihole 340K FTL-clients
2024-12-22 15:29:26.225 CET [1263M] INFO: rw------- pihole:pihole 4K FTL-domains
2024-12-22 15:29:26.226 CET [1263M] INFO: rw------- pihole:pihole 82K FTL-strings
2024-12-22 15:29:26.228 CET [1263M] INFO: rw------- pihole:pihole 136 FTL-settings
2024-12-22 15:29:26.229 CET [1263M] INFO: rw------- pihole:pihole 328 FTL-counters
2024-12-22 15:29:26.230 CET [1263M] INFO: rw------- pihole:pihole 56 FTL-lock
2024-12-22 15:29:26.230 CET [1263M] INFO: rw-r--r-- root:root 0 .tmpfs
2024-12-22 15:29:26.231 CET [1263M] INFO: ---------------------------------------------------
2024-12-22 15:29:26.231 CET [1263M] INFO: Please also include some lines from above the !!!!!!!!! header.
2024-12-22 15:29:26.231 CET [1263M] INFO: Thank you for helping us to improve our FTL engine!
2024-12-22 15:29:26.232 CET [1263M] INFO: Waiting for threads to join
2024-12-22 15:29:26.232 CET [1263M] INFO: Thread database (0) is idle, terminating it.
2024-12-22 15:29:26.233 CET [1263M] INFO: Thread housekeeper (1) is idle, terminating it.
2024-12-22 15:29:26.251 CET [1263/T1271] INFO: Terminating timer thread
2024-12-22 15:29:27.266 CET [1263/T1270] ERROR: Cannot receive UDP DNS reply: Timeout - no response from upstream DNS server
2024-12-22 15:29:27.266 CET [1263/T1270] INFO: Tried to resolve PTR "1.0.0.127.in-addr.arpa" on 127.0.0.1#53 (UDP)
2024-12-22 15:29:27.387 CET [1263/T1267] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-12-22 15:29:27.388 CET [1263/T1267] INFO: Time offset: 1.948163e+05 ms (excluded 1 outliers)
2024-12-22 15:29:27.388 CET [1263/T1267] INFO: Round-trip delay: 1.478938e+01 ms (excluded 1 outliers)
2024-12-22 15:32:42.212 CET [1263M] INFO: Thread dns-client (2) is still busy, cancelling it.
2024-12-22 15:32:42.000 CET [1263/T1267] INFO: Terminating NTP thread
2024-12-22 15:32:42.001 CET [1263M] INFO: All threads joined
2024-12-22 15:32:42.009 CET [1263M] INFO: Stored 1 API session in the database
2024-12-22 15:32:43.128 CET [1263M] INFO: ########## FTL terminated after 3m 21s (code 1)! ##########
Tried to restart it /etc/init.d/pihole-FTL start
... same problem.
Next, changed IP to 192.168.0.3 and re-enabled the old DNS (so people on my network can start surfing again). And now it boots fine ... go figure. No way to reproduce this ... after several restarts and cold boot, still working fine. Please note, this machine is not being queried by anything, so that might explain why it is still running and not crashing.
Thought this had something to do with IP address changes ... but then I started my vm on proxmox, also running pihole v6 and this one shows the same crash after running for 20 minutes or so. This was also after doing a pihole -up
.
2024-12-22 15:59:35.665 CET [1896M] INFO: ########## FTL started on dns01! ##########
2024-12-22 15:59:35.665 CET [1896M] INFO: FTL branch: development
2024-12-22 15:59:35.665 CET [1896M] INFO: FTL version: vDev-c04fa58
2024-12-22 15:59:35.666 CET [1896M] INFO: FTL commit: c04fa584
2024-12-22 15:59:35.666 CET [1896M] INFO: FTL date: 2024-12-22 08:39:41 +0100
2024-12-22 15:59:35.666 CET [1896M] INFO: FTL user: pihole
2024-12-22 15:59:35.666 CET [1896M] INFO: Compiled for linux/amd64 (compiled on CI) using cc (Alpine 14.2.0) 14.2.0
2024-12-22 15:59:35.667 CET [1896M] INFO: Wrote config file:
2024-12-22 15:59:35.667 CET [1896M] INFO: - 150 total entries
2024-12-22 15:59:35.667 CET [1896M] INFO: - 143 entries are default
2024-12-22 15:59:35.667 CET [1896M] INFO: - 7 entries are modified
2024-12-22 15:59:35.667 CET [1896M] INFO: - 0 entries are forced through environment
2024-12-22 15:59:35.668 CET [1896M] INFO: Parsed config file /etc/pihole/pihole.toml successfully
2024-12-22 15:59:35.670 CET [1896M] INFO: PID of FTL process: 1896
2024-12-22 15:59:35.670 CET [1896M] INFO: listening on 0.0.0.0 port 53
2024-12-22 15:59:35.670 CET [1896M] INFO: listening on :: port 53
2024-12-22 15:59:35.671 CET [1896M] INFO: PID of FTL process: 1896
2024-12-22 15:59:35.671 CET [1896M] INFO: Database version is 21
2024-12-22 15:59:35.671 CET [1896M] INFO: Database successfully initialized
2024-12-22 15:59:35.699 CET [1896M] INFO: Imported 2324 queries from the on-disk database (it has 414808 rows)
2024-12-22 15:59:35.699 CET [1896M] INFO: Parsing queries in database
2024-12-22 15:59:35.703 CET [1896M] INFO: Imported 2324 queries from the long-term database
2024-12-22 15:59:35.703 CET [1896M] INFO: -> Total DNS queries: 2324
2024-12-22 15:59:35.703 CET [1896M] INFO: -> Cached DNS queries: 1539
2024-12-22 15:59:35.703 CET [1896M] INFO: -> Forwarded DNS queries: 446
2024-12-22 15:59:35.703 CET [1896M] INFO: -> Blocked DNS queries: 323
2024-12-22 15:59:35.703 CET [1896M] INFO: -> Unknown DNS queries: 6
2024-12-22 15:59:35.703 CET [1896M] INFO: -> Unique domains: 368
2024-12-22 15:59:35.703 CET [1896M] INFO: -> Unique clients: 2
2024-12-22 15:59:35.703 CET [1896M] INFO: -> DNS cache records: 50
2024-12-22 15:59:35.703 CET [1896M] INFO: -> Known forward destinations: 1
2024-12-22 15:59:35.768 CET [1896M] INFO: FTL is running as user pihole (UID 999)
2024-12-22 15:59:35.768 CET [1896M] INFO: Reading certificate from /etc/pihole/tls.pem ...
2024-12-22 15:59:35.768 CET [1896M] INFO: Using SSL/TLS certificate file /etc/pihole/tls.pem
2024-12-22 15:59:35.769 CET [1896M] INFO: Restored 0 API sessions from the database
2024-12-22 15:59:35.771 CET [1896M] INFO: Blocking status is enabled
2024-12-22 15:59:35.869 CET [1896/T1898] INFO: Compiled 0 allow and 0 deny regex for 2 clients in 0.0 msec
2024-12-22 15:59:36.323 CET [1896M] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-12-22 15:59:36.323 CET [1896M] INFO: ----------------------------> FTL crashed! <----------------------------
2024-12-22 15:59:36.323 CET [1896M] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-12-22 15:59:36.323 CET [1896M] INFO: Please report a bug at https://github.com/pi-hole/FTL/issues
2024-12-22 15:59:36.323 CET [1896M] INFO: and include in your report already the following details:
2024-12-22 15:59:36.323 CET [1896M] INFO: FTL has been running for 1 seconds
2024-12-22 15:59:36.323 CET [1896M] INFO: FTL branch: development
2024-12-22 15:59:36.323 CET [1896M] INFO: FTL version: vDev-c04fa58
2024-12-22 15:59:36.323 CET [1896M] INFO: FTL commit: c04fa584
2024-12-22 15:59:36.323 CET [1896M] INFO: FTL date: 2024-12-22 08:39:41 +0100
2024-12-22 15:59:36.323 CET [1896M] INFO: FTL user: started as pihole, ended as pihole
2024-12-22 15:59:36.323 CET [1896M] INFO: Compiled for linux/amd64 (compiled on CI) using cc (Alpine 14.2.0) 14.2.0
2024-12-22 15:59:36.323 CET [1896M] INFO: Process details: MID: 1896
2024-12-22 15:59:36.323 CET [1896M] INFO: PID: 1896
2024-12-22 15:59:36.323 CET [1896M] INFO: TID: 1896
2024-12-22 15:59:36.323 CET [1896M] INFO: Name: pihole-FTL
2024-12-22 15:59:36.323 CET [1896M] INFO: Received signal: Segmentation fault
2024-12-22 15:59:36.323 CET [1896M] INFO: at address: 0x34
2024-12-22 15:59:36.323 CET [1896M] INFO: with code: SEGV_MAPERR (Address not mapped to object)
2024-12-22 15:59:36.323 CET [1896M] INFO: !!! INFO: pihole-FTL has not been compiled with glibc/backtrace support, not generating one !!!
2024-12-22 15:59:36.323 CET [1896M] INFO: ------ Listing content of directory /dev/shm ------
2024-12-22 15:59:36.323 CET [1896M] INFO: File Mode User:Group Size Filename
2024-12-22 15:59:36.323 CET [1896M] INFO: rwxrwxrwx root:root 360 .
2024-12-22 15:59:36.323 CET [1896M] INFO: rwxr-xr-x root:root 3K ..
2024-12-22 15:59:36.323 CET [1896M] INFO: rw------- pihole:pihole 786K FTL-recycler
2024-12-22 15:59:36.323 CET [1896M] INFO: rw------- pihole:pihole 4K FTL-dns-cache-lookup
2024-12-22 15:59:36.323 CET [1896M] INFO: rw------- pihole:pihole 4K FTL-domains-lookup
2024-12-22 15:59:36.323 CET [1896M] INFO: rw------- pihole:pihole 4K FTL-clients-lookup
2024-12-22 15:59:36.323 CET [1896M] INFO: rw------- pihole:pihole 569K FTL-fifo-log
2024-12-22 15:59:36.323 CET [1896M] INFO: rw------- pihole:pihole 4K FTL-per-client-regex
2024-12-22 15:59:36.323 CET [1896M] INFO: rw------- pihole:pihole 20K FTL-dns-cache
2024-12-22 15:59:36.323 CET [1896M] INFO: rw------- pihole:pihole 8K FTL-overTime
2024-12-22 15:59:36.323 CET [1896M] INFO: rw------- pihole:pihole 295K FTL-queries
2024-12-22 15:59:36.323 CET [1896M] INFO: rw------- pihole:pihole 29K FTL-upstreams
2024-12-22 15:59:36.323 CET [1896M] INFO: rw------- pihole:pihole 348K FTL-clients
2024-12-22 15:59:36.323 CET [1896M] INFO: rw------- pihole:pihole 12K FTL-domains
2024-12-22 15:59:36.323 CET [1896M] INFO: rw------- pihole:pihole 82K FTL-strings
2024-12-22 15:59:36.323 CET [1896M] INFO: rw------- pihole:pihole 144 FTL-settings
2024-12-22 15:59:36.324 CET [1896M] INFO: rw------- pihole:pihole 328 FTL-counters
2024-12-22 15:59:36.324 CET [1896M] INFO: rw------- pihole:pihole 88 FTL-lock
2024-12-22 15:59:36.324 CET [1896M] INFO: ---------------------------------------------------
2024-12-22 15:59:36.324 CET [1896M] INFO: Please also include some lines from above the !!!!!!!!! header.
2024-12-22 15:59:36.324 CET [1896M] INFO: Thank you for helping us to improve our FTL engine!
2024-12-22 15:59:36.324 CET [1896M] INFO: Waiting for threads to join
2024-12-22 15:59:36.324 CET [1896M] INFO: Thread database (0) is idle, terminating it.
2024-12-22 15:59:36.324 CET [1896M] INFO: Thread housekeeper (1) is idle, terminating it.
2024-12-22 15:59:36.324 CET [1896M] INFO: Thread dns-client (2) is idle, terminating it.
2024-12-22 15:59:36.324 CET [1896M] INFO: Thread timer (3) is idle, terminating it.
2024-12-22 15:59:38.324 CET [1896M] INFO: Thread ntp-client (4) is still busy, cancelling it.
2024-12-22 15:59:38.324 CET [1896M] INFO: All threads joined
2024-12-22 15:59:38.324 CET [1896M] INFO: Stored 0 API sessions in the database
2024-12-22 15:59:38.773 CET [1896M] INFO: ########## FTL terminated after 3s 107ms (code 1)! ##########
So something got merged this morning that might be causing this?