FTL Started Crashing

paul-da · April 20, 2024, 3:49pm

I noticed this maybe a day or two ago, and it's either got worse, or the weekend has made me notice it more. Although it's occurring regularly, I don't know what triggers it. Sorry it's a vague. I hope the logs below are some use, Let me know there's anything I can do to test/help.
Thanks.
Paul.

There's a note in /var/log/pihole-FTL.log about 'Connection error (127.0.0.1#5335): failed to send TCP' - I use unbound. But changing the forwarder to Quad9 doesn't solve it.

I tried uninstalling (removing the pihole folder from /etc and my home folder) and re-installing (install v5, update to v6). to no effect.

https://tricorder.pi-hole.net/OBdUJW2e/

https://tricorder.pi-hole.net/rKkK8XoV/

2024-04-20 16:23:33.459 [4914/F4797] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (No error information)
2024-04-20 16:23:33.462 [4914/F4797] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-04-20 16:23:33.462 [4914/F4797] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 16:23:33.462 [4914/F4797] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-04-20 16:23:33.462 [4914/F4797] INFO: Please report a bug at Issues · pi-hole/FTL · GitHub
2024-04-20 16:23:33.462 [4914/F4797] INFO: and include in your report already the following details:
2024-04-20 16:23:33.462 [4914/F4797] INFO: FTL has been running for 224 seconds
2024-04-20 16:23:33.463 [4914/F4797] INFO: FTL branch: development-v6
2024-04-20 16:23:33.463 [4914/F4797] INFO: FTL version: vDev-4734e01
2024-04-20 16:23:33.463 [4914/F4797] INFO: FTL commit: 4734e01b
2024-04-20 16:23:33.463 [4914/F4797] INFO: FTL date: 2024-04-18 21:28:52 +0200
2024-04-20 16:23:33.463 [4914/F4797] INFO: FTL user: started as pihole, ended as pihole
2024-04-20 16:23:33.463 [4914/F4797] INFO: Compiled for linux/arm/v6 (compiled on CI) using cc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924
2024-04-20 16:23:33.463 [4914/F4797] INFO: Process details: MID: 4797
2024-04-20 16:23:33.464 [4914/F4797] INFO: PID: 4914
2024-04-20 16:23:33.464 [4914/F4797] INFO: TID: 4914
2024-04-20 16:23:33.464 [4914/F4797] INFO: Name: pihole-FTL
2024-04-20 16:23:33.464 [4914/F4797] INFO: Received signal: Segmentation fault
2024-04-20 16:23:33.464 [4914/F4797] INFO: at address: 0x2
2024-04-20 16:23:33.464 [4914/F4797] INFO: with code: SEGV_MAPERR (Address not mapped to object)
2024-04-20 16:23:33.464 [4914/F4797] INFO: !!! INFO: pihole-FTL has not been compiled with glibc/backtrace support, not generating one !!!
2024-04-20 16:23:33.464 [4914/F4797] INFO: ------ Listing content of directory /dev/shm ------
2024-04-20 16:23:33.464 [4914/F4797] INFO: File Mode User:Group Size Filename
2024-04-20 16:23:33.465 [4914/F4797] INFO: rwxrwxrwx root:root 280 .
2024-04-20 16:23:33.465 [4914/F4797] INFO: rwxr-xr-x root:root 3K ..
2024-04-20 16:23:33.465 [4914/F4797] INFO: rw------- pihole:pihole 56 FTL-lock
2024-04-20 16:23:33.466 [4914/F4797] INFO: rw------- pihole:pihole 292 FTL-counters
2024-04-20 16:23:33.466 [4914/F4797] INFO: rw------- pihole:pihole 16 FTL-settings
2024-04-20 16:23:33.466 [4914/F4797] INFO: rw------- pihole:pihole 82K FTL-strings
2024-04-20 16:23:33.467 [4917/F4797] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (No error information)
2024-04-20 16:23:33.467 [4916/F4797] WARNING: Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (No error information)
2024-04-20 16:23:33.467 [4914/F4797] INFO: rw------- pihole:pihole 20K FTL-domains
2024-04-20 16:23:33.467 [4914/F4797] INFO: rw------- pihole:pihole 168K FTL-clients
2024-04-20 16:23:33.467 [4914/F4797] INFO: rw------- pihole:pihole 12K FTL-upstreams
2024-04-20 16:23:33.468 [4914/F4797] INFO: rw------- pihole:pihole 590K FTL-queries
2024-04-20 16:23:33.468 [4914/F4797] INFO: rw------- pihole:pihole 8K FTL-overTime
2024-04-20 16:23:33.469 [4914/F4797] INFO: rw------- pihole:pihole 41K FTL-dns-cache
2024-04-20 16:23:33.469 [4914/F4797] INFO: rw------- pihole:pihole 4K FTL-per-client-regex
2024-04-20 16:23:33.469 [4914/F4797] INFO: rw------- pihole:pihole 552K FTL-fifo-log
2024-04-20 16:23:33.469 [4914/F4797] INFO: ---------------------------------------------------
2024-04-20 16:23:33.469 [4914/F4797] INFO: Please also include some lines from above the !!!!!!!!! header.
2024-04-20 16:23:33.470 [4914/F4797] INFO: Thank you for helping us to improve our FTL engine!
2024-04-20 16:23:33.470 [4914/F4797] INFO: Asking parent pihole-FTL (PID 4797) to shut down
2024-04-20 16:23:33.470 [4914/F4797] INFO: FTL fork terminated!
2024-04-20 16:23:33.470 [4797M] INFO: Received: RT37 (37 -> 2)
2024-04-20 16:23:33.470 [4916/F4797] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-04-20 16:23:33.470 [4916/F4797] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 16:23:33.470 [4916/F4797] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-04-20 16:23:33.470 [4916/F4797] INFO: Please report a bug at Issues · pi-hole/FTL · GitHub
2024-04-20 16:23:33.470 [4797M] INFO: Asked to terminate by "/usr/bin/pihole-FTL -f" (PID 4797, user pihole UID 999)
2024-04-20 16:23:33.470 [4916/F4797] INFO: and include in your report already the following details:
2024-04-20 16:23:33.470 [4916/F4797] INFO: FTL has been running for 224 seconds
2024-04-20 16:23:33.470 [4916/F4797] INFO: FTL branch: development-v6
2024-04-20 16:23:33.471 [4797M] INFO: Shutting down... // exit code 1 // jmpret 0
2024-04-20 16:23:33.736 [4797M] INFO: Finished final database update
2024-04-20 16:23:33.736 [4797M] INFO: Waiting for threads to join
2024-04-20 16:23:33.736 [4797M] INFO: Thread database (0) is idle, terminating it.
2024-04-20 16:23:33.737 [4797M] INFO: Thread housekeeper (1) is idle, terminating it.
2024-04-20 16:23:33.737 [4797M] INFO: Thread DNS client (2) is idle, terminating it.
2024-04-20 16:23:33.737 [4797M] INFO: All threads joined
2024-04-20 16:23:33.742 [4797M] INFO: Stored 1 API session in the database
2024-04-20 16:23:34.603 [4797M] INFO: ########## FTL terminated after 3m 45s (code 1)! ##########
2024-04-20 16:23:40.064 [4936M] INFO: ########## FTL started on pi! ##########
2024-04-20 16:23:40.064 [4936M] INFO: FTL branch: development-v6
2024-04-20 16:23:40.064 [4936M] INFO: FTL version: vDev-4734e01
2024-04-20 16:23:40.064 [4936M] INFO: FTL commit: 4734e01b
2024-04-20 16:23:40.065 [4936M] INFO: FTL date: 2024-04-18 21:28:52 +0200
2024-04-20 16:23:40.065 [4936M] INFO: FTL user: pihole
2024-04-20 16:23:40.065 [4936M] INFO: Compiled for linux/arm/v6 (compiled on CI) using cc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924

$ sudo cat /var/log/pihole-FTL.log | grep 'FTL crashed'
2024-04-20 10:37:26.638 [27191/F22808] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 10:39:17.926 [27279/F27215] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 10:39:17.928 [27280/F27215] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 10:41:16.937 [27369/F27305] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 10:41:29.456 [27404/F27388] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 10:45:25.967 [27784/F27425] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 11:10:46.182 [1840/F1476] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 11:11:15.931 [1898/F1863] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 11:11:27.553 [1933/F1920] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 11:20:16.765 [2155/F1962] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 11:22:50.437 [2275/F2176] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 11:44:11.476 [2622/F2297] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 11:46:59.995 [2705/F2642] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 12:18:16.023 [15742/F14090] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 12:18:25.524 [15769/F15761] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 15:09:01.776 [2827/F524] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 15:22:03.241 [3060/F2854] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 15:28:52.003 [3234/F3087] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 15:33:57.322 [3593/F3573] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 15:45:51.647 [3909/F3778] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 15:59:38.271 [4218/F3929] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 16:05:34.166 [4325/F4253] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 16:15:28.468 [4637/F4414] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 16:15:59.025 [4674/F4660] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 16:19:43.024 [4773/F4698] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 16:19:43.026 [4772/F4698] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 16:23:33.462 [4914/F4797] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 16:23:33.470 [4916/F4797] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 16:23:41.649 [4954/F4936] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-20 16:23:41.649 [4956/F4936] INFO: ----------------------------> FTL crashed! <----------------------------

DL6ER · April 20, 2024, 5:07pm

Sorry for the inconvenience, there is already a bug fix available for this issue currently waiting review.

github.com/pi-hole/FTL

Fix a crash resulting from a bad interaction between PRs #1928 and #1930

pi-hole:development-v6 ← pi-hole:fix/connection_error_crash

opened 07:42AM - 20 Apr 24 UTC

DL6ER

+1 -1

# What does this implement/fix? Fix a crash resulting from a bad interaction …between PRs #1928 and #1930. The fix is very simple: #1928 made it unnecessary to specify the number of following arguments. Unfortunately, current `development-v6` misinterprets this causing a crash as reported in #1935 **Related issue or feature (if applicable):** fixes #1935 **Pull request in [docs](https://github.com/pi-hole/docs) with documentation (if applicable):** N/A --- **By submitting this pull request, I confirm the following:** 1. I have read and understood the [contributors guide](https://docs.pi-hole.net/guides/github/contributing/), as well as this entire template. I understand which branch to base my commits and Pull Requests against. 2. I have commented my proposed changes within the code. 3. I am willing to help maintain this change if there are issues with it later. 4. It is compatible with the [EUPL 1.2 license](https://opensource.org/licenses/EUPL-1.1) 5. I have squashed any insignificant commits. ([`git rebase`](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)) ## Checklist: - [x] The code change is tested and works locally. - [x] I based my code and PRs against the repositories `developmental` branch. - [x] I [signed off](https://docs.pi-hole.net/guides/github/how-to-signoff/) all commits. Pi-hole enforces the [DCO](https://docs.pi-hole.net/guides/github/dco/) for all contributions - [x] I [signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) all my commits. Pi-hole requires signatures to verify authorship - [x] I have read the above and my PR is ready for review.

paul-da · April 20, 2024, 6:13pm

No worries at all, thank you for update @DL6ER .

edgd1er · April 20, 2024, 6:17pm

I had a similar problem and the workaround was to remove ipv6 dns as I don't have ipv6 enabled at the moment.
It seems that the current pihole-FTL does not handle very well connection problems.
As you already have tried to change dns, I don't know what to suggest.

May be have a "depends_on unbound" if you are using docker compose...

paul-da · April 20, 2024, 8:15pm

I'm not using IPv6 so disabled it in the OS:

sudo nmcli con mod eth0 ipv6.method disable

Then rebooted, then switched PiHole to only forward ipv4 upstream (I'm using Quad9).
That seems to have sorted it, or at the very least reduced the occurrences.

Not sure if that's related or coincidence. I won't pretend to understand the underlying issue. But this might help someone else.
I'll check it again tomorrow morning.

seh2000 · April 21, 2024, 12:31am

Hi,
Just to share,
I have and just had (01:10am) same issue.
Can upload the debug log if needed.
Steen

2024-04-21 01:10:21.697 [858M] WARNING: Connection error (1.1.1.1#53): failed to send UDP request (Network unreachable)
2024-04-21 01:10:21.698 [858M] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-04-21 01:10:21.698 [858M] INFO: ---------------------------->  FTL crashed!  <----------------------------
2024-04-21 01:10:21.698 [858M] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-04-21 01:10:21.699 [858M] INFO: Please report a bug at https://github.com/pi-hole/FTL/issues
2024-04-21 01:10:21.699 [858M] INFO: and include in your report already the following details:
2024-04-21 01:10:21.699 [858M] INFO: FTL has been running for 17223 seconds
2024-04-21 01:10:21.699 [858M] INFO: FTL branch: development-v6
2024-04-21 01:10:21.699 [858M] INFO: FTL version: vDev-4734e01
2024-04-21 01:10:21.699 [858M] INFO: FTL commit: 4734e01b
2024-04-21 01:10:21.699 [858M] INFO: FTL date: 2024-04-18 21:28:52 +0200
2024-04-21 01:10:21.699 [858M] INFO: FTL user: started as pihole, ended as pihole
2024-04-21 01:10:21.699 [858M] INFO: Compiled for linux/arm64/v8 (compiled on CI) using cc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924
2024-04-21 01:10:21.699 [858M] INFO: Process details: MID: 858
2024-04-21 01:10:21.699 [858M] INFO:                  PID: 858
2024-04-21 01:10:21.699 [858M] INFO:                  TID: 858
2024-04-21 01:10:21.699 [858M] INFO:                  Name: pihole-FTL
2024-04-21 01:10:21.700 [858M] INFO: Received signal: Segmentation fault
2024-04-21 01:10:21.700 [858M] INFO:      at address: 0x2
2024-04-21 01:10:21.700 [858M] INFO:      with code:  SEGV_MAPERR (Address not mapped to object)
2024-04-21 01:10:21.700 [858M] INFO: !!! INFO: pihole-FTL has not been compiled with glibc/backtrace support, not generating one !!!
2024-04-21 01:10:21.700 [858M] INFO: ------ Listing content of directory /dev/shm ------
2024-04-21 01:10:21.700 [858M] INFO: File Mode User:Group      Size  Filename
2024-04-21 01:10:21.700 [858M] INFO: rwxrwxrwx root:root       280   .
2024-04-21 01:10:21.700 [858M] INFO: rwxr-xr-x root:root         4K  ..
2024-04-21 01:10:21.700 [858M] INFO: rw------- pihole:pihole    88   FTL-lock
2024-04-21 01:10:21.700 [858M] INFO: rw------- pihole:pihole   292   FTL-counters
2024-04-21 01:10:21.701 [858M] INFO: rw------- pihole:pihole    16   FTL-settings
2024-04-21 01:10:21.701 [858M] INFO: rw------- pihole:pihole   123K  FTL-strings
2024-04-21 01:10:21.701 [858M] INFO: rw------- pihole:pihole   106K  FTL-domains
2024-04-21 01:10:21.701 [858M] INFO: rw------- pihole:pihole    86K  FTL-clients
2024-04-21 01:10:21.701 [858M] INFO: rw------- pihole:pihole    29K  FTL-upstreams
2024-04-21 01:10:21.701 [858M] INFO: rw------- pihole:pihole     2M  FTL-queries
2024-04-21 01:10:21.701 [858M] INFO: rw------- pihole:pihole     8K  FTL-overTime
2024-04-21 01:10:21.701 [858M] INFO: rw------- pihole:pihole   123K  FTL-dns-cache
2024-04-21 01:10:21.702 [858M] INFO: rw------- pihole:pihole     4K  FTL-per-client-regex
2024-04-21 01:10:21.702 [858M] INFO: rw------- pihole:pihole   560K  FTL-fifo-log
2024-04-21 01:10:21.702 [858M] INFO: ---------------------------------------------------
2024-04-21 01:10:21.702 [858M] INFO: Please also include some lines from above the !!!!!!!!! header.

DL6ER · April 21, 2024, 8:29am

The bug should now be fixed, please update your Pi-hole.

paul-da · April 21, 2024, 12:16pm

Yes, that's sorted it.

I undid my changes, saw it start to crash in the logs, updated an and the crashing stopped.

Thanks @DL6ER !

seh2000 · April 21, 2024, 5:17pm

@DL6ER
Not sure if to open new topic or?
Updated both my Pi-hole.
Started - again - to use the Pi-hole as DHCP sever, after each crash I use the router as DHCP sever.
Shortly after it crashed, which caused the Pi to dies.
A reset brings the Pi back to life and so Pi-hole.
FTL log is empty except for this entry:

2024-04-21 17:56:31.175 WARNING API: Not found (/api/hardware)

Did Pihole -d here the log https://tricorder.pi-hole.net/LgckCPmt/
Issue appears only when the DHCP sever is active / used via the Pi-hole.
I have a 2nd Pi/P-ihole running where I am using the the DHCP sever via the Router, I'll try later tonight to use that Pi/Pi-hole as DHCP sever, to see if I get similar issue..
The Pi is a Pi4+ with 4Gb RAM and a 64GB SSD card.
Steen

RonV42 · April 21, 2024, 5:31pm

Thanks saw my first crash this morning and then this thread. I am running IPv6 but no DHCP through the instance. Just did the last update and keeping fingers crossed.

rdwebdesign · April 21, 2024, 8:07pm

Your debug log shows FTL.log file is 480Kb:

-rw-r----- 1 pihole pihole 480K Apr 21 17:56 /var/log/pihole/FTL.log

And it is showing many another lines. The last 5 lines are:

(...)
   2024-04-21 17:17:15.117 [848M] INFO: Using SSL/TLS certificate file /etc/pihole/tls.pem
   2024-04-21 17:17:15.122 [848M] INFO: Restored 0 API sessions from the database
   2024-04-21 17:17:15.132 [848M] INFO: Blocking status is enabled
   2024-04-21 17:17:15.463 [848/T864] INFO: Compiled 9 allow and 20 deny regex for 34 clients in 250.6 msec
   2024-04-21 17:56:31.175 [848/T876] WARNING: API: Not found (/api/hardware)

Nothing seems wrong.

If you think there is something broken, please open a new topic with a new debug log.

seh2000 · April 21, 2024, 9:16pm

Thanks and will do.
Something is definitely there.