FTL crash after update v4.3.1

MeekLogic · March 31, 2020, 2:23am

Same issue here. Ran pihole -up and FTL keeps crashing. I had to switch my routers DHCP back on til this gets resolved.

jfb · March 31, 2020, 2:45am

In your debug log:

2020-03-30 22:42:04: (mod_fastcgi.c.421) FastCGI-stderr: PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 14684160 bytes) in /var/www/html/admin/api_db.php on line 404

TerrorSource · March 31, 2020, 9:39am

Do others have the same error in their debug logs?

Moto · March 31, 2020, 11:02am

2 Piholes here (3b & 4 both Buster fully uptodate) - both had FTL crashing after this mornings Pihole -up.

Pi 4 working after a single reboot. Pi 3b needed 2 reboots

-----tail of pihole-FTL.log------
[2020-03-31 11:18:00.180 4668] ----------------------------> FTL crashed! <----------------------------
[2020-03-31 11:18:00.180 4668] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-03-31 11:18:00.180 4668] Please report a bug at Issues · pi-hole/FTL · GitHub
[2020-03-31 11:18:00.180 4668] and include in your report already the following details:
[2020-03-31 11:18:00.180 4668] FTL has been running for 59 seconds
[2020-03-31 11:18:00.180 4668] FTL branch: release/v5.0
[2020-03-31 11:18:00.180 4668] FTL version: vDev-b6364d0
[2020-03-31 11:18:00.180 4668] FTL commit: b6364d0
[2020-03-31 11:18:00.180 4668] FTL date: 2020-03-29 23:23:02 +0200
[2020-03-31 11:18:00.181 4668] FTL user: started as pihole, ended as pihole
[2020-03-31 11:18:00.181 4668] Compiled for armhf (compiled on CI) using arm-linux-gnueabihf-gcc (Debian 6.3.0-18) 6.3.0 20170516
[2020-03-31 11:18:00.181 4668] Received signal: Segmentation fault
[2020-03-31 11:18:00.181 4668] at address: 0x1
[2020-03-31 11:18:00.181 4668] with code: SEGV_MAPERR (Address not mapped to object)
[2020-03-31 11:18:00.181 4668] Backtrace:
[2020-03-31 11:18:00.181 4668] B[0000]: 0x499618, /usr/bin/pihole-FTL(+0x23618) [0x499618]
[2020-03-31 11:18:00.181 4668] B[0001]: 0xb6d77130, /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0xb6d77130]
[2020-03-31 11:18:00.182 4668] ------ Listing content of directory /dev/shm ------
[2020-03-31 11:18:00.182 4668] File Mode User:Group Filesize Filename
[2020-03-31 11:18:00.182 4668] rwxrwxrwx root:root 260 .
[2020-03-31 11:18:00.182 4668] rwxr-xr-x root:root 4K ..
[2020-03-31 11:18:00.182 4668] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-03-31 11:18:00.182 4668] rw------- pihole:pihole 4K FTL-dns-cache
[2020-03-31 11:18:00.183 4668] rw------- pihole:pihole 29K FTL-overTime
[2020-03-31 11:18:00.183 4668] rw------- pihole:pihole 1M FTL-queries
[2020-03-31 11:18:00.183 4668] rw------- pihole:pihole 20K FTL-upstreams
[2020-03-31 11:18:00.183 4668] rw------- pihole:pihole 20K FTL-clients
[2020-03-31 11:18:00.184 4668] rw------- pihole:pihole 66K FTL-domains
[2020-03-31 11:18:00.184 4668] rw------- pihole:pihole 29K FTL-strings
[2020-03-31 11:18:00.184 4668] rw------- pihole:pihole 12 FTL-settings
[2020-03-31 11:18:00.184 4668] rw------- pihole:pihole 124 FTL-counters
[2020-03-31 11:18:00.184 4668] rw------- pihole:pihole 28 FTL-lock
[2020-03-31 11:18:00.184 4668] ---------------------------------------------------
[2020-03-31 11:18:00.185 4668] Thank you for helping us to improve our FTL engine!
[2020-03-31 11:18:00.185 4668] FTL terminated!

ChurchOfNoise · March 31, 2020, 12:52pm

Same issue here, debug code dhwwvvkgd8

Seems ipV6 addresses are not the same

zadigre · March 31, 2020, 1:22pm

I posted something similar in the following thread :

see if setting MAXDBDAYS to 0 help to fix the problem temporarily.

Moto · March 31, 2020, 1:32pm

2 hours later I made the mistake of rebooting my Pi 3b - unable to get it working again with multiple reboots.
Same error
[2020-03-31 13:35:00.261 601] Received signal: Segmentation fault
[2020-03-31 13:35:00.262 601] at address: 0x1
[2020-03-31 13:35:00.262 601] with code: SEGV_MAPERR (Address not mapped to object)
[2020-03-31 13:35:00.262 601] Backtrace:
[2020-03-31 13:35:00.262 601] B[0000]: 0x43c618, /usr/bin/pihole-FTL(+0x23618) [0x43c618]
[2020-03-31 13:35:00.263 601] B[0001]: 0x76d7f130, /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0x76d7f130]
[2020-03-31 13:35:00.263 601] B[0002]: 0x434c1c, /usr/bin/pihole-FTL(parse_neighbor_cache+0x343) [0x434c1c]
[2020-03-31 13:35:00.263 601] B[0003]: 0x4375e8, /usr/bin/pihole-FTL(DB_thread+0xc7) [0x4375e8]

DontHaveAClue · March 31, 2020, 2:01pm

Thank you! I had just installed pihole on an atomic pi and could not get it stay up for more than 1 minute. So far so good since setting MAXDBDAYS=0.

ChurchOfNoise · March 31, 2020, 2:10pm

Works for me too (FTL is not stopping anymore), but no graphs in the web interface as a result...

DL6ER · March 31, 2020, 4:33pm

Thanks for your reports, we're currently trying to isolate the cause of the issue. It is interesting you note that the crash disappears when disabling the database. Please try again with the database enabled after updating your Pi-holes to the latest version of the beta (we just pushed a modification to the database code). I do not really expect this to resolve this bug but one never knows.

Now we know.

zadigre · March 31, 2020, 10:40am

FTL is crashing everytime data is saved to the FTL database.
My DBINTERVAL was set to 30... now set to 5 to test.
I tried to reboot my RPI.
I also tried to delete FTL DB file.

I'm using raspiBackup to do my daily backup on an external USB key.
this crash started happening right after the backup. It worked fine all day before this backup.
to make sure the FTL DB is consistent, I stop the FTL service before the backup.

FTL is not crashing when MAXDBDAYS is set to 0

here the FTL log for one of the crash:

[2020-03-31 06:10:00.311 736] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-03-31 06:10:00.311 736] ----------------------------> FTL crashed! <----------------------------
[2020-03-31 06:10:00.311 736] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-03-31 06:10:00.311 736] Please report a bug at Issues · pi-hole/FTL · GitHub
[2020-03-31 06:10:00.311 736] and include in your report already the following details:
[2020-03-31 06:10:00.311 736] FTL has been running for 462 seconds
[2020-03-31 06:10:00.311 736] FTL branch: release/v5.0
[2020-03-31 06:10:00.311 736] FTL version: vDev-b6364d0
[2020-03-31 06:10:00.311 736] FTL commit: b6364d0
[2020-03-31 06:10:00.311 736] FTL date: 2020-03-29 23:23:02 +0200
[2020-03-31 06:10:00.311 736] FTL user: started as pihole, ended as pihole
[2020-03-31 06:10:00.312 736] Compiled for armhf (compiled on CI) using arm-linux-gnueabihf-gcc (Debian 6.3.0-18) 6.3.0 20170516
[2020-03-31 06:10:00.312 736] Received signal: Segmentation fault
[2020-03-31 06:10:00.312 736] at address: 0x1
[2020-03-31 06:10:00.312 736] with code: SEGV_MAPERR (Address not mapped to object)
[2020-03-31 06:10:00.312 736] Backtrace:
[2020-03-31 06:10:00.312 736] B[0000]: 0x463618, /usr/bin/pihole-FTL(+0x23618) [0x463618]
[2020-03-31 06:10:00.312 736] B[0001]: 0xb6dae130, /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0xb6dae130]
[2020-03-31 06:10:00.313 736] B[0002]: 0x45bc1c, /usr/bin/pihole-FTL(parse_neighbor_cache+0x343) [0x45bc1c]
[2020-03-31 06:10:00.313 736] B[0003]: 0x45e5e8, /usr/bin/pihole-FTL(DB_thread+0xc7) [0x45e5e8]
[2020-03-31 06:10:00.313 736] ------ Listing content of directory /dev/shm ------
[2020-03-31 06:10:00.313 736] File Mode User:Group Filesize Filename
[2020-03-31 06:10:00.313 736] rwxrwxrwx root:root 260 .
[2020-03-31 06:10:00.313 736] rwxr-xr-x root:root 4K ..
[2020-03-31 06:10:00.313 736] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-03-31 06:10:00.314 736] rw------- pihole:pihole 4K FTL-dns-cache
[2020-03-31 06:10:00.314 736] rw------- pihole:pihole 29K FTL-overTime
[2020-03-31 06:10:00.314 736] rw------- pihole:pihole 229K FTL-queries
[2020-03-31 06:10:00.314 736] rw------- pihole:pihole 20K FTL-upstreams
[2020-03-31 06:10:00.315 736] rw------- pihole:pihole 20K FTL-clients
[2020-03-31 06:10:00.315 736] rw------- pihole:pihole 66K FTL-domains
[2020-03-31 06:10:00.315 736] rw------- pihole:pihole 8K FTL-strings
[2020-03-31 06:10:00.316 736] rw------- pihole:pihole 12 FTL-settings
[2020-03-31 06:10:00.316 736] rw------- pihole:pihole 124 FTL-counters
[2020-03-31 06:10:00.316 736] rw------- pihole:pihole 28 FTL-lock
[2020-03-31 06:10:00.316 736] ---------------------------------------------------
[2020-03-31 06:10:00.316 736] Thank you for helping us to improve our FTL engine!
[2020-03-31 06:10:00.316 736] FTL terminated!

debug token https://tricorder.pi-hole.net/rk8w1qzdwn

adding some info about my system :
uname -a
Linux raspberrypi 4.19.97-v7l+ #1294 SMP Thu Jan 30 13:21:14 GMT 2020 armv7l GNU/Linux

pihole -v
Pi-hole version is v4.3.5-459-g0fad979 (Latest: v4.4)
AdminLTE version is v4.3.2-439-g6fdeee5 (Latest: v4.3.3)
FTL version is vDev-b6364d0 (Latest: v4.3.1)

/proc/cpuinfo
Hardware : BCM2835
Revision : c03111
Model : Raspberry Pi 4 Model B Rev 1.1

ugsd · March 31, 2020, 2:22pm

I killed my entire Sunday trying to figure this one out. Ended up reverting back to stable, even though i've been running 5.0 since the day it was announced. Fixed a bunch of other things in the process, so i got that going for me...

2020-03-30 03:01:00.728 8956] dbquery: "SELECT id FROM network WHERE hwaddr = '68:c6:3a:af:2c:d4';"
[2020-03-30 03:01:00.728 8956] Querying gravity database for client 10.0.1.119 (counting)
[2020-03-30 03:01:00.729 8956] Querying gravity database for client 10.0.1.119 (counting)
[2020-03-30 03:01:00.730 8956] dbquery: "SELECT id FROM network WHERE hwaddr = 'ip-10.0.1.119' AND firstSeen > (cast(strftime('%s', 'now') as int)-3600);"
[2020-03-30 03:01:00.755 8956] Device with IP 10.0.1.119 not known and no recent mock-device found ---> creating new record
[2020-03-30 03:01:00.755 8956] dbquery: "INSERT INTO network (hwaddr,interface,firstSeen,lastQuery,numQueries,name,macVendor) VALUES ('68:c6:3a:af:2c:d4','enp2s0',4294967297, 0, 0, '', 'Espressif Inc.');"
[2020-03-30 03:01:00.756 8956] dbquery: "INSERT OR REPLACE INTO network_addresses (network_id,ip,lastSeen) VALUES(21,'10.0.1.119',(cast(strftime('%s', 'now') as int)));"
[2020-03-30 03:01:00.756 8956] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-03-30 03:01:00.756 8956] ---------------------------->  FTL crashed!  <----------------------------
[2020-03-30 03:01:00.756 8956] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-03-30 03:01:00.756 8956] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-03-30 03:01:00.757 8956] and include in your report already the following details:
[2020-03-30 03:01:00.757 8956] FTL has been running for 18 seconds
[2020-03-30 03:01:00.757 8956] FTL branch: release/v5.0
[2020-03-30 03:01:00.757 8956] FTL version: vDev-b6364d0
[2020-03-30 03:01:00.757 8956] FTL commit: b6364d0
[2020-03-30 03:01:00.757 8956] FTL date: 2020-03-29 23:23:02 +0200
[2020-03-30 03:01:00.757 8956] FTL user: started as pihole, ended as pihole
[2020-03-30 03:01:00.758 8956] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2020-03-30 03:01:00.758 8956] Received signal: Segmentation fault
[2020-03-30 03:01:00.758 8956]      at address: 0x7f4d00000001
[2020-03-30 03:01:00.758 8956]      with code: SEGV_MAPERR (Address not mapped to object)
[2020-03-30 03:01:00.759 8956] Backtrace:
[2020-03-30 03:01:00.759 8956] B[0000] 0x564f734b3189, /usr/bin/pihole-FTL(+0x31189) [0x564f734b3189]
[2020-03-30 03:01:00.760 8956] B[0001] 0x7f4d5de97890, /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f4d5de97890]
[2020-03-30 03:01:00.760 8956] B[0002] 0x7f4d5dbc8452, /lib/x86_64-linux-gnu/libc.so.6(__vasprintf_chk+0x102) [0x7f4d5dbc8452]
[2020-03-30 03:01:00.760 8956] B[0003] 0x7f4d5dbc832f, /lib/x86_64-linux-gnu/libc.so.6(__asprintf_chk+0x8f) [0x7f4d5dbc832f]
[2020-03-30 03:01:00.760 8956] B[0004] 0x564f734a8c78, /usr/bin/pihole-FTL(parse_neighbor_cache+0x248) [0x564f734a8c78]
[2020-03-30 03:01:00.760 8956] B[0005] 0x564f734ac87d, /usr/bin/pihole-FTL(DB_thread+0x10d) [0x564f734ac87d]
[2020-03-30 03:01:00.760 8956] B[0006] 0x7f4d5de8c6db, /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f4d5de8c6db]
[2020-03-30 03:01:00.760 8956] B[0007] 0x7f4d5dbb588f, /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f4d5dbb588f]
[2020-03-30 03:01:00.760 8956] ------ Listing content of directory /dev/shm ------
[2020-03-30 03:01:00.760 8956] File Mode User:Group  Filesize Filename
[2020-03-30 03:01:00.761 8956] rwxrwxrwx root:root 260 .
[2020-03-30 03:01:00.761 8956] rwxr-xr-x root:root 4K ..
[2020-03-30 03:01:00.761 8956] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-03-30 03:01:00.762 8956] rw------- pihole:pihole 4K FTL-dns-cache
[2020-03-30 03:01:00.762 8956] rw------- pihole:pihole 12K FTL-overTime
[2020-03-30 03:01:00.762 8956] rw------- pihole:pihole 524K FTL-queries
[2020-03-30 03:01:00.763 8956] rw------- pihole:pihole 4K FTL-upstreams
[2020-03-30 03:01:00.763 8956] rw------- pihole:pihole 340K FTL-clients
[2020-03-30 03:01:00.763 8956] rw------- pihole:pihole 98K FTL-domains
[2020-03-30 03:01:00.764 8956] rw------- pihole:pihole 12K FTL-strings
[2020-03-30 03:01:00.764 8956] rw------- pihole:pihole 12 FTL-settings
[2020-03-30 03:01:00.764 8956] rw------- pihole:pihole 124 FTL-counters
[2020-03-30 03:01:00.765 8956] rw------- pihole:pihole 48 FTL-lock
[2020-03-30 03:01:00.765 8956] ---------------------------------------------------
[2020-03-30 03:01:00.765 8956] Thank you for helping us to improve our FTL engine!
[2020-03-30 03:01:00.765 8956] FTL terminated!

zadigre · March 31, 2020, 4:33pm

FTL has been updated to d5253f1... and it didn't fix the problem for me.

DL6ER · March 31, 2020, 5:14pm

9 posts were split to a new topic: No blocking without the long-term database

DL6ER · March 31, 2020, 5:19pm

A post was merged into an existing topic: No blocking without the long-term database

DL6ER · March 31, 2020, 5:45pm

2 posts were merged into an existing topic: No blocking without the long-term database

DL6ER · March 31, 2020, 5:51pm

Just FYI, we're also tracking the issue here:

github.com/pi-hole/FTL

FTL crashing while DHCP server is enabled

opened 06:37PM - 30 Mar 20 UTC

closed 07:16PM - 02 Apr 20 UTC

igalita

Bug

**In raising this issue, I confirm the following (please check boxes, eg [X]) Fa…ilure to fill the template will close your issue:** - [x] I have read and understood the [contributors guide](https://github.com/pi-hole/pi-hole/blob/master/CONTRIBUTING.md). - [x] The issue I am reporting can be *replicated* - [x] The issue I am reporting isn't a duplicate **How familiar are you with the codebase?:** 7 --- **[ISSUE] Expected Behaviour:** After update to `vDev-b6364d0`, FTL should not crash while DHCP server is enabled. **[BUG] Actual Behaviour:** FTL became unstable after update to `vDev-b6364d0`. It crashes after 10-30 seconds. **[BUG | ISSUE] Steps to reproduce:** Enable DHCP server. FTL should crash soon after that. **Log file output [if available]** [https://tricorder.pi-hole.net/dli98t4why](https://tricorder.pi-hole.net/dli98t4why) ```sh (gdb) handle SIGHUP nostop SIGPIPE nostop Signal Stop Print Pass to program Description SIGHUP No Yes Yes Hangup SIGPIPE No Yes Yes Broken pipe (gdb) continue Continuing. [Detaching after fork from child process 5222] [New Thread 0x739ff460 (LWP 5242)] [Thread 0x739ff460 (LWP 5242) exited] [New Thread 0x739ff460 (LWP 5243)] [Thread 0x739ff460 (LWP 5243) exited] (gdb) [Detaching after fork from child process 5244] Thread 5 "database" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x75369460 (LWP 5191)] __strchrnul (s=0x2 <error: Cannot access memory at address 0x2>, c_in=37) at strchrnul.c:50 50 strchrnul.c: No such file or directory. (gdb) backtrace #0 __strchrnul (s=0x2 <error: Cannot access memory at address 0x2>, c_in=37) at strchrnul.c:50 #1 0x76d75174 in __find_specmb (format=0x2 <error: Cannot access memory at address 0x2>) at printf-parse.h:108 #2 _IO_vfprintf_internal (s=s@entry=0x75368ab8, format=format@entry=0x2 <error: Cannot access memory at address 0x2>, ap=..., ap@entry=...) at vfprintf.c:1315 #3 0x76e1d024 in __GI___vasprintf_chk (result_ptr=result_ptr@entry=0x75368c60, flags=flags@entry=1, format=0x2 <error: Cannot access memory at address 0x2>, format@entry=0x0, args=..., args@entry=...) at vasprintf_chk.c:66 #4 0x76e1cf30 in __asprintf_chk (result_ptr=result_ptr@entry=0x75368c60, flags=flags@entry=1, format=0x2 <error: Cannot access memory at address 0x2>) at asprintf_chk.c:32 #5 0x00448a76 in asprintf (__fmt=0x532f5c "SELECT id FROM network WHERE hwaddr = '%s';", __ptr=0x75368c60) at /usr/arm-linux-gnueabihf/include/bits/stdio2.h:178 #6 parse_neighbor_cache () at src/database/network-table.c:325 #7 0x0044b5e8 in DB_thread (val=<optimized out>) at src/database/database-thread.c:64 #8 0x76e88494 in start_thread (arg=0x75369460) at pthread_create.c:486 #9 0x76e0b578 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6 Backtrace stopped: previous frame identical to this frame (corrupt stack?) ``` **Device specifics** Hardware Type: rPi 3B+ OS: Raspbian Buster _This template was created based on the work of [`udemy-dl`](https://github.com/nishad/udemy-dl/blob/master/LICENSE)._

TerrorSource · March 31, 2020, 6:04pm

I am not using the DHCP function of PiHole (My Unifi Controller does that).

I did however had Conditional Forwarding set up in PiHole, dont know if that helps?

DL6ER · March 31, 2020, 6:07pm

DHCP is likely a red herring. We're still trying to identify the bug.