@DL6ER Running the proposed fix now. Will let you know if I continue to have issues. Thanks for all the help
1 Like
@DL6ER Seems the API wants to keep shutting down. When this happens I have to reboot the RPi. It will work fine for a few minutes then the API stops again
Could you check why this is happening? What are the last lines in /var/log/pihole-FTL.log when the API goes offline?
I restarted the pihole this morning and copied the last part of the log. The gap in the log was placed there by me. It was just redundancy of the error some 20 times. Here it is
[2020-02-04 22:56:28.885 648] New forward server: 1.1.1.1 (-1/1024)
[2020-02-04 22:56:28.885 648] FATAL: Trying to access forward ID -1, but maximum is 1024
[2020-02-04 22:56:28.885 648] found in findForwardID() (src/datastructure.c:54)
[2020-02-04 22:56:28.885 648] ERROR: Encountered serious memory error in findForwardID()
[2020-02-04 22:56:28.909 648] New forward server: 1.1.1.1 (-1/1024)
[2020-02-04 22:56:28.909 648] FATAL: Trying to access forward ID -1, but maximum is 1024
[2020-02-04 22:56:28.909 648] found in findForwardID() (src/datastructure.c:54)
[2020-02-04 22:56:28.909 648] ERROR: Encountered serious memory error in findForwardID()
[2020-02-04 22:57:18.724 648] New forward server: 1.1.1.1 (-9/1024)
[2020-02-04 22:57:18.724 648] FATAL: Trying to access forward ID -9, but maximum is 1024
[2020-02-04 22:57:18.724 648] found in findForwardID() (src/datastructure.c:54)
[2020-02-04 22:57:18.724 648] ERROR: Encountered serious memory error in findForwardID()
[2020-02-04 22:57:19.526 648] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-02-04 22:57:19.526 648] ----------------------------> FTL crashed! <----------------------------
[2020-02-04 22:57:19.526 648] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-02-04 22:57:19.526 648] Please report a bug at Issues · pi-hole/FTL · GitHub
[2020-02-04 22:57:19.526 648] and include in your report already the following details:
[2020-02-04 22:57:19.526 648] FTL has been running for 119 seconds
[2020-02-04 22:57:19.526 648] FTL branch: fix/negative_blocked_queries
[2020-02-04 22:57:19.526 648] FTL version: vDev-6562823
[2020-02-04 22:57:19.526 648] FTL commit: 6562823
[2020-02-04 22:57:19.527 648] FTL date: 2020-02-04 20:52:00 +0100
[2020-02-04 22:57:19.527 648] FTL user: started as pihole, ended as pihole
[2020-02-04 22:57:19.527 648] Compiled for armhf (compiled on CI) using arm-linux-gnueabihf-gcc (Debian 6.3.0-18) 6.3.0 20170516
[2020-02-04 22:57:19.527 648] Received signal: Segmentation fault
[2020-02-04 22:57:19.527 648] at address: 0xf3afe9bc
[2020-02-04 22:57:19.527 648] with code: SEGV_MAPERR (Address not mapped to object)
[2020-02-04 22:58:15.949 633] Using log file /var/log/pihole-FTL.log
[2020-02-04 22:58:15.950 633] ########## FTL started! ##########
[2020-02-04 22:58:15.950 633] FTL branch: fix/negative_blocked_queries
[2020-02-04 22:58:15.950 633] FTL version: vDev-6562823
[2020-02-04 22:58:15.950 633] FTL commit: 6562823
[2020-02-04 22:58:15.950 633] FTL date: 2020-02-04 20:52:00 +0100
[2020-02-04 22:58:15.950 633] FTL user: pihole
[2020-02-04 22:58:15.950 633] Compiled for armhf (compiled on CI) using arm-linux-gnueabihf-gcc (Debian 6.3.0-18) 6.3.0 20170516
[2020-02-04 22:58:15.950 633] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
[2020-02-04 22:58:15.950 633] SOCKET_LISTENING: only local
[2020-02-04 22:58:15.950 633] AAAA_QUERY_ANALYSIS: Show AAAA queries
[2020-02-04 22:58:15.951 633] MAXDBDAYS: max age for stored queries is 365 days
[2020-02-04 22:58:15.951 633] RESOLVE_IPV6: Resolve IPv6 addresses
[2020-02-04 22:58:15.951 633] RESOLVE_IPV4: Resolve IPv4 addresses
[2020-02-04 22:58:15.951 633] DBINTERVAL: saving to DB file every minute
[2020-02-04 22:58:15.951 633] DBFILE: Using /etc/pihole/pihole-FTL.db
[2020-02-04 22:58:15.951 633] MAXLOGAGE: Importing up to 24.0 hours of log data
[2020-02-04 22:58:15.951 633] PRIVACYLEVEL: Set to 0
[2020-02-04 22:58:15.951 633] IGNORE_LOCALHOST: Show queries from localhost
[2020-02-04 22:58:15.951 633] BLOCKINGMODE: Null IPs for blocked domains
[2020-02-04 22:58:15.951 633] ANALYZE_ONLY_A_AND_AAAA: Disabled. Analyzing all queries
[2020-02-04 22:58:15.952 633] DBIMPORT: Importing history from database
PS In the time it has taken me to retrieve and post this the unit has shut down again.
I see, something is really going bad here, I wonder why nobody else is seeing/reporting this. Unfortunately, I'm traveling until at least Tuesday and cannot really do any testing myself, however, may I ask you to attach the debugger to FTL and generate a proper backtrace for us?
See instructions here: Debugging FTLDNS - Pi-hole documentation
OK When I get some free time I'll restart the 5.0 beta machine and do that.
I'll be in touch
This is what came up........
[New Thread 0x735ff460 (LWP 5866)]
[Thread 0x735ff460 (LWP 5866) exited]
[New Thread 0x735ff460 (LWP 5867)]
[Thread 0x735ff460 (LWP 5867) exited]
[New Thread 0x735ff460 (LWP 5868)]
[Thread 0x735ff460 (LWP 5868) exited]
[New Thread 0x735ff460 (LWP 5879)]
[Thread 0x735ff460 (LWP 5879) exited]
[New Thread 0x735ff460 (LWP 5880)]
[Thread 0x735ff460 (LWP 5880) exited]
[New Thread 0x735ff460 (LWP 5881)]
[Thread 0x735ff460 (LWP 5881) exited]
[New Thread 0x735ff460 (LWP 5882)]
Thread 824 "telnet-18" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x735ff460 (LWP 5882)]
cmpdesc (a=0xf35fe9a8, b=0x735fe9c0) at src/api/api.c:53
53 src/api/api.c: No such file or directory.
(gdb) continue
Continuing.
Thread 824 "telnet-18" received signal SIGSEGV, Segmentation fault.
0x76ea365c in _Unwind_VRS_Pop () from /lib/arm-linux-gnueabihf/libgcc_s.so.1
(gdb)
(gdb) backtrace
#0 0x76ea365c in _Unwind_VRS_Pop ()
from /lib/arm-linux-gnueabihf/libgcc_s.so.1
#1 0x76ea3bac in ?? () from /lib/arm-linux-gnueabihf/libgcc_s.so.1
#2 0x76ea31b0 in ?? () from /lib/arm-linux-gnueabihf/libgcc_s.so.1
#3 0x76ea2f4c in ?? () from /lib/arm-linux-gnueabihf/libgcc_s.so.1
#4 0x76ea39ac in _Unwind_Backtrace ()
from /lib/arm-linux-gnueabihf/libgcc_s.so.1
#5 0x76df7098 in __GI___backtrace (array=array@entry=0x735fe058,
size=size@entry=255) at ../sysdeps/arm/backtrace.c:105
#6 0x0043badc in SIGSEGV_handler (sig=11, si=0x735fe478,
unused=) at src/signals.c:56
#7
#8 cmpdesc (a=0xf35fe9a8, b=0x735fe9c0) at src/api/api.c:53
#9 0x76d3e280 in _quicksort (pbase=0x20, pbase@entry=0x735fe9c0,
total_elems=total_elems@entry=4294967291, size=size@entry=8,
cmp=cmp@entry=0x4385e9 , arg=arg@entry=0x0) at qsort.c:122
#10 0x76d3eae0 in __GI___qsort_r (b=0x735fe9c0,
b@entry=0x437dff <process_request+430>, n=4294967291, s=s@entry=8,
cmp=0x4385e9 , arg=arg@entry=0x0) at msort.c:216
#11 0x76d3ebb8 in __GI_qsort (b=b@entry=0x437dff <process_request+430>,
n=, s=s@entry=8, cmp=) at msort.c:308
#12 0x0043927c in getForwardDestinations (
client_message=client_message@entry=0x737005c8 ">forward-dest",
--Type for more, q to quit, c to continue without paging--
try=0x735fea28) at src/api/api.c:496
#13 0x00437dfe in process_request (client_message=client_message@entry=0x737005c8 ">forward-dest", sock=sock@entry=0x735fea28) at src/api/request.c:65
#14 0x0043710a in telnet_connection_handler_thread (socket_desc=0x73600748) at src/api/socket.c:336
#15 0x76e65494 in start_thread (arg=0x735ff460) at pthread_create.c:486
#16 0x76de8578 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)
Just as an aside - is your OS up to date? Seeing as it's shutting itself down and having process/memory issues, you could try changing the power supply - I've seen users report flaky behaviour dues to a "bad" power supply.
I started with a fresh download and update when I signed on for the beta. The board and power supply were working without issue when I was using the stable release.
OK, just a suggestion. The Raspbian packages are updated fairly often, including the kernel, so it doesn't hurt. Also the PSU change is usually a quick thing to try - the unit might simply be drawing more power now and sometimes things coincidentally fail.
No crazy looking errors in /var/log/syslog about the time of shutdown?
Okay, so the crash happened in quicksort, this is a Linux builtin. I've not seen a crash therein before. Does it always crash at the same place?
If this is an issue concerning part of the linux system should I try a fresh install?
No, I think I've found what is causing problems. Please try
pihole checkout ftl fix/negative_forwardIDs
Please update your Pi-hole, the issue should be fixed in the most recent version.
I updated the current machine and it appears to run a little longer but still loses the connection to api
You're not having much luck are you?! Perhaps the output of another 'screen' backtrace will help them to see if it's the same issue. You are definitely on the latest via pihole -v?
well it made it 1 hr and 48 min this time before FTL stopped
Okay so it has stopped a few times but it is lasting longer. What info would you like?