PiHole FTL Crashing

Help
1

Hello,

I've been experiencing issues with my FTL Service crashing randomly. Sometimes after a few days, sometimes after a few hours. At first it was once a week, so I set up a reoccurring reboot in the mornings during low traffic times, now it seems that even that isn't stopping the errors.

I've uploaded my debug log:
https://tricorder.pi-hole.net/tlzzra2baj

(I had restarted the FTL service before uploading the log, so it may have messed up some of the data, if so, let me know and I can upload it next time it happens)

I'm currently running PiHole on Ubuntu 16.04 LTS, on an ESXi VM.

I also see that it's been a common issue, with lots of other posts about it, but I never seen any actual resolution solution, therefore I made my own post. (Sorry if this wasn't the correct step)

Thanks a lot for any support,
-Arctic

2

These are abnormal in your debug log, and at the time the debug log was generated indicate a connectivity problem:

*** [ DIAGNOSING ]: Networking
[✗] No IPv4 address(es) found on the ens160 interface.

[✗] No IPv6 address(es) found on the ens160 interface.

[i] Default IPv4 gateway: 10.191.100.2

*** [ DIAGNOSING ]: Name resolution (IPv4) using a random blocked domain and a known ad-serving domain
[✓] metrics.oregonlive.com is 0.0.0.0 via localhost (127.0.0.1)
[✗] Failed to resolve metrics.oregonlive.com via Pi-hole (10.191.100.142)

The 24 hour history of DNS queries shows quite a bit of activity.

   [2020-05-28 12:12:22.319 14491] Imported 498203 queries from the long-term database
   [2020-05-28 12:12:22.319 14491]  -> Total DNS queries: 498203
   [2020-05-28 12:12:22.319 14491]  -> Cached DNS queries: 486642
   [2020-05-28 12:12:22.319 14491]  -> Forwarded DNS queries: 10348
   [2020-05-28 12:12:22.319 14491]  -> Blocked DNS queries: 1193
   [2020-05-28 12:12:22.319 14491]  -> Unknown DNS queries: 20
   [2020-05-28 12:12:22.319 14491]  -> Unique domains: 371
   [2020-05-28 12:12:22.319 14491]  -> Unique clients: 26
   [2020-05-28 12:12:22.319 14491]  -> Known forward destinations: 4

Check in the following logs for any errors:

/var/log/pihole.log
/var/log/pihole.log.1
/var/log/pihole-FTL.log
/var/log/pihole-FTL.log.1
/var/log/syslog
3

Hello,

Thanks for the quick response. I'm not seeing anything odd in the /var/log/syslog.
The pihole-FTL.log displays the crashing errors:


[2020-05-28 09:41:53.009 1404] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-28 09:41:53.010 1404] ---------------------------->  FTL crashed!  <----------------------------
[2020-05-28 09:41:53.010 1404] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-28 09:41:53.010 1404] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-05-28 09:41:53.010 1404] and include in your report already the following details:
[2020-05-28 09:41:53.010 1404] FTL has been running for 287705 seconds
[2020-05-28 09:41:53.010 1404] FTL branch: master
[2020-05-28 09:41:53.010 1404] FTL version: v5.0
[2020-05-28 09:41:53.010 1404] FTL commit: 3d7c095
[2020-05-28 09:41:53.010 1404] FTL date: 2020-05-10 18:58:38 +0100
[2020-05-28 09:41:53.010 1404] FTL user: started as pihole, ended as pihole
[2020-05-28 09:41:53.010 1404] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2020-05-28 09:41:53.010 1404] Received signal: Segmentation fault
[2020-05-28 09:41:53.010 1404]      at address: (nil)
[2020-05-28 09:41:53.010 1404]      with code: Unknown (128)
[2020-05-28 09:41:53.011 1404] Backtrace:
[2020-05-28 09:41:53.011 1404] B[0000]: 0x561fc63136f9, /usr/bin/pihole-FTL(+0x316f9) [0x561fc63136f9]
[2020-05-28 09:41:53.011 1404] B[0001]: 0x7f2d94a43390, /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f2d94a43390]
[2020-05-28 09:41:53.011 1404] B[0002]: 0x7f2d946f3746, /lib/x86_64-linux-gnu/libc.so.6(strlen+0x26) [0x7f2d946f3746]
[2020-05-28 09:41:53.011 1404] B[0003]: 0x561fc6340582, /usr/bin/pihole-FTL(answer_request+0x26f2) [0x561fc6340582]
[2020-05-28 09:41:53.011 1404] B[0004]: 0x561fc6332283, /usr/bin/pihole-FTL(receive_query+0xb43) [0x561fc6332283]
[2020-05-28 09:41:53.011 1404] B[0005]: 0x561fc634894b, /usr/bin/pihole-FTL(+0x6694b) [0x561fc634894b]
[2020-05-28 09:41:53.011 1404] B[0006]: 0x561fc634a7bc, /usr/bin/pihole-FTL(main_dnsmasq+0x129c) [0x561fc634a7bc]
[2020-05-28 09:41:53.011 1404] B[0007]: 0x561fc63068ac, /usr/bin/pihole-FTL(main+0xdc) [0x561fc63068ac]
[2020-05-28 09:41:53.011 1404] B[0008]: 0x7f2d94688830, /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f2d94688830]
[2020-05-28 09:41:53.011 1404] B[0009]: 0x561fc6306a1a, /usr/bin/pihole-FTL(_start+0x2a) [0x561fc6306a1a]
[2020-05-28 09:41:53.011 1404] ------ Listing content of directory /dev/shm ------
[2020-05-28 09:41:53.011 1404] File Mode User:Group  Filesize Filename
[2020-05-28 09:41:53.012 1404] rwxrwxrwx root:root 260 .
[2020-05-28 09:41:53.012 1404] rwxr-xr-x root:root 4K ..
[2020-05-28 09:41:53.012 1404] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-05-28 09:41:53.012 1404] rw------- pihole:pihole 33K FTL-dns-cache
[2020-05-28 09:41:53.012 1404] rw------- pihole:pihole 12K FTL-overTime
[2020-05-28 09:41:53.012 1404] rw------- pihole:pihole 32M FTL-queries
[2020-05-28 09:41:53.012 1404] rw------- pihole:pihole 4K FTL-upstreams
[2020-05-28 09:41:53.012 1404] rw------- pihole:pihole 20K FTL-clients
[2020-05-28 09:41:53.012 1404] rw------- pihole:pihole 98K FTL-domains
[2020-05-28 09:41:53.012 1404] rw------- pihole:pihole 45K FTL-strings
[2020-05-28 09:41:53.012 1404] rw------- pihole:pihole 12 FTL-settings
[2020-05-28 09:41:53.013 1404] rw------- pihole:pihole 124 FTL-counters
[2020-05-28 09:41:53.013 1404] rw------- pihole:pihole 48 FTL-lock
[2020-05-28 09:41:53.013 1404] ---------------------------------------------------
[2020-05-28 09:41:53.013 1404] Thank you for helping us to improve our FTL engine!
[2020-05-28 09:41:53.013 1404] FTL terminated!

As for the pihole.log it has a massive amount of dnsmasq queries. 600MB+ for just today.
It's taking quite a while to parse it to see if there's anything outside of those lines. (I will respond if / when my computer finishes reading these)
I realize there's a few SOCKS5 Proxy users (it's a larger network, the unique clients are just the NAT devices poling the data), but I think that even that shouldn't be making logs this large.

4

Hello, I apologize. It turns out I had a runaway server poling my dns at magnitudes per second.
I'm going to attempt a rebuild on this server and the PiHole server just to clear out aynthing.
I'll create a new forum post if the issues reoccur after this.

Thanks for suggestion to look at the logs, they showed a specific client that was just abusing the dns. :slight_smile:

You can mark this as completed.

5

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.