Pihole repeatedly stops working and I get "Lost Connection to API" in the web UI after updating to v5

[travola9]

Expected Behaviour:

DNS is working and pihole web UI shows stats in the dashboard

Actual Behaviour:

DNS not working, and web UI says "Lost Connection to API".
Pihole has been running for months without any issues (been absolutely rock solid), but since updating to v5 it will intermittently stop working and when I go to the web UI I see the "Lost Connection to API" on the dashboard. Rebooting the pi fixes things for a while, running "service pihole-FTL restart" also fixes for a while but eventually it fails again.

I am running it on Raspbian Buster Lite on a Raspberry Pi (I think it is a Model B Revision 2).

Debug Token:

Cannot generate debug token as the upload fails. I manually copied the debug log from the device, as well as the following logs:

/var/log/pihole.log

/var/log/pihole-FTL.log

/var/log/syslog

Logs uploaded to Dropbox here: Dropbox - pihole logs - Simplify your life

Thanks in advance for your help.

[DanSchaper]

[2020-05-17 00:22:59.457 6217] Resizing "/FTL-strings" from 32768 to 36864
[2020-05-17 00:23:02.178 6217] Resizing "/FTL-dns-cache" from 12288 to 16384
[2020-05-17 00:36:58.971 6217] Resizing "/FTL-dns-cache" from 16384 to 20480
[2020-05-17 00:40:45.791 6217] Resizing "/FTL-strings" from 36864 to 40960
[2020-05-17 09:58:41.554 6217] Shutting down...
[2020-05-17 09:58:41.760 6217] Finished final database update
[2020-05-17 09:58:41.794 6217] ########## FTL terminated after 6.078855e+03 s! ##########
[2020-05-17 09:59:30.648 485] Using log file /var/log/pihole-FTL.log
[2020-05-17 09:59:30.655 485] ########## FTL started! ##########
[2020-05-17 09:59:30.655 485] FTL branch: master
[2020-05-17 09:59:30.655 485] FTL version: v5.0
[2020-05-17 09:59:30.656 485] FTL commit: 3d7c095
[2020-05-17 09:59:30.656 485] FTL date: 2020-05-10 18:58:38 +0100
[2020-05-17 09:59:30.656 485] FTL user: pihole
[2020-05-17 09:59:30.656 485] Compiled for arm (compiled on CI) using arm-linux-gnueabihf-gcc (crosstool-NG crosstool-ng-1.22.0-88-g8460611) 4.9.3
[2020-05-17 09:59:30.657 485] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
[2020-05-17 09:59:30.657 485]    SOCKET_LISTENING: only local
[2020-05-17 09:59:30.657 485]    AAAA_QUERY_ANALYSIS: Show AAAA queries
[2020-05-17 09:59:30.658 485]    MAXDBDAYS: max age for stored queries is 365 days
[2020-05-17 09:59:30.658 485]    RESOLVE_IPV6: Resolve IPv6 addresses
[2020-05-17 09:59:30.658 485]    RESOLVE_IPV4: Resolve IPv4 addresses
[2020-05-17 09:59:30.659 485]    DBINTERVAL: saving to DB file every minute
[2020-05-17 09:59:30.659 485]    DBFILE: Using /etc/pihole/pihole-FTL.db
[2020-05-17 09:59:30.660 485]    MAXLOGAGE: Importing up to 24.0 hours of log data
[2020-05-17 09:59:30.670 485]    PRIVACYLEVEL: Set to 0
[2020-05-17 09:59:30.671 485]    IGNORE_LOCALHOST: Show queries from localhost
[2020-05-17 09:59:30.671 485]    BLOCKINGMODE: Null IPs for blocked domains
[2020-05-17 09:59:30.671 485]    ANALYZE_ONLY_A_AND_AAAA: Disabled. Analyzing all queries
[2020-05-17 09:59:30.672 485]    DBIMPORT: Importing history from database
[2020-05-17 09:59:30.672 485]    PIDFILE: Using /var/run/pihole-FTL.pid
[2020-05-17 09:59:30.672 485]    PORTFILE: Using /var/run/pihole-FTL.port
[2020-05-17 09:59:30.673 485]    SOCKETFILE: Using /var/run/pihole/FTL.sock
[2020-05-17 09:59:30.673 485]    SETUPVARSFILE: Using /etc/pihole/setupVars.conf
[2020-05-17 09:59:30.673 485]    MACVENDORDB: Using /etc/pihole/macvendor.db
[2020-05-17 09:59:30.674 485]    GRAVITYDB: Using /etc/pihole/gravity.db
[2020-05-17 09:59:30.674 485]    PARSE_ARP_CACHE: Active
[2020-05-17 09:59:30.674 485]    CNAME_DEEP_INSPECT: Active
[2020-05-17 09:59:30.675 485]    DELAY_STARTUP: No delay requested.
[2020-05-17 09:59:30.675 485]    BLOCK_ESNI: Enabled, blocking _esni.{blocked domain}
[2020-05-17 09:59:30.676 485] Finished config file parsing
[2020-05-17 09:59:30.857 485] Database version is 5
[2020-05-17 09:59:30.858 485] Database successfully initialized
[2020-05-17 09:59:30.920 485] New upstream server: 192.168.1.1 (0/1024)
[2020-05-17 09:59:30.948 485] Resizing "/FTL-strings" from 4096 to 8192
[2020-05-17 09:59:31.139 485] Resizing "/FTL-strings" from 8192 to 12288
[2020-05-17 09:59:31.318 485] Resizing "/FTL-queries" from 229376 to 458752
[2020-05-17 09:59:31.713 485] Resizing "/FTL-queries" from 458752 to 688128
[2020-05-17 09:59:31.785 485] Resizing "/FTL-strings" from 12288 to 16384
[2020-05-17 09:59:32.156 485] Resizing "/FTL-queries" from 688128 to 917504
[2020-05-17 09:59:32.425 485] Resizing "/FTL-strings" from 16384 to 20480
[2020-05-17 09:59:32.670 485] Resizing "/FTL-queries" from 917504 to 1146880
[2020-05-17 09:59:32.749 485] Resizing "/FTL-strings" from 20480 to 24576
[2020-05-17 09:59:33.013 485] Imported 17985 queries from the long-term database
[2020-05-17 09:59:33.022 485]  -> Total DNS queries: 17985
[2020-05-17 09:59:33.022 485]  -> Cached DNS queries: 2334
[2020-05-17 09:59:33.022 485]  -> Forwarded DNS queries: 12932
[2020-05-17 09:59:33.022 485]  -> Blocked DNS queries: 1627
[2020-05-17 09:59:33.023 485]  -> Unknown DNS queries: 1092
[2020-05-17 09:59:33.023 485]  -> Unique domains: 952
[2020-05-17 09:59:33.023 485]  -> Unique clients: 9
[2020-05-17 09:59:33.024 485]  -> Known forward destinations: 1
[2020-05-17 09:59:33.024 485] Successfully accessed setupVars.conf
[2020-05-17 09:59:33.088 510] PID of FTL process: 510
[2020-05-17 09:59:33.094 510] Listening on port 4711 for incoming IPv4 telnet connections
[2020-05-17 09:59:33.096 510] Listening on port 4711 for incoming IPv6 telnet connections
[2020-05-17 09:59:33.096 510] Listening on Unix socket
[2020-05-17 09:59:33.113 510] Reloading DNS cache
[2020-05-17 09:59:33.114 510] Blocking status is enabled
[2020-05-17 09:59:33.202 510] INFO: No regex blacklist entries found
[2020-05-17 09:59:33.204 510] INFO: No regex whitelist entries found
[2020-05-17 09:59:33.322 510] Compiled 0 whitelist and 0 blacklist regex filters in 125.1 msec
[2020-05-17 10:00:50.094 510] Resizing "/FTL-strings" from 24576 to 28672
[2020-05-17 10:08:21.836 510] Resizing "/FTL-dns-cache" from 4096 to 8192
[2020-05-17 10:26:02.491 510] Resizing "/FTL-strings" from 28672 to 32768
[2020-05-17 10:40:09.054 510] Resizing "/FTL-dns-cache" from 8192 to 12288
[2020-05-17 12:04:53.600 510] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-17 12:04:53.600 510] ---------------------------->  FTL crashed!  <----------------------------
[2020-05-17 12:04:53.600 510] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-17 12:04:53.601 510] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-05-17 12:04:53.601 510] and include in your report already the following details:
[2020-05-17 12:04:53.601 510] FTL has been running for 7523 seconds
[2020-05-17 12:04:53.602 510] FTL branch: master
[2020-05-17 12:04:53.602 510] FTL version: v5.0
[2020-05-17 12:04:53.602 510] FTL commit: 3d7c095
[2020-05-17 12:04:53.602 510] FTL date: 2020-05-10 18:58:38 +0100
[2020-05-17 12:04:53.603 510] FTL user: started as pihole, ended as pihole
[2020-05-17 12:04:53.603 510] Compiled for arm (compiled on CI) using arm-linux-gnueabihf-gcc (crosstool-NG crosstool-ng-1.22.0-88-g8460611) 4.9.3
[2020-05-17 12:04:53.604 510] Received signal: Segmentation fault
[2020-05-17 12:04:53.604 510]      at address: 0x612e7564
[2020-05-17 12:04:53.604 510]      with code: SEGV_MAPERR (Address not mapped to object)
[2020-05-17 12:04:53.605 510] Backtrace:
[2020-05-17 12:04:53.607 510] B[0000]: 0x46d704, /usr/bin/pihole-FTL(+0x2c704) [0x46d704]
[2020-05-17 12:04:53.607 510] B[0001]: 0xb6d6e130, /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0xb6d6e130]
[2020-05-17 12:04:53.607 510] B[0002]: 0x4a3b2c, /usr/bin/pihole-FTL(+0x62b2c) [0x4a3b2c]
[2020-05-17 12:04:53.608 510] B[0003]: 0x4a5cfc, /usr/bin/pihole-FTL(main_dnsmasq+0x1238) [0x4a5cfc]
[2020-05-17 12:04:53.608 510] B[0004]: 0x45fd04, /usr/bin/pihole-FTL(main+0xfc) [0x45fd04]
[2020-05-17 12:04:53.608 510] B[0005]: 0xb6d58718, /lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x10c) [0xb6d58718]
[2020-05-17 12:04:53.609 510] ------ Listing content of directory /dev/shm ------
[2020-05-17 12:04:53.609 510] File Mode User:Group  Filesize Filename
[2020-05-17 12:04:53.610 510] rwxrwxrwx root:root 260 .
[2020-05-17 12:04:53.610 510] rwxr-xr-x root:root 4K ..
[2020-05-17 12:04:53.612 510] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-05-17 12:04:53.613 510] rw------- pihole:pihole 12K FTL-dns-cache
[2020-05-17 12:04:53.614 510] rw------- pihole:pihole 29K FTL-overTime
[2020-05-17 12:04:53.615 510] rw------- pihole:pihole 1M FTL-queries
[2020-05-17 12:04:53.616 510] rw------- pihole:pihole 20K FTL-upstreams
[2020-05-17 12:04:53.616 510] rw------- pihole:pihole 643K FTL-clients
[2020-05-17 12:04:53.617 510] rw------- pihole:pihole 66K FTL-domains
[2020-05-17 12:04:53.618 510] rw------- pihole:pihole 33K FTL-strings
[2020-05-17 12:04:53.619 510] rw------- pihole:pihole 12 FTL-settings
[2020-05-17 12:04:53.620 510] rw------- pihole:pihole 124 FTL-counters
[2020-05-17 12:04:53.621 510] rw------- pihole:pihole 28 FTL-lock
[2020-05-17 12:04:53.621 510] ---------------------------------------------------
[2020-05-17 12:04:53.622 510] Thank you for helping us to improve our FTL engine!
[2020-05-17 12:04:53.622 510] FTL terminated!

[DanSchaper]

Thank you for the detailed logs. We'll check it over and follow up with any more needed information.

[travola9]

Thanks for your help. Let me know if any other info needed from me.

[DL6ER]

This is an original dnsmasq bug and likely the same as already reported here

FTL crashed (main_dnsmasq+0x1238)

[travola9]

Just thought I would update that it has not crashed since I the time on those logs (56 hrs ago). Previously it was crashing twice a day every day...

I haven’t changed anything on the pi-hole. The only thing I did was temporarily add an additional DNS entry (1.1.1.1) in the DHCP scope on my router so internet would be unaffected when the pi-hole did crash. After noticing it stopped crashing I changed back to only having the pi-hole as the DNS address and has been ok for the last 24hrs... I’ll continue to monitor.

[DL6ER]

Thanks, any further information is helpful, even when it is not crashing any longer. This at least tells us that you didn't loose interest or something.

[travola9]

I’ll let you know if it crashes again. Let me know if there is anything you want me to try. Thanks for your help.

[elylv]

The issue has just happened again for me, and interestingly, in pihole.log, it is the exact same domain as the one I had before the crash last time:

May 19 14:18:50 dnsmasq[25952]: reply hel-efz.office.com is 2603:1026:7:3c::2
May 19 14:18:50 dnsmasq[25952]: reply hel-efz.office.com is 2603:1026:7:3d::2
May 19 14:18:50 dnsmasq[25952]: reply hel-efz.office.com is 2603:1026:7:52::2
May 19 14:18:50 dnsmasq[25952]: reply hel-efz.office.com is 2603:1026:7:53::2
May 19 14:18:50 dnsmasq[25952]: reply hel-efz.office.com is 2603:1026:7:66::2
May 19 15:22:15 dnsmasq[26918]: started, version pi-hole-2.81 cachesize 10000
May 19 15:22:15 dnsmasq[26918]: compile time options: IPv6 GNU-getopt no-DBus no-UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect ino$

I don't know if that's what's causing the problem, or if it's just a symptom and there's some program on one of my machines that constantly tries to access that domain.

Here's my FTL log from the same crash:

[2020-05-19 12:59:00.335 1138] Notice: Database size is 187.80 MB, deleted 2065 rows
[2020-05-19 13:06:42.768 1138] Resizing "/FTL-dns-cache" from 139264 to 143360
[2020-05-19 13:11:52.521 1138] Resizing "/FTL-strings" from 172032 to 176128
[2020-05-19 13:27:52.914 1138] Resizing "/FTL-dns-cache" from 143360 to 147456
[2020-05-19 13:30:02.430 1138] Resizing "/FTL-strings" from 176128 to 180224
[2020-05-19 13:41:41.229 1138] Resizing "/FTL-dns-cache" from 147456 to 151552
[2020-05-19 13:48:50.355 25541] Note: FTL forked to handle TCP requests
[2020-05-19 14:00:00.298 1138] Notice: Database size is 187.82 MB, deleted 1721 rows
[2020-05-19 14:05:10.577 1138] Resizing "/FTL-strings" from 180224 to 184320
[2020-05-19 14:18:50.597 1138] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-19 14:18:50.597 1138] ---------------------------- FTL crashed!  <----------------------------
[2020-05-19 14:18:50.598 1138] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-19 14:18:50.598 1138] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-05-19 14:18:50.598 1138] and include in your report already the following details:
[2020-05-19 14:18:50.598 1138] FTL has been running for 75924 seconds
[2020-05-19 14:18:50.599 1138] FTL branch: master
[2020-05-19 14:18:50.600 1138] FTL version: v5.0
[2020-05-19 14:18:50.600 1138] FTL commit: 3d7c095
[2020-05-19 14:18:50.600 1138] FTL date: 2020-05-10 18:58:38 +0100
[2020-05-19 14:18:50.600 1138] FTL user: started as pihole, ended as pihole
[2020-05-19 14:18:50.600 1138] Compiled for arm (compiled on CI) using arm-linux-gnueabihf-gcc (crosstool-NG crosstool-ng-1.22.0-88-g8460611) 4.9.3
[2020-05-19 14:18:50.601 1138] Received signal: Segmentation fault
[2020-05-19 14:18:50.601 1138]      at address: 0x7a66652d
[2020-05-19 14:18:50.601 1138]      with code: SEGV_MAPERR (Address not mapped to object)
[2020-05-19 14:18:50.602 1138] Backtrace:
[2020-05-19 14:18:50.604 1138] B[0000]: 0x4d4704, /usr/bin/pihole-FTL(+0x2c704) [0x4d4704]
[2020-05-19 14:18:50.604 1138] B[0001]: 0xb6dd4130, /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0xb6dd4130]
[2020-05-19 14:18:50.604 1138] B[0002]: 0x50ab2c, /usr/bin/pihole-FTL(+0x62b2c) [0x50ab2c]
[2020-05-19 14:18:50.604 1138] B[0003]: 0x50ccfc, /usr/bin/pihole-FTL(main_dnsmasq+0x1238) [0x50ccfc]
[2020-05-19 14:18:50.605 1138] B[0004]: 0x4c6d04, /usr/bin/pihole-FTL(main+0xfc) [0x4c6d04]
[2020-05-19 14:18:50.605 1138] B[0005]: 0xb6dbe718, /lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x10c) [0xb6dbe718]
[2020-05-19 14:18:50.605 1138] ------ Listing content of directory /dev/shm ------
[2020-05-19 14:18:50.605 1138] File Mode User:Group  Filesize Filename
[2020-05-19 14:18:50.606 1138] rwxrwxrwx root:root 260 .
[2020-05-19 14:18:50.606 1138] rwxr-xr-x root:root 4K ..
[2020-05-19 14:18:50.607 1138] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-05-19 14:18:50.608 1138] rw------- pihole:pihole 152K FTL-dns-cache
[2020-05-19 14:18:50.608 1138] rw------- pihole:pihole 29K FTL-overTime
[2020-05-19 14:18:50.609 1138] rw------- pihole:pihole 5M FTL-queries
[2020-05-19 14:18:50.609 1138] rw------- pihole:pihole 20K FTL-upstreams
[2020-05-19 14:18:50.610 1138] rw------- pihole:pihole 643K FTL-clients
[2020-05-19 14:18:50.611 1138] rw------- pihole:pihole 131K FTL-domains
[2020-05-19 14:18:50.611 1138] rw------- pihole:pihole 184K FTL-strings
[2020-05-19 14:18:50.612 1138] rw------- pihole:pihole 12 FTL-settings
[2020-05-19 14:18:50.612 1138] rw------- pihole:pihole 124 FTL-counters
[2020-05-19 14:18:50.613 1138] rw------- pihole:pihole 28 FTL-lock
[2020-05-19 14:18:50.614 1138] ---------------------------------------------------
[2020-05-19 14:18:50.614 1138] Thank you for helping us to improve our FTL engine!
[2020-05-19 14:18:50.614 1138] FTL terminated!
[2020-05-19 15:22:01.900 26915] Using log file /var/log/pihole-FTL.log
[2020-05-19 15:22:01.900 26915] ########## FTL started! ##########
[2020-05-19 15:22:01.900 26915] FTL branch: master
[2020-05-19 15:22:01.900 26915] FTL version: v5.0
[2020-05-19 15:22:01.901 26915] FTL commit: 3d7c095
[2020-05-19 15:22:01.901 26915] FTL date: 2020-05-10 18:58:38 +0100
[2020-05-19 15:22:01.901 26915] FTL user: pihole
[2020-05-19 15:22:01.901 26915] Compiled for arm (compiled on CI) using arm-linux-gnueabihf-gcc (crosstool-NG crosstool-ng-1.22.0-88-g8460611) 4.9.3
[2020-05-19 15:22:01.901 26915] Starting config file parsing (/etc/pihole/pihole-FTL.conf)

[travola9]

Just crashed again for me too.

New logs can be downloaded here:
https://www.dropbox.com/sh/0952cpss8p2d1ze/AADSLga40HSDqsfkUdqBDPXwa?dl=0

Similar to elylv it is the same domain this time that preceded the last crash.

[DL6ER]

Thanks, unfortunately, the normal crash report is lacking some information here, we'll have to wait on the first results from a crashed FTL with a logger attached to it. I will do some more research if we can improve on our embedded crash analysis algorithm.

[travola9]

Is it easy to add a logger? If so, let me know and I can try And set it up to get you more info.

[DL6ER]

@travola9

Is it easy to add a logger?

Yes, fairly easy I'd say at least. I've written a documentation for this, you can find it here:

FTL crashed (main_dnsmasq+0x1238)

In case you need any assistance with this, don't hesitate to ask.

[lukeski14]

Hi, relative pihole noob here. I'm experiencing the same issue as @travola9. I was going to create a new topic, but found this one.

Here's my debug token after restarting the dns service: https://tricorder.pi-hole.net/pntm3b6c1k
I will add another comment here when I experience the crash again with a log.

[yubiuser]

Thanks for sharing.

It would be really helpful if you would attach a debugger to pihole-ftl as written here.

FTL crashed (main_dnsmasq+0x1238)