FTL not starting, CPU/mem usage high, pihole-FTL.log fills up

Help
1

Greetings,

Reaching out to the help community for some assistance on an issue that occurred this morning and I am unable to resolve. I'm experiencing the same issue on two RaspberryPis with the same PiHole install config. I do use captive DNS on my router to force hard-coded DNS devices thru the PiHole.

Expected Behaviour:

PiHole starts and runs as described, blocks ads, executes DNS queries, etc.

Actual Behaviour:

Web interface has "NaN" across the top, system is non responsive after running restartdns command or flushing logs. Must reboot to get the system to be responsive again. The FTL log seems to be filling up even despite changing my router's DNS to quad 8.

PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian

Debug Token:

Pi-Hole 2 - https://tricorder.pi-hole.net/osMxgLRf/

Thank you!

2

There are a few items shown in your debug log.

Out of memory error in lighttpd:

-rw-r--r-- 1 www-data www-data 2.7K Jan 29 17:36 /var/log/lighttpd/error.log
   -----head of error.log------
...
   2022-01-29 14:04:24: (mod_fastcgi.c.421) FastCGI-stderr: PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 50331656 bytes) in /var/www/html/admin/scripts/pi-hole/php/FTL.php on line 82

You are getting a lot of queries, which is consuming memory, causing rate limits and contributing to your growing logs:

-rw-r--r-- 1 pihole pihole 113M Jan 29 17:48 /var/log/pihole-FTL.log
   -----head of pihole-FTL.log------
   [2022-01-29 00:00:10.740 22974/T22979] Still rate-limiting 10.0.0.1 as it made additional 1197 queries
   [2022-01-29 00:01:10.803 22974/T22979] Ending rate-limitation of 10.0.0.1
   [2022-01-29 00:02:41.927 22974M] Rate-limiting 10.0.0.1 for at least 29 seconds
   [2022-01-29 00:03:10.926 22974/T22979] Ending rate-limitation of 10.0.0.1
   [2022-01-29 00:08:44.121 22974M] Rate-limiting 10.0.0.1 for at least 26 seconds
   [2022-01-29 00:09:10.301 22974/T22979] Ending rate-limitation of 10.0.0.1
...
-----tail of pihole-FTL.log------
   [2022-01-29 17:48:25.128 1676M] Resizing "FTL-queries" from 860389376 to (19558400 * 44) == 860569600 (/dev/shm: 861.4MB used, 2.0GB total, FTL uses 861.4MB)

Why do you have these DNS servers specified? Is unbound running on the Pi shown in this debug log? If so, you should be using the loopback IP (127.0.0.1).

    IPV4_ADDRESS=10.0.0.26/24
    IPV6_ADDRESS=
    PIHOLE_DNS_1=10.0.0.26#5335
    PIHOLE_DNS_2=10.0.0.25#5335

Let's take a look at some of your parameters for query volume, log size, etc. Please post the output of the following commands from the Pi terminal:

echo ">stats >quit" | nc localhost 4711

echo ">top-clients >quit" | nc localhost 4711

echo ">top-domains >quit" | nc localhost 4711

echo ">top-ads >quit" | nc localhost 4711

ls -lha /var/log/pihole*.log*

1 Like
3

Hi jfb,

Appreciate you taking a moment to look into this.

This device is running unbound and I have changed DNS 1 back to loopback and removed DNS 2. After saving, I had to reboot the system as it became non-responsive. Query counts are certainly quite large.

echo ">stats >quit" | nc localhost 4711

domains_being_blocked 98928
dns_queries_today 43561328
ads_blocked_today 2102
ads_percentage_today 0.004825
unique_domains 1791
queries_forwarded 43543875
queries_cached 14885
clients_ever_seen 39
unique_clients 37
dns_queries_all_types 43561328
reply_NODATA 0
reply_NXDOMAIN 0
reply_CNAME 0
reply_IP 0
privacy_level 0
status enabled

echo ">top-clients >quit" | nc localhost 4711


0 43549445 10.0.0.1 MRT001
1 4324 127.0.0.1 localhost
2 2679 10.0.0.5 MSV001
3 1862 10.0.0.10 MDT002
4 1318 10.0.0.225
5 1191 10.0.0.199
6 860 10.0.0.219
7 821 10.0.0.204
8 770 :: pi.hole
9 619 10.0.0.223

echo ">top-domains >quit" | nc localhost 4711

0 20703682 30736.content.swrve.com.mds
1 5274173 233.0.0.10.in-addr.arpa
2 4904746 mpr002.mds
3 3647544 211.0.0.10.in-addr.arpa
4 2786621 p.typekit.net.mds
5 2243308 msv001.mds
6 1391014 203.0.0.10.in-addr.arpa
7 1217614 data.emb-api.com.mds
8 1153198 api.facebook.com.mds
9 93961 wpad.mds

echo ">top-ads >quit" | nc localhost 4711

0 298 trace.svc.ui.com
1 282 inapps.appsflyer.com
2 145 dpm.demdex.net
3 101 www.googleadservices.com
4 79 ssl.google-analytics.com
5 72 smetrics.hrblock.com
6 70 30736.content.swrve.com
7 68 app-measurement.com
8 57 30736.api.swrve.com
9 51 analytics.localytics.com

ls -lha /var/log/pihole*.log*

-rw-r--r-- 1 root   pihole  76K Jan 29 17:48 /var/log/pihole_debug.log
-rw-r--r-- 1 pihole pihole 115M Jan 29 21:49 /var/log/pihole-FTL.log
-rw-r--r-- 1 pihole pihole  23K Jan 29 00:00 /var/log/pihole-FTL.log.1
-rw-r--r-- 1 pihole pihole 2.4K Jan 28 00:00 /var/log/pihole-FTL.log.2.gz
-rw-r--r-- 1 pihole pihole 1.7K Jan 27 00:00 /var/log/pihole-FTL.log.3.gz
-rw-r--r-- 1 pihole pihole 2.8M Jan 29 21:49 /var/log/pihole.log
-rw-r--r-- 1 pihole pihole 1.5G Jan 29 17:13 /var/log/pihole.log.1
-rw-r--r-- 1 pihole pihole 5.4M Jan 29 17:34 /var/log/pihole.log.1.gz
-rw-r--r-- 1 pihole pihole  14M Jan 29 14:12 /var/log/pihole.log.2.gz
-rw-r--r-- 1 pihole pihole 593K Jan 29 00:00 /var/log/pihole.log.3.gz
-rw-r--r-- 1 pihole pihole 608K Jan 28 00:00 /var/log/pihole.log.4.gz
-rw-r--r-- 1 pihole pihole 521K Jan 27 00:00 /var/log/pihole.log.5.gz
-rw-r--r-- 1 pihole pihole 616K Jan 26 00:00 /var/log/pihole.log.6.gz
-rw-r--r-- 1 root   root   1.4K Jan 23 04:56 /var/log/pihole_updateGravity.log
4

Have you exempted the Pi-hole host from above filtering?
Else DNS queries might get trapped in an endless loop.

3 Likes
5

Thanks, deHakkelaar. I did disable the captive DNS rules on the firewall yesterday and numbers significantly dropped. Also did some investigating and may have found a couple of culprits in my TP-Link Kasa smart plugs. Power cycled several of them that seemed chatty to see if helps. Looks like they have a knack for blast requests to ntp servers (to which I added a wildcard whitelisting to my piholes).

I've added an exemption for my piholes and turned captive DNS back on. Will monitor for a bit to see if the situation improves.

1 Like
6

Three days later and all Pi-Holes are stable, I'm not being blasted by DNS queries, and I've done some extra research to modify config files on IoT devices to manually turn of analytics reporting and phoning home. Appreciate the guidance!

1 Like
7

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.