I just setup pi-hole on my home network and everything went smooth. I updated my DHCP server to issue the pi-hole IP address as the DNS server which caused the clients to slowly starting using the pi-hole as their DNS server. I also enabled the conditional forwarder feature to point to my router so the DNS clients can resolve local DNS names. Around 5:00 AM, my pi-hole system died. When I tried to start it in the morning, it appears to be in an endless loop. The logs indicate the following error before it restarts.
[ Log items removed ... ]
[2021-01-26 10:18:57.192 5101M] FATAL: realloc_shm(): Failed to resize "FTL-queries" (19) to 66846720: No space left on device (28)
[2021-01-26 10:18:57.202 23955M] ########## FTL started! ##########
[ Loads again, but runs out of memory, then repeats ...]
I followed the instructions provided in a similar post to get my pi-hole running again.
sudo service pihole-FTL stop
sudo mv /etc/pihole/pihole-FTL.db /etc/pihole/pihole-FTL-old.db
sudo service pihole-FTL start
ls -lah /etc/pihole/pihole-FTL-old.db
-rw-r--r-- 1 root root 88M Jan 27 01:39 /etc/pihole/pihole-FTL-old.db
Here is the output of the temp fs:
root@pihole-pihole1:/var/log# df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
shm 64M 592K 64M 1% /dev/shm
Does anyone have any ideas on how to deal with this memory issue?
Pi-hole makes efficient use of memory and uses little memory to begin with, so I doubt it's a memory issue per se.
My prime suspect would be a (partial) DNS loop, and Conditional Forwarding may have introduced this - if your router also would be using Pi-hole's IP address as its upstream DNS server.
See if disabling CF would solve your issue, and check your router's upstream DNS configuration.
Let us know if your problem persists, and share a debug token in that case.
I think I found the issue. I have a Zabbix server that's monitoring my internal network. It was responsible for > 90% of the DNS queries within my network. The pi-hole was recording over 30,000 DNS queries per hour from that system alone. I enabled a local resolver cache on the Zabbix server and enabled a 300 second TTL for DHCP leases which results in just a few thousand DNS queries per hour. The memory usage of the temp file system is now a few hundred KB. I'll report back if it fails again. Thank you.
I attempted to migrate my DNS to the pi-hole running in a docker container again with conditional forwarding disabled. It crashed again after running for about 12 hours. I am seeing about 20,000 queries per hour, mostly originating from a single client (the Zabbix server). My clients are configured to query the pi-hole, and the pi-hole is configured to query opendns. Any suggestions?
My suggestion would be: Add a file /etc/pihole/pihole-FTL.conf with:
DEBUG_ALL=true
and restart your Pi-hole. When it crashes again, provide the last maybe 100 log lines from /var/log/pihole-FTL.log so the developers can see what is crashing and where.
20,000 requests per hour is way too excessive for a tool that monitors your local network (unless that network would indeed accomodate a few thousand clients).
As your observation seems specific to your network monitoring server, that would suggest that Zabbix could be trying to resolve hostnames unnessarily on each single request instead of caching the results.
My advice would be to look for ways to control the amount of DNS queries in that Zabbix software.
I am not familiar with Zabbix at all, but I'd look into using IPs for contacting/polling your network instead of names, and maybe increase polling intervals.
I implemented local caching on the Zabbix server which reduced the queries to less than a hundred per hour. I also moved my pi-hole to a Ubuntu VM. It's been humming along for a few days now without any issues.