As of this week, Pi-Hole has been continually shutting down in the middle of the day, and using all of the space within the /dev/shm. Rate limiting is disabled at the location and the issue has been persisting with only one of the pi-holes. Is there a way to resolve this issue permanently?
A few thoughts:
You are running an unsupported OS. This is likely unrelated to the problem, but should be noted.
*** [ DIAGNOSING ]: Operating system
[✗] Distro: Almalinux
[✓] dig return code: 0
[i] dig response: "Raspbian=10,11,12 Ubuntu=20,22,23 Debian=10,11,12 Fedora=36,37,38 CentOS=8,9"
[✗] Error: Almalinux is not a supported distro (https://docs.pi-hole.net/main/prerequisites/)
The usage of SHM by Pi-hole is related to how many domains you have in gravity and the query volume received, among other things.
You have zero domains on gravity:
*** [ DIAGNOSING ]: Info table
property value
-------------------- ----------------------------------------
version 15
updated 1700989029
gravity_count 0
Last gravity run finished at: Sun Nov 26 03:57:09 EST 2023
And a message in your diagnosis log reporting a problem withone of your two adlists:
*** [ DIAGNOSING ]: Pi-hole diagnosis messages
count last timestamp type message blob1 blob2 blob3 blob4 blob5
------ ------------------- -------------------- ------------------------------------------------------------ -------------------- -------------------- -------------------- -------------------- --------------------
1 2023-11-30 13:41:06 ADLIST https://mirror1.malwaredomains.com/files/justdomains 2
Likely, Pi-hole was not able to rebuild gravity because of the memory shortage. Your Pi-hole is using quite a bit of memory:
-----tail of FTL.log------
[2023-11-30 13:41:06.677 998/T1795] ADLIST WARNING: Adlist with ID 2 (https://mirror1.malwaredomains.com/files/justdomains) was inaccessible during last gravity run
[2023-11-30 13:41:06.681 998/T1795] Blocking status is enabled
[2023-11-30 13:41:14.533 998M] Resizing "FTL-dns-cache" from 4096 to (512 * 16) == 8192 (/dev/shm: 376.8MB used, 380.4MB total, FTL uses 376.8MB)
[2023-11-30 13:41:14.533 998M] WARNING: RAM shortage (/dev/shm) ahead: 99% is used (/dev/shm: 376.8MB used, 380.4MB total, FTL uses 376.8MB)
[2023-11-30 13:41:45.901 998M] Resizing "FTL-queries" from 375259136 to (6705152 * 56) == 375488512 (/dev/shm: 376.9MB used, 380.4MB total, FTL uses 376.8MB)
[2023-11-30 13:41:45.902 998M] WARNING: RAM shortage (/dev/shm) ahead: 99% is used (/dev/shm: 376.9MB used, 380.4MB total, FTL uses 376.8MB)
The following shows the SHM being used by Pi-hole. FTL-queries is the big hitter.
*** [ DIAGNOSING ]: contents of /dev/shm
total 361M
-rw------- 1 pihole pihole 84K Nov 30 13:38 FTL-clients
-rw------- 1 pihole pihole 248 Nov 30 13:38 FTL-counters
-rw------- 1 pihole pihole 24K Nov 30 13:38 FTL-dns-cache
-rw------- 1 pihole pihole 492K Nov 30 13:44 FTL-domains
-rw------- 1 pihole pihole 88 Nov 30 13:38 FTL-lock
-rw------- 1 pihole pihole 8.0K Nov 30 13:38 FTL-overTime
-rw------- 1 pihole pihole 4.0K Nov 30 13:38 FTL-per-client-regex
-rw------- 1 pihole pihole 360M Nov 30 13:45 FTL-queries
-rw------- 1 pihole pihole 16 Nov 30 13:38 FTL-settings
-rw------- 1 pihole pihole 640K Nov 30 13:44 FTL-strings
-rw------- 1 pihole pihole 308K Nov 30 13:38 FTL-upstreams
How much query volume is your Pi seeing? Please post the outputs of the following from the Pi-hole host OS terminal:
echo ">stats >quit" | nc localhost 4711
echo ">top-clients >quit" | nc localhost 4711
echo ">top-domains >quit" | nc localhost 4711
echo ">top-ads >quit" | nc localhost 4711
Looking at this line, we see that you had 6,705,152 queries in memory at the time this was logged. There is a certain limit what you can do when only offering less than 400 MB of RAM memory (in the form of shm here).
You should really look into increasing this if such a massive amount of queries is expected in your network. If you do not expect such an extraordinary amount of queries (it corresponds to the normal activity of hundreds of individual devices), then you should start investigation if you have a (partial?) DNS loop somewhere
Almalinux on Synology?
Is there a reference document or tool I can use for scaling so I can see how much memory I should give the server?
There is no reference document available. Currently, FTL needs roughly 60 bytes of RAM per query so if you expect 6.7 million queries (as above), this corresponds to 402 MB of needed RAM memory. You will need additional memory for domains and clients but this is typically on the order of only very few MB. Precise numbers could be given but it some work and I don't have the time to do this right now.
And when I say "RAM" above, I actually mean /dev/shm. A good strategy seems to be increasing your /dev/shm to like 1 GB and see how this goes. If this is still not enough, you should really start investigating
If this is a recent observation for a group of Pi-holes that has been running without such hiccups until recently, I'd recommend to scrutinise the amount of DNS requests and investigate the presumably sudden rise of requests before considering to adopt memory usage.
A common reason for an excessive amount of DNS requests would be a DNS loop of sorts, or a client that goes frantic about a domain that is blocked by Pi-hole.
You should exclude any of those or similar root causes before considering to adjust Pi-hole's settings or its RAM quota.
1 Like