@bcambl This probably isn't related, but this is CentOS 7, can you give it a sanity check?
The more prior posts I'm reading the more it looks like I'm running out of shared memory space. I certainly can add more - but that probably will just delay the inevitable. How much is enough? 
I have some very chatty servers that make a ton of requests repeatedly:
# echo ">top-clients" | nc localhost 4711 -w1
0 9474471 xx app04.
1 3581851 xx web03
2 129169 xx attic
3 96880 xx appd02
4 35113 xx appd05
Appreciate the visibility on this.
Thanks!
Without trying to be flippant, enough memory to keep things from crashing. It looks like you have approx. 1G allocated, which follows convention of 1/4th the RAM. Even then, that's a large amount of memory. Throwing more SHM at it is going to be a band-aid solution.
I suppose that was my thinking as well. If 1GB isn't enough - why would 2GB or 3GB be enough? Why isn't it releasing it back? When does it get "full" - since the memory requirements seem to be "only need 512MB on your server" yet I'm at 4GB and having an issue after 12 hours. Does 8 GB get me 24 hours before it crashes? I just want to avoid the crash 
Do I need enough to get past 24 hours and some memory clean up happens?
In the last hour:
# df /dev/shm
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 1940108 956324 983784 50% /dev/shm
Thanks!
It doesn't release it back, it's storing queries. The more queries you throw at it, the more memory you will need.
If you want a true solution to your problem, find out why you have clients with near 15 million queries.
I know why - I'm polling 150 devices every 10-30 seconds for a bunch of IOT metrics ingestion to InfluxDB across a handful of APM testing efforts I'm working. Is it a matter of just excluding that one collector that's disproportionally hammering the number of requests it's making for DNS lookups? Does it hold every query in memory based on:
MAXDBDAYS=365
If so - setting this to 0 still had me crashing previously.
I don't need to hold onto them for a year - maybe just a day.
I bumped up the memory to the server to 16 GB of memory, shared went to 8 GB:
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 8133260 97584 8035676 2% /dev/shm
And:
MAXDBDAYS=1
Will report back if that gets me back to safety. Thanks for the guidance.
No. Only the most recent 24 hours of activity are in RAM. The long term database is the sole repository for the data more than 24 hours old.
1 Like
Adding more memory has helped. Still online:
-rw------- 1 pihole pihole 81920 May 18 20:26 FTL-clients
-rw------- 1 pihole pihole 124 May 18 17:33 FTL-counters
-rw------- 1 pihole pihole 163840 May 19 16:25 FTL-dns-cache
-rw------- 1 pihole pihole 196608 May 19 15:01 FTL-domains
-rw------- 1 pihole pihole 48 May 19 10:13 FTL-lock
-rw------- 1 pihole pihole 12288 May 19 10:13 FTL-overTime
-rw------- 1 pihole pihole 4096 May 18 17:33 FTL-per-client-regex
-rw------- 1 pihole pihole 2159542272 May 19 15:59 FTL-queries
-rw------- 1 pihole pihole 12 May 18 17:33 FTL-settings
-rw------- 1 pihole pihole 151552 May 19 15:36 FTL-strings
-rw------- 1 pihole pihole 4096 May 18 17:33 FTL-upstreams
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 8132956 2109472 6023484 26% /dev/shm
I should be crossing that 24 hour time period in a few more hours for that RAM activity to settle down.
[2020-05-19 16:00:03.411 8542] Notice: Database size is 2723.14 MB, deleted 1195129 rows
[2020-05-19 16:25:12.749 8542] Resizing "/FTL-dns-cache" from 159744 to 163840
[2020-05-19 17:00:03.120 8542] Notice: Database size is 2723.14 MB, deleted 1410599 rows
[2020-05-19 17:57:18.632 8542] Resizing "/FTL-strings" from 151552 to 155648
[2020-05-19 17:58:03.037 8542] Resizing "/FTL-queries" from 2159542272 to 2159804416
[2020-05-19 17:58:10.757 8542] Resizing "/FTL-queries" from 2159804416 to 2160066560
[2020-05-19 17:58:30.517 8542] Resizing "/FTL-queries" from 2160066560 to 2160328704
[2020-05-19 17:58:33.178 8542] Resizing "/FTL-queries" from 2160328704 to 2160590848
[2020-05-19 17:58:45.561 8542] Resizing "/FTL-queries" from 2160590848 to 2160852992
[2020-05-19 17:59:02.954 8542] Notice: Database size is 2723.14 MB, deleted 1360720 rows
Made it far enough along to consider this no longer an issue:
[2020-05-20 15:00:06.480 8542] Notice: Database size is 2723.14 MB, deleted 1411391 rows
[2020-05-20 15:55:20.444 8542] Resizing "/FTL-queries" from 2261778432 to 2262040576
[2020-05-20 15:55:32.485 8542] Resizing "/FTL-queries" from 2262040576 to 2262302720
[2020-05-20 15:55:40.125 8542] Resizing "/FTL-queries" from 2262302720 to 2262564864
[2020-05-20 15:55:50.500 8542] Resizing "/FTL-queries" from 2262564864 to 2262827008
[2020-05-20 15:56:02.657 8542] Resizing "/FTL-queries" from 2262827008 to 2263089152
[2020-05-20 15:56:10.043 8542] Resizing "/FTL-queries" from 2263089152 to 2263351296
[2020-05-20 15:56:20.420 8542] Resizing "/FTL-queries" from 2263351296 to 2263613440
[2020-05-20 15:56:30.478 8542] Resizing "/FTL-queries" from 2263613440 to 2263875584
[2020-05-20 15:56:40.004 8542] Resizing "/FTL-queries" from 2263875584 to 2264137728
[2020-05-20 15:56:50.381 8542] Resizing "/FTL-queries" from 2264137728 to 2264399872
[2020-05-20 15:57:02.608 8542] Resizing "/FTL-queries" from 2264399872 to 2264662016
[2020-05-20 15:57:06.821 8542] Resizing "/FTL-queries" from 2264662016 to 2264924160
[2020-05-20 15:57:20.418 8542] Resizing "/FTL-queries" from 2264924160 to 2265186304
[2020-05-20 15:57:30.503 8542] Resizing "/FTL-queries" from 2265186304 to 2265448448
[2020-05-20 15:57:36.877 8542] Resizing "/FTL-queries" from 2265448448 to 2265710592
[2020-05-20 15:57:50.356 8542] Resizing "/FTL-queries" from 2265710592 to 2265972736
[2020-05-20 15:58:00.514 8542] Resizing "/FTL-queries" from 2265972736 to 2266234880
[2020-05-20 15:58:04.368 8542] Resizing "/FTL-queries" from 2266234880 to 2266497024
[2020-05-20 15:58:20.329 8542] Resizing "/FTL-queries" from 2266497024 to 2266759168
[2020-05-20 15:58:30.448 8542] Resizing "/FTL-queries" from 2266759168 to 2267021312
[2020-05-20 15:58:35.113 8542] Resizing "/FTL-queries" from 2267021312 to 2267283456
[2020-05-20 15:58:50.379 8542] Resizing "/FTL-queries" from 2267283456 to 2267545600
[2020-05-20 16:00:06.510 8542] Notice: Database size is 2723.14 MB, deleted 1398303 rows
Every hour it seems to be purging out whatever it needs for the 24 hour bucket and it hasn't crashed any more.
Thanks!
This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.