When pihole-FTL starts, the number of domains loaded from the lists reflects the number of domains actually contained in the lists.
Actual Behaviour:
After upgrading to v4.0, I noticed that the memory usage on the raspberry pi was higher than usual. The user interface shows the correct number of domains loaded (gravity.list + black.list), however when pihole-FTL loads, the following is given:
Aug 15 22:30:04 dnsmasq[24567]: read /etc/hosts - 10 addresses Aug 15 22:30:04 dnsmasq[24567]: read /etc/pihole/local.list - 2 addresses Aug 15 22:30:04 dnsmasq[24567]: read /etc/pihole/black.list - 24 addresses Aug 15 22:30:48 dnsmasq[24567]: read /etc/pihole/gravity.list - 5359020 addresses
so it seems that pihole-FTL is somehow duplicating the lists internally and using up memory to store these duplicate entries. Can someone else verify this behaviour? I was on the beta branch and switched to master after discovering the increased memory issue, but it's on the master branch as well. Pi-hole Version v4.0 Web Interface Version v4.0 FTL Version v4.0
First of all: This is not a bug. The memory consumption of pihole-FTL is higher than with Pi-hole pre-v4.0. This is caused by dnsmasq now being part of pihole-FTL and should be compensated by dnsmasq not being present any longer. The query datastructure grew slightly because we can now analyze and store more details but the overall effect on memory should not be more than maybe 5-10%.
Let me explain what happens and it will immediately become obvious why you see a doubled count in pihole.log while the numbers in pihole-FTL.log and the dashboard are correct:
For each blocking mode (except NXDOMAIN), we have to create two cache entries - one for the A (IPv4) and one for the AAAA (IPv6) reply. If you chose NULL this will be
A: 0.0.0.0
AAAA: ::
If you chose IP, it will be the IPs of your Pi-hole. As you can see, we always need to create two cache entries for each domain to cover both, IPv4 and IPv6 lookups. The numbers that are shown in the log,
are determined by subtracting the cache size before and after the list parsing, hence an injection of two records per domain makes the resolver think that the list contained the double amount of domains (although it didn't).
Side note: An exception to this is the blocking mode NXDOMAIN where we only store
ANY: (NXDOMAIN)
hence, there is no double count in this mode and the memory consumption is lower as the cache is only half of the size.
I should point out that this duplication was exactly the same for Pi-hole pre-v4.0 as we, instead of storing only the domains in a simple format, always added two lines per domain (one A and one AAAA) record in the list. So, back then, wc -l would have actually shown you a doubled count of domains.
TL;DR: The fact that pihole.log shows a double amount is not a bug, it is a technical details. The reason for this is explained in this post.
I've noticed a near 30% increase in memory usage by pihole-FTL, however I admit my block list is quite large.
That's strange since I've been testing piholeFTL since beta and didn't notice the increased memory usage until the upgrade.
I appreciate the detailed explanation. The increased memory usage had me concerned. This can be added to the release post for v4.0 as it wasn't mentioned there. Thanks.
Hi, So the problem of mrzephyrus I can only confirm. After a certain update of FTLDNS Beta, the Ram consumption has doubled and has since been the same. I have previously had ~ 31% utilization on 4 million entries and now it is ~ 70%. I can even provide a screenshot, but I can not generate a debug code anymore.
During the beta phase, we experimented with various blocking methods. Please try using the NXDOMAIN blocking mode and see if this gives you again the ~ 31% memory usage. See Blocking mode - Pi-hole documentation
You need to restart / reload (SIGHUP) pihole-FTL to enable the change.
Hey, thank you for the reply. I have just added the blocking mode NXDOMAIN and entered the command SIGHOL pihole -ftl but also execute from pihole -g nothing has changed. I had to restart the pihole after the commands and now I'm at 38% Ram utilization. Thank you! Finally a solution
I also have another problem but I could also send you a message in German.
