FTL V4.2.2/4.2.3 consumes nearly 2x more memory then V4.11

Help
1

Please follow the below template, it will help us to help you!

Expected Behaviour:

The memory usage of PiHole 4.22 must be the same as 4.11

Actual Behaviour:

Nearly 2 times more memory needed by FTL

Debug Token: l2iogeui0w

Today I have updated my PiHole from 4.11 to 4.22 (latest version). No problems with updating.

Because I knew the memory problem because I have done a lot of experiments with another PiHole (see below), I have taken print screens before and after update for showing the diffrence.
The FTL proces uses 341 MB in version 4.11, now it’s 616 MB. Nearley 2 times more. Nothing is changed in configuration just before and after the update. This PiHole is running on a RaspberryPi3 with Raspbian Stretch Lite and all updates. The Pi is operational without further problems.
My pihole-FTL.conf looks like:

AAAA_QUERY_ANALYSIS=no
MAXDBDAYS=365
API_EXCLUDE_DOMAINS=*.workgroup
BLOCKINGMODE=IP
DBINTERVAL=5.0
RESOLVE_IPV6=no
PRIVACYLEVEL=0

Screen pictures before updating:

1074×1121 92.1 KB

875×425 58.6 KB

1203×953 208 KB

The pictures after updating:

1074×1121 93.5 KB

731×617 74 KB

1051×937 200 KB

My family has a PiOne, running PiHole 4.11 on a actual Armbian (Stretch lite).
At my home I also have a PiOne for experiments. I take the image of the PiOne of my family and restore it on my own PiOne. The only difference is that I have to disable DHCP on my PiOne afterwards. After updating my PiOne to v4.22, the same memory issue. Before update FTL uses 371 MB. Because the PiOne has 500 MB RAM on board, I have to reduce the blocklist from 3.6 Million to 2 M. to get it running. Now FTL uses 357 MB (total memory usage, displayed in admin webpage = 79.6%, was in V4.11 75,7% ). Because it’s not in use, no network traffic uses this PiHole (no queries only them from PiOne himselve).

I don’t have IPv6.
More that a year ago I have choosen for blockingmode = IP, because in that case each domain is stored in memory only once to reduce memory usage. Is this changed in V4.22, so stored 2 times? Why, because only blocked domains are enough; blocking mode can applied realtime from the config file. Or is it a bug that in case of RESOLVE_IPV6 .AND. BLOCKINGMODE=IP still stores IPv4 and Ipv6 addresses in memory for each domain?

Otherwise: what is going wrong with FTL and their memory usage in V4.22?

2

Used memory: 63.80 MB 680k domians in list

1 Like
3

Install update 4.2.3 for FTL and see if it is fixed now.

4

In the test above I already have installed FTL 4.23 on my PI3 during the update from 4.11. I thought that 4.22 was the actual version but in the meantime it's 4.23. I have changed the topic title.

As mentioned above I also have a PiOne for tests and experiments. That one has FTL 4.22. So I have made some print screens, do the update and take new print screens.
There is no difference in memory usage (a little higher because about 1800 domains more on block list).

So FTL 423 is no solution.

Before updating:

1075×1121 83.7 KB

After the update

1139×1074 88 KB

1003×1033 212 KB

5

after i tried this my pihole went from 60% ram to >93% ram,
deleting this entry did not help to get back to normal-behavior..

6

You may not delete this line, because this
RESOLVE_IPV6=yes|no.
The first mentioned entry is the default.
So deleting this line means that also IPv6 addresses are stored in memory, which gives extra RAM usage.
See Configuration - Pi-hole documentation

I think when you define RESOLVE_IPV6=no you should go back to the normal behaviour

7

this line was not in my conf so i thought deleting may solve smthng..
but did not.
so i tried all my entries - and the reason was 'BLOCKINGMODE=IP' that brings my RAM from 60% to >92%

8

OK. Let's put this problem to rest. I took one for the team and gave up over an hour of my life to configure a Pi-3B+ running Stretch full latest and Pi-Hole latest with your white, black, regex and blocklists. Here are the results:

Building gravity from your 130 block lists took about 7 minutes, and they did not all load. The last list in your adlists.list (tspprs.com) would not download and could not be loaded directly from that URL, so you are likely using a cached copy of this list.

So, my Pi loaded the following:

  [i] Number of domains being pulled in by gravity: 4749693
  [✓] Removing duplicate domains
  [i] Number of unique domains trapped in the Event Horizon: 2903071
  [i] Number of whitelisted domains: 144
  [i] Number of blacklisted domains: 39
  [i] Number of regex filters: 175

Dashboard shows 2.9 million domains on blocklist, 45.7% memory used. Plenty of memory left.

free
              total        used        free      shared  buff/cache   available
Mem:         961628      371760      106012       23628      483856      500972
Swap:        102396        4096       98300

A second pihole -g also failed to load the last tspprs list, so the results were the same as above.

I took a close look at the domains contained in /etc/pihole/gravity.list. In my opinion, there is a huge amount of junk in there. I'm not sure what your selection criteria were for using this selection of blocklists; just because a list maintainer puts a domain on their list doesn't add any quality. As an example, the tspprs lists are generally poorly curated and have a number of problems with false positives.

Take a look for yourself and see what's in your gravity list.

As a quick test, I grabbed 10 random domains from the middle of the gravity list, ran a batch dig on these to Cloudflare (bypassing Pi-Hole), and they all returned NXDOMAIN (they don't exist on the internet). If the first 10 I chose at random were NXDOMAIN, then this indicates that there are many more like this.

0070wv056sv7gzg1kvi.co.ve
0070wv056sv7gzg1kvi.com.ve
0070wv056sv7gzg1kvi.info.ve
0070wv056sv7gzg1kvi.net.ve
0070wv056sv7gzg1kvi.org.ve
0070wv056sv7gzg1kvi.web.ve
0070zw2sqpfz4cahq45.co.ve
0070zw2sqpfz4cahq45.com.ve
0070zw2sqpfz4cahq45.info.ve
0070zw2sqpfz4cahq45.org.ve
0070zw2sqpfz4cahq45.web.ve

My advice - stop worrying about why memory usage has changed and put more time into thoughtful selection of well curated blocklists, or even better, use the seven standard lists and then fine tune locally with white and black lists as needed.

Note that when I ran your configuration, I had problems with a number of sites (Reddit, for example). I couldn't restore my original configuration fast enough.

When I went back to my original configuration, with the WaLLy3K ticked lists and my white, black and regex lists, 714K domains on blocklist, 32.6% memory used, and a much better user experience (and no ads)

 free
              total        used        free      shared  buff/cache   available
Mem:         961628      210272       87136       26604      664220      659428
Swap:        102396         768      101628
2 Likes
9

Of note, i am running blocking mode = NULL, so our configuration differs here.

11

Thank you for all your tests.

With 2.7 million domains the memory usage is 93% on a PiOne (with 500 MB RAM) This is with Blockingmode=IP.
I changed the Blockingmode to NULL, did a power down and up (to eliminate the fixed memory allocation of FTL), and now the memory usage = 60.4%
HTOP shows 277 MB used memory for FTL (this was 471 MB with Blockingmode=IP)

I did a pihole -g -f to get all actual files. With my local stored files, defined in the addlist, I now have 3.3 million blocked domains and 68% memory usage. HTOP shows that FTL uses 319MB.

          total        used        free      shared  buff/cache   available

Mem: 505064 326872 54468 580 123724 167324
Swap: 252528 48164 204364

Indeed the https://tspprs.com/ site is offline. And I will put the .ve toplevel domain in the regex file.

The strange effect is/was that with FTL4.23 with Blockingmode=IP uses nearly 2 times more memory that FTL 4.11 without any change in configuration or addlist.
Now changing the blocking mode reduce the memory usage.

In the meantime I have made a clean-up script for the gravity.list and run this on my own grafity.list file.
It removes all domains with illegal characters in it and also removes IP addresses like 199.172.228.6 and also like 199.193.232.43-static.reverse.softlayer.com.
It also uses the 'iconv -f utf-8 -t utf-8' command of msatter.
This kind of domains are not removed: живи-просто.рф
They all are at the end of the list.
Maybe a 'iconv -f utf-8 -t ISO-8859-1' can be used better in this case.

The cleanup-script (with iconv .........> tempfile is 1 line) is:

#!/bin/bash
iconv -f utf-8 -t utf-8 -c /etc/pihole/gravity.list | sed '/[`~!@#$%^&()=+{}";:?/><,|]/d; /'''/d; s/^\s+[ \t]//g; s/\s+[ \t]$//g; /^\s$/d; / /d' | grep -E -v "(([0-9]{1,3}[.]){3}[0-9]{1,3})" > tempfile
rm /etc/pihole/gravity.list
mv tempfile /etc/pihole/gravity.list
sudo service pihole-FTL restart

I'am not a programmer; maybe others can do it better. The grep command is the slowest part. But I cannot get a working sed replacement for this.

I will review my blocklists. Thanks again.

13

This seems like a lot of work on your part to clean up garbage in subscribed block lists. It would seem easier to use fewer/better block lists and avoid this work. Why clean up somebody else's crap?

16

I have to use a tempfile. The script stops with a gravity file of 0 bytes. Result: all illegal characters are gone, also живи-просто.рф

But the IPaddres are still in the the result file. Any idea to make this grep part faster?
grep -E -v “(([0-9]{1,3}[.]){3}[0-9]{1,3})”

18

The file is smaller (73.111 KB; was 73.199 KB), but it doesn't work

There are still IPaddresses in the file like
193.151.89.76
193.17.41.98
193.190.196.104.bc.googleusercontent.com
193.2.115.196
193.2.129.159
193.226.227.73
193.226.238.82.pool.invitel.hu
193.226.239.49.pool.invitel.hu
193.226.38.28

Used command:
iconv -f utf-8 -t utf-8 -c /etc/pihole/gravity.list | awk /[1]+$/ | awk '!/[0-254.]$/' > tempfile

  1. a-zA-Z0-9._- ↩︎

19

Specifically sed -nr -e 's/\.{2,}/./g' -e '/\./p' > ${destination}

23

Back at home I have done the test again.

Sorry to tell that awk '!/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$/' doesn't work. The IP addresses remain still in the gravity file. I don't why because it looks that it should work.

I'll tried several other possibilities like
awk '!/^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/',
awk '!/^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$/' and
awk '!/([0-9]{1,3}[\.]){3}[0-9]{1,3}$/' but the IP's are still in the file.

Also sed -e '/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$/ d' doesn't work.

The grep command, mentioned above works perfect, but is very slow in handling the file.

What is the reason for using awk '!a[$0]++'? What will this do?
Looking at this text processing - How does awk '!a[$0]++' work? - Unix & Linux Stack Exchange, it should for removing duplicated lines. But this already is done in the 'pihole -g' process and not necessary.

26

I thought ! is 'NOT match'.

awk '/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$/' < /etc/pihole/gravity.list > tempfile
gives an empty file.

sed -e '/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$/p' < /etc/pihole/gravity.list > tempfile
gives the complete gravity.list file

Removing the ;1 works using sed 's/\;1//'
Later I thought, it's only end of line: sed 's/\;1$//' ; but not tested yet.

28

Sorry to tell but your command gives in my piOne (Armbian Stretch) and PI3 (Raspian lite Stretch) an empty result file.
awk '/^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$/' < /etc/pihole/gravity.list > tempfile
or the same:
awk NF /etc/pihole/gravity.list | awk '/^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$/' > tempfile

The command below gives only IP addresses in result file. Maybe all (78 KB).
awk NF /etc/pihole/gravity.list | awk '/^[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*$/' > tempfile
This can be the solution if this also works on your site

Reason that it works on your PC with Linux/Pi3 (?) and not my PI3/Armbian OS is the --re-interval option of AWK.
See f.i. regex - Does GNU awk accept intervals specified using braces in regular expressions? - Ask Ubuntu ;
Awk regular expression syntax with number of repetition - different handling between gawk 3 and gawk 4 - Unix & Linux Stack Exchange
The GNU Awk User’s Guide
Here ("awk '/[0-9]{4}/' file" not works, why ? - Shell Programming and Scripting - Unix Linux Community) is a question with a solution, but it's not working on my Pi3/PiOne systems.
And I cannot view my AWK version number (command = awk --version)

I have tried the other following command with the results below the command.
awk NF /etc/pihole/gravity.list | awk '/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}$/' > tempfile
Empty result file

awk NF /etc/pihole/gravity.list | awk '!/^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$/' > tempfile
All domains + IP's in result file

awk NF /etc/pihole/gravity.list | awk '/^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$/' > tempfile
Empty result file

awk NF /etc/pihole/gravity.list | awk '/[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}*$/' > tempfile
An empty result file

awk NF /etc/pihole/gravity.list | awk '/^[0-9]\.[0-9]\.[0-9]\.[0-9]$/' > tempfile
result file with only 0.0.0.0

awk NF /etc/pihole/gravity.list | awk '/[0-9]\.[0-9]\.[0-9]\.[0-9]$/' > tempfile
Result file with 0.0.0.0, 24.5.5.0, st.adxxx.o0.0.0.0

awk NF /etc/pihole/gravity.list | awk '/[0-9]\.[0-9]\.[0-9]\.[0-9]{1,3}$/' > tempfile
An empty result file

awk NF /etc/pihole/gravity.list | awk '/[0-9]{3}\.[0-9]{3}\.[0-9]{3}\.[0-9]{2}$/' > tempfile
An empty result file while 198.111.182.16 is in the gravity.list file

awk NF /etc/pihole/gravity.list | awk '/[0-9]{1,3}\.[0-9]\.[0-9]\.[0-9]$/' > tempfile
An empty result file

awk NF /etc/pihole/gravity.list | awk '/[0-9]*\.[0-9]\.[0-9]\.[0-9]$/' > tempfile
Result file with 0.0.0.0, 24.5.5.0, st.adxxx.o0.0.0.0

awk NF /etc/pihole/gravity.list | awk '/[[:digit:]]\.[[:digit:]]\.[[:digit:]]\.[[:digit:]]$/' > tempfile
An empty result file

Regarding the sed command
sed -e '/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$/p' < /etc/pihole/gravity.list
gives all domains including IP addresses on my system.

sed -e '/^[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*$/p' < /etc/pihole/gravity.list
Also gives all domains & IP's on my system.

29

went down to 60%,
but now cant whitelist directy on block..
and dhcp still not working ;(

30

went / moved to https://blocklist.site/app

31

But the lists appear to be as dodgy as before. Two screen shots of domains added to their lists in error, and then removed. This has been a recurring problem with that group of lists.

image
image1300×796 52.8 KB

image
image1286×690 55.5 KB