Problem since update to 4.2.*

Help
1

Expected Behaviour:

working as normal .. memory-usage around 60% as in 4.1.*

Actual Behaviour:

at 4.2.1: memory-usage >90% - laggy, but working..
at 4.2.2: memory-usage >95% until pihole-FTL service crashes

Debug Token:

pi3b+ / 4.2.2: djtn0l4iqu

additional facts:

i have 2 quite the same piholes:
they have the same upstream-DNS', same adlists whitelist blacklist, same dhcp-reserved-leases.
they both run hostapd.
they have different IPs (whoa), different dhcp-pools and run on different hardware: raspi3b and raspi3b+
the mentioned adlists is 6 KB - after consolidating around 6 mio blocked domains.

running pihole 4.1.x on pi3b without any problems - 60% mem-usage
running pihole 4.2.1 on pi3b+ was laggy but ran,
updated pi3b+ to 4.2.2 hours ago today: now it cannot start pihole-FTL, it crashes after minutes of >95% mem-usage.
doing here 'htop' flickers.. also the ssh-session lags.. (needs seconds for reaction, even key-input)
i deleted all but one list in adlists.list, ~1mio blocked domains, pi3b+ at 4.2.2 is at 30% mem-usage and works fine... but not with all i want to have blocked..

done so far:
https://www.reddit.com/r/pihole/comments/an0o06/mem_usage_after_update/
https://www.reddit.com/r/pihole/comments/aox4an/downgrade_pihole_to_version_before_feb2019/

question

with how many blocked domains are new releases tested?

2

It would help us to see the output from top . This will show where the memory is being used.

3

This debug log shows that your gravity list is empty.

*** [ DIAGNOSING ]: Gravity list
   -----head of gravity.list------
   -----tail of gravity.list------
4

sry, thats part of my fault - its a debug while loading pihole-FTL with full adlists.list
but it was never loaded..

here is a debug token with only the first entry in my adlists.list
swe0avvxwe
here it is noT empty.

5

This debug log looks normal, with one exception. There appear to have been local changes made to the web version:

*** [ DIAGNOSING ]: Web version
[i] Web: v4.2 (https://discourse.pi-hole.net/t/how-do-i-update-pi-hole/249)
[i] Branch: master
[i] Commit: v4.2-0-g347994db-dirty
6

Note that this many domains on blocklist is going to tax any Pi.

7
[Album] imgur.com

highest seen for pihole-FTL was 91,7 .. at this moment web-dashboard needed 5mins to load and showed >98% mem-usage, key-press needs 1-5 full seconds to see in ssh-session..
thats mostly when i kill it and reboot after deleting gravity.list and shorten adlists.list..

8

not on (from around 3.2/3.3 to) 4.1.2 .. my pi3b is with the same adlists on that version at 60% mem-usage..
as i read in the update notes to 4.2 wasn't there smthng about 'uses less mem' ?

9

What is the top output from the Pi with lower memory load? And, which of the two Pi's receives the majority of the DNS queries?

10

hm ok - seems to be smthng from my downgrading, thought i brought it all back..
pihole -up does nothing change, as the versions seem up2date,
how do i get rid of that?

11

this is 4.1.2-dirty whatever

[Album] imgur.com

they get around the same queries, now mostly this one, the 3b.
before 4.2 the 3b+ was mostly faster for a dhcp-lease (..who gives the lease gives himself as prim-dns and the other as sec-dns)

12

cd/var/www/html/admin

git status to see what is modified.

sudo git reset --hard HEAD to reset the files, and then you should be able to update.

13

done .. there was nothing to update..

update: kkz494zora
dont know why, but now gravity.list is not found (and for that failed to load names) during debug-creation - but its there ..

14

This debug looks normal and the dirty commit is cleared. Gravity shows 1029127 domains.

15

so .. why is the ram-usage then so high / this much higher than at version 4.1.x with my real adlists.list ?

16

any ideas why ram-usage is that high as 4.2.x was released with 'uses less ram' ?

17

I do not know. Perhaps one of the developers will have some thoughts.

18

One million domains is a lot. We need some more information to trace why so much memory is used:

  • What are the typical numbers of your Pi-hole (Queries and number of clients within the last 24 hours)
  • What device are you running Pi-hole on?
  • What blocking mode are you using (or default?)

Please also provide the output of

ls -lh /dev/shm/FTL-*
19

1million is nothing, until v4.2 6mio were no problem..

3k-6k queries per pihole and per day
10 clients
pihole 4.1.x runs on 3b with 6mio blocked domains
pihole 4.2.2 runs on 3b+ with 1mio blocked domains (deleted adlists to have it run and not stuck)
both blocking mode: IP

output on pi3b: nothing, no such file or directory
output on 3b+:
pi@PiHole2:~ $ ls -lh /dev/shm/FTL-*
-rw------- 1 pihole pihole 2.5M Feb 22 23:36 /dev/shm/FTL-clients
-rw------- 1 pihole pihole 108 Feb 22 23:36 /dev/shm/FTL-counters
-rw------- 1 pihole pihole 128K Feb 22 23:36 /dev/shm/FTL-domains
-rw------- 1 pihole pihole 160K Feb 22 23:36 /dev/shm/FTL-forwarded
-rw------- 1 pihole pihole 28 Feb 22 23:37 /dev/shm/FTL-lock
-rw------- 1 pihole pihole 8.0K Feb 22 23:36 /dev/shm/FTL-overTime
-rw------- 1 pihole pihole 448K Feb 22 23:59 /dev/shm/FTL-queries
-rw------- 1 pihole pihole 36K Feb 22 23:36 /dev/shm/FTL-strings

edit:
i just dont get what has changed.. i used to have the same adlists.list on both piholes..
version 4.1.2 on my pi3b runs with 60% mem including kodi..
v4.2.1 on my pi3b+ was over 90% ram-usage, but was running..
v4.2.2 on my pi3b+ needs more ram than the hardware gives.. and can never load pihole-FTL ..

edit2: i could give out my adlists.list to have it tested by yourself..
when i have time .. i will setup a new pihole with my big adlists.list and have a look if i misconfigured smthng..

21

Well, I disagree about the "nothing", but I agree on that it should be possible. I'm not aware of a change that could have made the situation worse. Furthermore, looking at your /dev/shm, everything looks fine and pihole-FTL shouldn't be using so much memory. What is the size of your /etc/pihole/gravity.list?