Hi,
I am currently upgrading pihole to newest version and somehow list retrieving/extraction is killing my pi:
hanging in that single list since now more that 30mins, no end in sight ... should I kill the process or let it run?
1.3 mio entries in the list - is that too much?
Is there a way that pihole notice a kind of "overkill" and stops processing or does it run "forever"?
that single list ran for app. 2hours ....
shouldnt there be a "timeout" to gracefully end a sort process if such an overkill ?
"overkill"-lists are:
https://raw.githubusercontent.com/mkb2091/blockconvert/master/output/domains.txt
Even with moderately sized lists the new Pi-hole goes berserk on my NanoPi NEO2 (aarch64) /w 1GB ram while processing the adlist(s). First series of SED command are no problem but GREP is choking my system. Seems new Pi-hole release has not been tested thoroughly.
Maybe it should first check if list is plain or ADB and when plain skip the cleanup process.
Would like to roll-back to previous version for now because this is insane.
if once done at least I will inactive the BIG lists ... hoping for a pihole fix ...
I have reverted back to previous version.
How do I revert to a previous version of Pi-hole? - FAQs - Pi-hole Userspace
thanks for the link - I'll let it run, wont kill the process(es), not what happens if I would do so
my list update took app. 3hours with new updated pihole version ..... normally I am done in app 20min I guess .... so for now I am disabling my big lists - now building gravity ... so the end is near ... 
app 3hours update is finally done -instead of 20-30 mins .... due to the new long running list update
'hagezi' ultimate list (huge!) updated with previous Pi-hole version in couple of minutes, latest Pi-hole version took hours and still did not complete.
Moreover the new gravity update procedure (i.e. the GREP command) is writing an awful lot to the swapfile which, I am afraid, will ruin my sd-card in no time when it is doing so for hours...
for mitigation reasons I have disabled the 2 lists causing the hours of updating, hope there will be a fix soon to it
I saw the same. It "hung" on a list which is 58meg.
I left this running overnight and it completed after 6 hours.
As a test, I split the file in to (6x) 10meg chunks and this processed in 8 minutes.
Looking at "top" while the 58meg file was processing, all the pi's ram and swap was 100% used.
My guess is that the grep -Fvf xxxx process is the cause. It seems to be inefficient when it processes large files, although it will process the same data as smaller files without this issue.
NB. This didn't happen prior to this latest update. The lists in the pihole are the same, and I did not experience this issue or need to split these large lists prior to this update.
Agree. It should not use swap since it will definitely shorten SD-card life time a lot.
I presume the grep command is introduced to process ADB lists...but applied regardless of the lists used.
For comparison sake, here are the results of running pihole -g with those two particular adlists on two different installs. First, a NanoPi NEO running the current (latest) Pi-hole version:
pihole -v
Pi-hole version is v5.16.1 (Latest: v5.16.1)
AdminLTE version is v5.19 (Latest: v5.19)
FTL version is v5.22 (Latest: v5.22)
time pihole -g
[i] Neutrino emissions detected...
[✓] Pulling blocklist source list into range
[✓] Preparing new gravity database
[i] Using libz compression
[i] Target: https://raw.githubusercontent.com/mkb2091/blockconvert/master/output/domains.txt
[✓] Status: Retrieval successful
[i] Imported 1279214 domains
[i] List has been updated
[i] Target: https://divested.dev/hosts-domains
[✓] Status: Retrieval successful
[i] Imported 1180736 domains
[✓] Creating new gravity databases
[✓] Storing downloaded domains in new gravity database
[✓] Building tree
[✓] Swapping databases
[✓] The old database remains available.
[i] Number of gravity domains: 2459950 (1968354 unique domains)
[i] Number of exact blacklisted domains: 1
[i] Number of regex blacklist filters: 24
[i] Number of exact whitelisted domains: 26
[i] Number of regex whitelist filters: 0
[✓] Flushing DNS cache
[✓] Cleaning up stray matter
[✓] FTL is listening on port 53
[✓] UDP (IPv4)
[✓] TCP (IPv4)
[✓] UDP (IPv6)
[✓] TCP (IPv6)
[✓] Pi-hole blocking is enabled
real 5m19.964s
user 3m54.376s
sys 0m28.478s
Next, from a Pi Zero W running a previous Pi-hole version d
pihole -v
Pi-hole version is v5.15.5 (Latest: v5.15.5)
AdminLTE version is v5.18.4 (Latest: v5.18.4)
FTL version is v5.21 (Latest: v5.21)
time pihole -g
[i] Neutrino emissions detected...
[✓] Pulling blocklist source list into range
[✓] Preparing new gravity database
[i] Using libz compression
[i] Target: https://raw.githubusercontent.com/mkb2091/blockconvert/master/output/domains.txt
[✓] Status: Retrieval successful
[i] Imported 1279214 domains
[i] Target: https://divested.dev/hosts-domains
[✓] Status: Retrieval successful
[i] Imported 1180736 domains
[✓] Creating new gravity databases
[✓] Storing downloaded domains in new gravity database
[✓] Building tree
[✓] Swapping databases
[✓] The old database remains available.
[i] Number of gravity domains: 2459950 (1968354 unique domains)
[i] Number of exact blacklisted domains: 4
[i] Number of regex blacklist filters: 25
[i] Number of exact whitelisted domains: 18
[i] Number of regex whitelist filters: 0
[✓] Flushing DNS cache
[✓] Cleaning up stray matter
[✓] FTL is listening on port 53
[✓] UDP (IPv4)
[✓] TCP (IPv4)
[✓] UDP (IPv6)
[✓] TCP (IPv6)
[✓] Pi-hole blocking is enabled
real 6m52.923s
user 5m44.170s
sys 0m36.464s
@jfb so no issues on ur side?
But seems to be an upgrade related issue, before the latest pihole upgrade my update time was app 20-30 min as I mentioned previously
so, retest with pihole -g - without the 2 mentioned lists above - less then 20mins
Hi jfb,
Thanks, but on my system, and of others', after Pihole update it take hours...where it was minutes with previous version...(Hagezi Pro+ list)
In htop I see memory usage and swap-file usage 100%, system becomes unresponsive.
A grep command by Pihole seems the cause...
What to do?
Maarten
Rasp. Pi2B, 1GB
653000 domains in block lists.
Update ran in less than 5 Minutes.
NanoPI NEO2, 1GB
2.207.630 domains in one block list (hagezi ultimate list) Pi-hole chokes, becomes totally unresponsive. With previous Pi-hole versions processing the block list (sed, grep) took less than 1 min. With latest version processing it takes at least more than 10 minutes to proces the much smaller ABP version of the same domain list. I killed the pihole 'grep' process (do not want to ruin my SD-card).
New version cannot handle one big list, seems to have less problems with multiple smaller lists.
Processing has become very, very, very memory hungry (100% RAM, 100% swap-file) and hence very, very slow due to continuous disk r/w operations (swap-file).
Preliminary conclusions
With the new Pi-hole version:
So the new Pi-hole version is at least 10x (but probably 100x) slower in updating gravity database (due to changed procedure of processing block lists) when using bigger block lists. Besides I fear SD-card will wear out fast. I am not taking the risk, so I rolled-back to v5.21
I'm seeing similar issues with a Pi Zero 2 and a large porn block list (>1000000 domains). I let it run for over 15 minutes and it never did finish. Seems to consume all the limited RAM on that device. It's definitely different that the previous version.
My Debug token is: https://tricorder.pi-hole.net/5LAiuKmW/
It's currently running fine because I copy the database over from a Pi4 (gravity-sync). That machine manages to run the gravity update to completion.
The tmp.*.phgpb files are created by curl when downloading lists. You have a very large number of them, how many lists do you use exactly?