All of those should be removed.
sounds like a plan, thanks for your efforts so far .... yepp, and thx to homeoffice enabling us to be flexible in replying 
Thanks, your help is much apreciated!
If you'd be willing to take a risk and change some code, you could give me a hand at checking a potential mitigation. 
The following has allowed me to complete a gravity run on my NanoPi/256M with your problematic lists where it has failed previously.
Before we start:
Please make a backup of your Pi-hole configuration via Settings | Teleporter.
Make sure that there is no temporary gravity database:
sudo rm /etc/pihole/gravity.db_temp
And no left-over tmp files:
sudo rm -r /tmp/tmp.*
Be careful to run above exactly as provided.
Next, please make a backup of your current original gravity.sh:
sudo cp /opt/pihole/gravity.sh /opt/pihole/gravity.sh.backup
Then edit that file:
sudo nano /opt/pihole/gravity.sh
We are going to add at line 763 to remove some tmp files earlier in the process:
We are inserting two lines here (763 and 764).
This section should then read as follows
763 # @remove-fix remove tmp files early
764 rm "${GRAVITY_TMPDIR}"/*.phgpb 2> /dev/null
765
766 # Do we need to fall back to a cached list (if available)?
Once you've done that, could you try to run pihole -g and report the findings?
EDIT: This change would primarily benefit installations with more than just one or two blocklists, especially when /tmp is mounted on a tmpfs filesystem.
Would like to take the risk, will let you know!
I can do that later on after work and having backed up my SD-card and report back, will then re-enabling the 2 big lists for testing in addition to my 70+
OK, we'll get back in contact then.
It may be as well worth waiting - perhaps our development team will be able to share some additional insights by then.
Hi Bucking_Horn (pi-hole.net),
I had time to check your potential mitigation. Unfortunately to no avail. Followed your steps precisely.
Gravity update still choked my system:
- NanoPI Neo2, running DietPi, 1 GB RAM + 1GB swap-file
- 1 blocklist: Hagezi Ultimate (ABP version)
I killed the process (after 12 mins):
Thanks anyway for your help, unfortunately it did not work.
I tried this as well and the process works normally until it hits the large porn blocklist.
At that point, the size of the swap file slowly grows until it reaches about 750MB out of 1080MB max size.
As this is happening, the tmp files are fairly small compared to the memory consumed:
pi@pi0-2:~$ sudo du -h /tmp/tmp.*
49M /tmp/tmp.4voOUeZw8z.phgpb
8.6M /tmp/tmp.ZM3X0vs3kk.gravity
I stopped the process after about 15 minutes.
This is the blocklist it choked on: https://raw.githubusercontent.com/chadmayfield/my-pihole-blocklists/master/lists/pi_blocklist_porn_all.list
I run pi-hole on a Raspberry Pi 3B+, 1 GB RAM, 1 GB swap file, bare metal no docker, on Raspbian 10 Buster. I have 31 active adlists, almost 1,270,000 domains combined in the DB. All of the lists are
small lists. Either in the latest pihole update, or before, it never took more than a few minutes to compile the database. I did not see anything unusual with this update core 5.16.1, FTL 5.22
@Bucking_Horn / devs
I've hosted the atop data capture and sent the link and some notes to tricorder
https://tricorder.pi-hole.net/hKOKCLC9/
My Pi-hole's own debug log is quoted above from earlier
Edit: below database issue is resolved. A reboot fixed it – following the gravity update Pi-hole went offline resulting in no graph or query log all day, a reboot now has resumed normal FTL operation.
Summary
Following the problematic gravity update I still have non-stop entries like this now in FTL.log. I can open a new thread if that's more suitable
[2023-03-24 20:20:00.096 561/T626] Notice: Queries stored in long-term database: 0 (took 2.8 ms, last SQLite ID 913643)
[2023-03-24 20:20:00.097 561/T626] Queries from earlier attempt(s) stored successfully
[2023-03-24 20:20:01.117 561/T626] ERROR: SQL query "DELETE FROM query_storage WHERE timestamp <= 1679084400" failed: database is locked (SQLITE_BUSY)
[2023-03-24 20:20:01.117 561/T626] delete_old_queries_in_DB(): Deleting queries due to age of entries failed!
[2023-03-24 20:20:02.237 561/T626] ERROR: SQL query "END TRANSACTION" failed: database is locked (SQLITE_BUSY)
[2023-03-24 20:20:02.238 561/T626] WARNING: Storing devices in network table failed: database is locked
Yeah, that's expected for your configuration, Maarten.
My hacked change won't affect single block-list installations like yours. It would only come into play if your /tmp storage would fill up with curl buffers from multiple lists.
It doesn't address speed directly, nor any potential issues caused by single humongous blocklists.
@Max-Mustermann, as the topic creator, your setup with ~80 of blocklists would probably benefit the most here.
But you could also consider to wait a bit longer:
We are making progress towards a solution addressing all of those issues, and it's starting to look like a fix may not be far away.
1 Like
Hi,
Ok, thanks anyway. Good to hear the devs are working on a soution.
@Bucking_Horn thanks for the information, my apologies, I read that @Maarten has tested and it didnt work out, so I skipped the testing ....
I am about to backup my system now and could try out your solution - if still there is a need OR what for the fix and test the fix 
lemme know if I can assist further
You can try the (still work in progress) branch gravity-test with pihole checkout core gravity-test (take a backup first just in case, but should be fine!)
ok, thx, how can I revert back to stable branch once test completed?
pihole checkout core master
Or pihole checkout master
Either will work in this case.
What a difference!
I switched from the base to the gravity-test branch on my Pi Zero 2 and the gravity update ran to completion in a "normal" amount of time. Memory consumption stayed in a normal range.
Not sure what else is needed to bring it out of the WIP status but it's a great start!! 
Agree, gravity-test build differs day and night.
This rocks! Hagezi Ultimate ABP style list was processed within one minute!
Great work! Big ran of applause to the devs !!!
How do you gravity-test ???