started test with previously disabled 2 big lists (in addition to my 70+), so fully loaded (5.7mio entries all in all):
whole process (pihole upgrade, list upgrade to building new gravitiy) took from
16:36 - 16:57 - 21 mins
mem just fine - 96MB system swap used after process completed
VERY IMPRESSIVE — thanks a ton 
seeing significantly drop in items on lists (expected 5.7 - currently 4.4)
I wanted to post and share my experience with this latest update.
The update ran incredibly slow when updating gravity lists. I've yet to get it to complete on any raspberry pi with the newest/latest version.
I have 5.4M entries on my block list, so a few large lists.
I ran the gravity-test branch, and it seems to have parsed the lists quicker, I can't say for sure if it is back to previous version speeds or not.
What is a previous version that I could go back to and test? I forgot to note it down before upgrading. Or where can I find information on previous versions so I can potentially choose?
When can we expect a new update?
When it's ready.
Apologies for breaking into your great work, I've been trying to follow what it is your doing, although not easy...
I've noticed the change you made (exit 1) when the database creation fails, thus continuing with the old database, but preventing service interruption. great idea.
Would it be possible to create a diagnostics message before exiting. gravity runs in background (most users), a warning message in the web interface might draw attention to the problem.
Again, thanks for the great work.
P.S. on the dutch forum, there are already some reports of success with the gravity-test branch, using the lists that have been reported to cause the problem.
Yeah, probably. We can add the window dressing later after we get a fix for what we changed in the wrong direction.
No apologies necessary.
That was the intent from the very start of things. There is code that return 1 when the database fails, the problem was that we never checked for that return value and kept going no matter what.
+1 thanks for the quick investigation and fix
Now on gravity-test temporarily and updates are back to normal times, and metrics like cpu, scheduling, memory, disk are also back to normal. It also appears to have cleared the database readonly errors I was getting.
Just pushed a new hotfix release to v5.16.2
Pi-hole version is v5.16.2 (Latest: v5.16.2)
AdminLTE version is v5.19 (Latest: v5.19)
FTL version is v5.22 (Latest: v5.22)
time pihole -g
real 3m29.127s
user 2m2.416s
sys 0m17.518s
used to be 6 to 7 minutes.
thanks development team for the great work!
edit
/tmp size, measured every second during pihole -g (used to be 160Mb)
84K /tmp
6.7M /tmp
5.4M /tmp
18M /tmp
36M /tmp
5.4M /tmp
19M /tmp
30M /tmp
31M /tmp
32M /tmp
36M /tmp
37M /tmp
38M /tmp
41M /tmp
42M /tmp
43M /tmp
44M /tmp
45M /tmp
46M /tmp
47M /tmp
50M /tmp
64M /tmp
76M /tmp
84K /tmp
/edit
many thanks to the whole dev team for the quick reaction and fixing - this time even my swap space was not touched
100% agree, great work!
thx
Thankyou to all involved for the troubleshooting and the quick update. 5.16.2 has fixed everything here, gravity updates back to normal times and resources.
Not sure why 5.16.2 not showing up when trying to check for update - is there some specific option i need to use?
21:34:13-pi@dnspi3b:~ $pihole -up --check-only
[✓] Update local cache of available packages
[i] Existing PHP installation detected : PHP version 7.4.33
[✓] Checking for git
[✓] Checking for iproute2
[✓] Checking for dialog
[✓] Checking for ca-certificates
[i] Checking for updates...
[i] Pi-hole Core: update available
[i] Web Interface: update available
[i] FTL: update available
21:34:29-pi@dnspi3b:~ $pihole -v
Pi-hole version is v5.15.5 (Latest: v5.16.1)
AdminLTE version is v5.18.4 (Latest: v5.19)
FTL version is v5.21 (Latest: v5.2
requires a single sqlite3 command in the script:
sudo pihole-FTL sqlite3 "/etc/pihole/pihole-FTL.db" \
"insert or ignore into 'message' \
( timestamp, type, message, blob1 ) \
values ('$(date +%s)', 'SCRIPT', 'Failed to update gravity database.', '$(echo $(basename $0) | base64 -)');"
and an additional case in /var/www/html/admin/scripts/pi-hole/js/messages.js
case "SCRIPT":
decodedblob1=atob(row.blob1)
return (
"Warning generated by <code>" + decodedblob1 + "</code><pre>" +
row.message +
'</pre>'
);
your opinion?
implies @DL6ER that the SCRIPT type must be reserved in https://raw.githubusercontent.com/pi-hole/FTL/master/src/database/message-table.c
Unfortunately, it is a bit more tricky than that. Reason is v6.0 which relies on the messages being generated solely by FTL so API calls to /api/info/messages do not really have to connect to the database but can use internal buffers to provide this. I'm not sure what is the best strategy right now, probably we have to do this and then we have to give up on this simplification in FTL.
Add. comment after tonights gravity run, saw nearly a clean swap file after adlist upgrade, seems not only fixing gravity but also tuning the process
Pi-hole [v5.15.5]
FTL [v5.21]
Web Interface [v5.18.4]
|real|12m15.138s|
|user|6m5.210s|
|sys|1m6.457s|
Pi-hole [v5.16.2]
FTL [v5.22]
Web Interface [v5.19]
|real|11m13.220s|
|user|4m13.723s|
|sys|1m13.703s|
This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.