I think it may be useful to add a variable to the setupVars.conf file to be able to control the timeout of Gravity updates.
It seems to me that the implementation would be very simple.
1.- Add the variable in the setupVars.conf file (for example: GRAVITY_TIMEOUT=10)
2.- Use it on line 644 of the /opt/pihole/gravity.sh by adding the -m ${GRAVITY_TIMEOUT} parameter.
I think it is interesting to speed up the update of the Block List when there are many and may not be updated
Please elaborate on the use case here.
Regardless of how long fetching and updating adlists takes, Pi-hole is only offline for the short period of time when the new gravity database is swapped for the old one.
The problem is not how long PIHole is standing still while making the gravity database change. The problem is how long it can take to get the content of all the URLs in the adlists.
As a curl is executed without controlling the timeout to get the content the command uses the default timeout, which is I think is 120sec. if this happens with multiple urls the command to update the lists may take many minutes. On my case the update gravity take over 1 hour:
real 59m57.061s
user 2m30.792s
sys 0m27.374s
Why is this a problem?
What is the output of the following command from your Pi terminal:
sqlite3 /etc/pihole/gravity.db "SELECT address FROM adlist where enabled=1"
On what device are you running Pi-hole?
Hi, send the file with URLs.
The problem is not the number of URLs that must be checked, but if any of those URLs are not accessible, in that case the curl that is done, as it does not have a timeout parameter takes the default value of the curl command in linux, which sounds to me that it is two minutes. I think the hardware in this case is indifferent in problem is not in the generation, nor in change of the database, it is in the curl calls to the URLs.
gravity.txt (6.5 KB)
I think there may be several solutions:
1.- Add a timeout parameter
2.- Do a preliminary check if that URL is accessible.
3.- Add a process prior to updating gravity checking which URLs are accessible and if they are not accessible, deactivate them. I pass an example script
test_gravity_urls.sh.zip (586 Bytes)
After apply this script, the process pass from 1H to 3min
| real | 3m25.831s |
|---|---|
| user | 2m3.704s |
| sys | 0m22.302s |
Attach new version of script checking valid mime-type of ad list
test_gravity_urls_v1.sh.zip (745 Bytes)
I will propose a different solution.
In your specific case, you are subscribing to a number of adlists that don't exist. They don't exist now, and they won't exist in the future.
Additionally, you are subscribing to an adlist that is not compatible with Pi-hole, and every entry on that list was rejected by Pi-hole.
https://raw.githubusercontent.com/BlackJack8/iOSAdblockList/master/Hosts.txt
Another list with many entries that Pi-hole has to filter - 29553 domains invalid
https://smokingwheels.github.io/Pi-hole/allhosts
This next URL is not an adlist - it a list of URL's. If you want to use all these URL's, the URL's themselves need to be added to Pi-hole (not the web page that shows the URLs):
https://v.firebog.net/hosts/lists.php?type=tick
Here is the complete output of pihole -g -f after I added the complete list of adlist URLs you provided to a test Pi-hole. This is a Pi-3B+ running Raspbian Bullseye. I will note that there were no significant delays loading any of the lists. They either came back "can't find the list" or the list loaded immediately, with none of the two minute delays you reported. The majority of the time was spent creating the new gravity database from the 8M+ domains.
time pihole -g -f
[✓] Deleting existing list cache
[i] Neutrino emissions detected...
[✓] Pulling blocklist source list into range
[✓] Preparing new gravity database
[i] Using libz compression
[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
[✓] Status: Retrieval successful
[i] Analyzed 136942 domains
[i] Target: https://mirror1.malwaredomains.com/files/justdomains
[✗] Status: Not found
[✗] List download failed: no cached list available
[i] Target: http://sysctl.org/cameleon/hosts
[✓] Status: Retrieval successful
[i] Analyzed 20562 domains
[i] Target: https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist
[✗] Status: Connection Refused
[✗] List download failed: no cached list available
[i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
[✓] Status: Retrieval successful
[i] Analyzed 34 domains
[i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
[✓] Status: Retrieval successful
[i] Analyzed 2701 domains
[i] Target: https://hosts-file.net/ad_servers.txt
[✗] Status: Not found
[✗] List download failed: no cached list available
[i] Target: https://hosts-file.net/grm.txt
[✗] Status: Not found
[✗] List download failed: no cached list available
[i] Target: https://reddestdream.github.io/Projects/MinimalHosts/etc/MinimalHostsBlocker/minimalhosts
[✓] Status: Retrieval successful
[i] Analyzed 710 domains
[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/KADhosts/hosts
[✓] Status: Retrieval successful
[i] Analyzed 59079 domains
[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Spam/hosts
[✓] Status: Retrieval successful
[i] Analyzed 57 domains
[i] Target: https://v.firebog.net/hosts/static/w3kbl.txt
[✓] Status: Retrieval successful
[i] Analyzed 356 domains
[i] Target: https://v.firebog.net/hosts/BillStearns.txt
[✓] Status: Retrieval successful
[i] Analyzed 8778 domains
[i] Target: https://www.dshield.org/feeds/suspiciousdomains_Low.txt
[✓] Status: Retrieval successful
[i] Analyzed 0 domains
[i] Target: https://www.dshield.org/feeds/suspiciousdomains_Medium.txt
[✓] Status: Retrieval successful
[i] Analyzed 0 domains
[i] Target: https://www.dshield.org/feeds/suspiciousdomains_High.txt
[✓] Status: Retrieval successful
[i] Analyzed 0 domains
[i] Target: https://www.joewein.net/dl/bl/dom-bl-base.txt
[✓] Status: Retrieval successful
[i] Received empty file
[✗] List download failed: no cached list available
[i] Target: https://raw.githubusercontent.com/matomo-org/referrer-spam-blacklist/master/spammers.txt
[✓] Status: Retrieval successful
[i] Analyzed 2119 domains
[i] Target: https://hostsfile.org/Downloads/hosts.txt
[✓] Status: Retrieval successful
[i] Analyzed 25263 domains
[i] Target: https://someonewhocares.org/hosts/zero/hosts
[✓] Status: Retrieval successful
[i] Analyzed 11387 domains
[i] Target: https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt
[✓] Status: Retrieval successful
[i] Analyzed 7066 domains
[i] Target: https://raw.githubusercontent.com/vokins/yhosts/master/hosts
[✓] Status: Retrieval successful
[i] Analyzed 8929 domains
[i] Target: http://winhelp2002.mvps.org/hosts.txt
[✓] Status: Retrieval successful
[i] Analyzed 8730 domains
[i] Target: https://hostsfile.mine.nu/hosts0.txt
[✓] Status: Retrieval successful
[i] Analyzed 111391 domains
[i] Target: https://v.firebog.net/hosts/Kowabit.txt
[✓] Status: Retrieval successful
[i] Analyzed 7680 domains
[i] Target: https://adblock.mahakala.is
[✓] Status: Retrieval successful
[i] Analyzed 293121 domains
[i] Target: https://adaway.org/hosts.txt
[✓] Status: Retrieval successful
[i] Analyzed 7194 domains
[i] Target: https://v.firebog.net/hosts/AdguardDNS.txt
[✓] Status: Retrieval successful
[i] Analyzed 46921 domains
[i] Target: https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
[✓] Status: Retrieval successful
[i] Analyzed 42553 domains
[i] Target: https://v.firebog.net/hosts/Easylist.txt
[✓] Status: Retrieval successful
[i] Analyzed 18404 domains
[i] Target: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts;showintro=0
[✓] Status: Retrieval successful
[i] Analyzed 3634 domains
[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/UncheckyAds/hosts
[✓] Status: Retrieval successful
[i] Analyzed 9 domains
[i] Target: https://www.squidblacklist.org/downloads/dg-ads.acl
[✗] Status: Connection Timed Out (Cloudflare)
[✗] List download failed: no cached list available
[i] Target: https://v.firebog.net/hosts/Easyprivacy.txt
[✓] Status: Retrieval successful
[i] Analyzed 12919 domains
[i] Target: https://v.firebog.net/hosts/Prigent-Ads.txt
[✓] Status: Retrieval successful
[i] Analyzed 3675 domains
[i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt
[✓] Status: Retrieval successful
[i] Analyzed 14956 domains
[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.2o7Net/hosts
[✓] Status: Retrieval successful
[i] Analyzed 1286 domains
[i] Target: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
[✓] Status: Retrieval successful
[i] Analyzed 347 domains
[i] Target: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/android-tracking.txt
[✓] Status: Retrieval successful
[i] Analyzed 80 domains
[i] Target: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV.txt
[✓] Status: Retrieval successful
[i] Analyzed 225 domains
[i] Target: https://v.firebog.net/hosts/Airelle-trc.txt
[✗] Status: Not found
[✗] List download failed: no cached list available
[i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
[✓] Status: Retrieval successful
[i] Analyzed 2735 domains
[i] Target: https://hosts-file.net/exp.txt
[✗] Status: Not found
[✗] List download failed: no cached list available
[i] Target: https://hosts-file.net/emd.txt
[✗] Status: Not found
[✗] List download failed: no cached list available
[i] Target: https://hosts-file.net/psh.txt
[✗] Status: Not found
[✗] List download failed: no cached list available
[i] Target: https://mirror.cedia.org.ec/malwaredomains/immortal_domains.txt
[✗] Status: Not found
[✗] List download failed: no cached list available
[i] Target: https://www.malwaredomainlist.com/hostslist/hosts.txt
[✓] Status: Retrieval successful
[i] Analyzed 0 domains
[i] Target: https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
[✓] Status: Retrieval successful
[i] Analyzed 2046 domains
[i] Target: https://v.firebog.net/hosts/Prigent-Malware.txt
[✓] Status: Retrieval successful
[i] Analyzed 61316 domains
[i] Target: https://v.firebog.net/hosts/Prigent-Phishing.txt
[✓] Status: Retrieval successful
[i] Analyzed 61316 domains
[i] Target: https://phishing.army/download/phishing_army_blocklist_extended.txt
[✓] Status: Retrieval successful
[i] Analyzed 112346 domains
[i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
[✓] Status: Retrieval successful
[i] Analyzed 302 domains
[i] Target: https://ransomwaretracker.abuse.ch/downloads/RW_DOMBL.txt
[✗] Status: https://ransomwaretracker.abuse.ch/downloads/RW_DOMBL.txt (503)
[✗] List download failed: no cached list available
[i] Target: https://ransomwaretracker.abuse.ch/downloads/CW_C2_DOMBL.txt
[✗] Status: https://ransomwaretracker.abuse.ch/downloads/CW_C2_DOMBL.txt (503)
[✗] List download failed: no cached list available
[i] Target: https://ransomwaretracker.abuse.ch/downloads/LY_C2_DOMBL.txt
[✗] Status: https://ransomwaretracker.abuse.ch/downloads/LY_C2_DOMBL.txt (503)
[✗] List download failed: no cached list available
[i] Target: https://ransomwaretracker.abuse.ch/downloads/TC_C2_DOMBL.txt
[✗] Status: https://ransomwaretracker.abuse.ch/downloads/TC_C2_DOMBL.txt (503)
[✗] List download failed: no cached list available
[i] Target: https://ransomwaretracker.abuse.ch/downloads/TL_C2_DOMBL.txt
[✗] Status: https://ransomwaretracker.abuse.ch/downloads/TL_C2_DOMBL.txt (503)
[✗] List download failed: no cached list available
[i] Target: https://v.firebog.net/hosts/Shalla-mal.txt
[✓] Status: Retrieval successful
[i] Analyzed 19239 domains
[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Risk/hosts
[✓] Status: Retrieval successful
[i] Analyzed 2190 domains
[i] Target: https://www.squidblacklist.org/downloads/dg-malicious.acl
[✗] Status: Connection Timed Out (Cloudflare)
[✗] List download failed: no cached list available
[i] Target: https://raw.githubusercontent.com/HorusTeknoloji/TR-PhishingList/master/url-lists.txt
[✓] Status: Retrieval successful
[i] Analyzed 826138 domains
[i] Target: https://v.firebog.net/hosts/Airelle-hrsk.txt
[✗] Status: Not found
[✗] List download failed: no cached list available
[i] Target: https://zerodot1.gitlab.io/CoinBlockerLists/hosts
[✓] Status: Retrieval successful
[i] Analyzed 273733 domains
[i] Target: https://raw.githubusercontent.com/deathbybandaid/piholeparser/master/Subscribable-Lists/CombinedBlacklists/CombinedBlackLists.txt
[✓] Status: Retrieval successful
[i] Analyzed 2077732 domains
[i] Target: ~
[✗] Status: Connection Refused
[✗] List download failed: no cached list available
[i] Target: 0,0-1
[✗] Status: Connection Refused
[✗] List download failed: no cached list available
[i] Target: All
[✗] Status: Connection Refused
[✗] List download failed: no cached list available
[i] Target: https://zerodot1.gitlab.io/CoinBlockerLists/list.txt
[✓] Status: Retrieval successful
[i] Analyzed 273733 domains
[i] Target: https://zerodot1.gitlab.io/CoinBlockerLists/list_browser.txt
[✓] Status: Retrieval successful
[i] Analyzed 3496 domains
[i] Target: https://zerodot1.gitlab.io/CoinBlockerLists/list_optional.txt
[✓] Status: Retrieval successful
[i] Analyzed 1580 domains
[i] Target: https://smokingwheels.github.io/Pi-hole/allhosts
[✓] Status: Retrieval successful
[i] Analyzed 1301770 domains, 29553 domains invalid!
Sample of invalid domains:
- 001linker.net;1
- 0104joe.com;1
- 01cas.info;1
- 02troy84.net;1
- 0d1dnfu-x5esmlc-m6u1x2.com;1
[i] Target: https://tspprs.com/dl/spotify
[✓] Status: Retrieval successful
[i] Analyzed 0 domains
[i] Target: https://gist.githubusercontent.com/BBcan177/4a8bf37c131be4803cb2/raw/343ff780e15205b4dd0de37c86af34cfb26b2fbe/MS-2
[✓] Status: Retrieval successful
[i] Analyzed 16326 domains
[i] Target: https://gist.githubusercontent.com/BBcan177/b6df
[✗] Status: Not found
[✗] List download failed: no cached list available
[i] Target: https://adaway.org/hosts.txthttps://v.firebog.net/hosts/AdguardDNS.txt
[✗] Status: Not found
[✗] List download failed: no cached list available
[i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txthttps://hosts-file.net/ad_servers.txt
[✗] Status: Forbidden
[✗] List download failed: no cached list available
[i] Target: https://v.firebog.net/hosts/Easylist.txthttps://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext
[✗] Status: Not found
[✗] List download failed: no cached list available
[i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
[✓] Status: Retrieval successful
[i] Analyzed 9 domains
[i] Target: https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts
[✓] Status: Retrieval successful
[i] Analyzed 17392 domains
[i] Target: https://raw.githubusercontent.com/jdlingyu/ad-wars/master/hosts
[✓] Status: Retrieval successful
[i] Analyzed 1665 domains
[i] Target: https://codeberg.org/spootle/blocklist/raw/branch/master/blocklist.txt
[✓] Status: Retrieval successful
[i] Analyzed 117229 domains
[i] Target: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt
[✓] Status: Retrieval successful
[i] Analyzed 8343 domains
[i] Target: https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt
[✓] Status: Retrieval successful
[i] Analyzed 37 domains
[i] Target: https://v.firebog.net/hosts/Prigent-Crypto.txt
[✓] Status: Retrieval successful
[i] Analyzed 14326 domains
[i] Target: https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt
[✓] Status: Retrieval successful
[i] Analyzed 8147 domains
[i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
[✓] Status: Retrieval successful
[i] Analyzed 2190 domains
[i] Target: https://urlhaus.abuse.ch/downloads/hostfile/
[✓] Status: Retrieval successful
[i] Analyzed 726 domains
[i] Target: https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser
[✓] Status: Retrieval successful
[i] Analyzed 3496 domains
[i] Target: https://raw.githubusercontent.com/chadmayfield/my-pihole-blocklists/master/lists/pi_blocklist_porn_all.list
[✓] Status: Retrieval successful
[i] Analyzed 1906594 domains
[i] Target: https://raw.githubusercontent.com/chadmayfield/my-pihole-blocklists/master/lists/pi_blocklist_porn_top1m.list
[✓] Status: Retrieval successful
[i] Analyzed 11868 domains
[i] Target: https://raw.githubusercontent.com/anudeepND/blacklist/master/facebook.txt
[✓] Status: Retrieval successful
[i] Analyzed 3996 domains
[i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
[✓] Status: Retrieval successful
[i] Analyzed 1286 domains
[i] Target: https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt
[✓] Status: Retrieval successful
[i] Analyzed 13992 domains
[i] Target: https://hostfiles.frogeye.fr/multiparty-trackers-hosts.txt
[✓] Status: Retrieval successful
[i] Analyzed 15614 domains
[i] Target: https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt
[✓] Status: Retrieval successful
[i] Analyzed 426744 domains
[i] Target: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/AmazonFireTV.txt
[✓] Status: Retrieval successful
[i] Analyzed 17 domains
[i] Target: https://v.firebog.net/hosts/Admiral.txt
[✓] Status: Retrieval successful
[i] Analyzed 636 domains
[i] Target: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext
[✓] Status: Retrieval successful
[i] Analyzed 3634 domains
[i] Target: https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts_without_controversies.txt
[✓] Status: Retrieval successful
[i] Analyzed 0 domains
[i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
[✓] Status: Retrieval successful
[i] Analyzed 57 domains
[i] Target: https://winhelp2002.mvps.org/hosts.txt
[✓] Status: Retrieval successful
[i] Analyzed 8730 domains
[i] Target: https://v.firebog.net/hosts/neohostsbasic.txt
[✓] Status: Retrieval successful
[i] Analyzed 2411 domains
[i] Target: https://raw.githubusercontent.com/RooneyMcNibNug/pihole-stuff/master/SNAFU.txt
[✓] Status: Retrieval successful
[i] Analyzed 63943 domains
[i] Target: https://paulgb.github.io/BarbBlock/blacklists/hosts-file.txt
[✓] Status: Retrieval successful
[i] Analyzed 550 domains
[i] Target: https://raw.githubusercontent.com/BlackJack8/iOSAdblockList/master/Hosts.txt
[✓] Status: Retrieval successful
[i] Analyzed 14342 domains, 14342 domains invalid!
Sample of invalid domains:
- ||050y9lhe8p.com^$important
- ||0914.global.ssl.fastly.net^$important
- ||0c3-a.tlnk.io^$important
- ||0i0i0i0.com^$important
- ||1-100-62-1.b.cdn13.com^$important
[i] Target: https://v.firebog.net/hosts/lists.php?type=tick
[✓] Status: Retrieval successful
[i] Analyzed 0 domains
[✓] Creating new gravity databases
[✓] Storing downloaded domains in new gravity database
[✓] Building tree
[✓] Swapping databases
[✓] The old database remains available.
[i] Number of gravity domains: 8497185 (5673886 unique domains)
[i] Number of exact blacklisted domains: 1
[i] Number of regex blacklist filters: 23
[i] Number of exact whitelisted domains: 23
[i] Number of regex whitelist filters: 1
[i] FTL is not running
[✓] Cleaning up stray matter
[✓] FTL is listening on port 53
[✓] UDP (IPv4)
[✓] TCP (IPv4)
[✓] UDP (IPv6)
[✓] TCP (IPv6)
[✓] Pi-hole blocking is enabled
real 12m4.628s
user 7m24.057s
sys 0m52.197s
Rather than writing a script to do this, look at the output of the gravity update process, clean up your adlists, and see if the problem resolves. Or, if you prefer to see information on the admin web GUI, the details for each of your subscribed adlists are shown in the Adlists tab:
You can't feed garbage into the gravity process and expect it to produce good results.
Given the number of domains you are importing (after the bad URL's are eliminated), it's still going to take a while for the Pi to process all the domains and rebuild the new database.
An additional thought - is there any actual value adding 5.67M domains to your blocklist?
In your lifetime, your clients would only attempt to visit a tiny fraction of these. As an example, for my daily use Pi-hole, this command shows the number of blocked domains blocked by Pi-hole (for any reason) in the past 90 days:
sqlite3 /etc/pihole/pihole-FTL.db "select domain from queries where status in (1,4,5,9,10,11) group by domain order by domain" |wc -l
5043
A subset of these were blocked by gravity (including cname blocking):
sqlite3 /etc/pihole/pihole-FTL.db "select domain from queries where status in (1,9) group by domain order by domain" | wc -l
968
Having 550K domains on blocklist (my case), or having 5.67M domains on blocklist (your case) won't change this appreciably.
For now you can set a .curlrc to set any configuration flag you'd like.
Hello, I think what you say is correct, but I think it would be good if the update process had the ability to maintain the lists that work and those that do not work. I know that the case that I am putting to or better is very extreme because of the volume of blocked domains but I think it would be very useful to have the option that the update process itself deactivate the lists that are not accessible. I do not know how many of us look at the result of updating the lists, I certainly do not know much. Maybe it is as simple as having the option of "Do you want to deactivate the lists that are not accessible?" and that the update process itself (I insist that the curl that is done in any of the cases should have a timeout parameter) deactivates the inaccessible lists. I think what we all look at is the blocked domain counter, but not the result of the gravity update, especially since being in a cron you forget about that process.
I totally agree with what jfb said. However, I do see some value in limiting the --connect-timeout (max time to allow curl to establish a connection). I prepared a special branch adding this option with a timeout of 10 seconds.
If you want, you can try it and see if it solves the "long-running gravity" issue for you.
pihole checkout core curl_timeout.
To go back on track and before the next Pi-hole update please run pihole checkout master.
P.S. I don't want to set –max-time for curl because users might have a really slow internet connection and try to download huge lists.
Perfect works fine.
Thanks
PD: I think that the timeout can set on setupVars.conf, with default value to 10 this way you avoid hardcoding that and if necessary it is easily modifiable
I'm not a fan of making this small setting user-configurable. This is not something users should (need) to change on a regular basis. There is no gain in switching from 10 seconds to 12 seconds. Either you can access it or you can't. We should rather set a reasonable default limit (can be discussed) and that's it. The coding and maintenance overhead is not justified in my opinion.
We are also discussing making inaccessible adlist more prominent for users.
Ok, I really think that would be the optimal solution, to warn the user that there are lists and which ones cannot be processed so that they can be reviewed.
We have this displayed now in the web admin GUI, next to each adlist.
Those are not times we'd expect to observe on a regular gravity run, even on low end machines. On my RPi Zero, updating your lists took just under 15 minutes.
When did you observe those numbers?
Did you perhaps trigger another gravity update while the previous one hadn't finished yet?
I fixed it 4 months ago but only today pushed the changes to github.
| https://smokingwheels.github.io/Pi-hole/allhosts | EnabledDisabled | Default | ||
|---|---|---|---|---|
| Health status of this list: | List download was successful (OK) | |||
| --- | --- | |||
| This list was added to Pi-hole | 4 months ago (2022-05-25 22:45:56) | |||
| Database entry was last modified | 4 months ago (2022-05-25 22:45:56) | |||
| The list contents were last updated | 5 days ago (2022-09-19 19:26:03) | |||
| Number of domains on this list: | 1239324 | |||
| Number of invalid domains on this list: | 0 | |||
| Database ID of this list: | 2 |