Pi-hole update reduced number of blocked domains

That update reduced my number of blocked domains from 1,5 Million to 250.000. Same at a friend's. Something changed severly! Hardly anything is bloced anymore!

Nothing in the latest updated touched the code regarding the adlists.

Please upload a debug log and post just the token URL that is generated after the log is uploaded by running the following command from the Pi-hole host terminal:

pihole -d

or do it through the Web interface:

Tools > Generate Debug Log
__

Also, please post the output of pihole -g

Well, as I am not the only one having lost most blocked daomains after Update, the problem must be there. I did not change anything at my lists.

I attached some logs. Thanks!

https://tricorder.pi-hole.net/X9FQrWGU/

I will post the output of pihole -g later. The content has many links. New users are not allowed to post more than 5 links in this forum.

Here the output of pihole -g (part of it, Original has too many links). It receives a lot less domains than before! Same lists produced way more before. Or did all lists recently get reduced?

[i] Neutrino emissions detected...
[✓] Pulling blocklist source list into range

(......)

[✓] Storing downloaded domains in new gravity database
[✓] Building tree
[✓] Swapping databases
[✓] The old database remains available.
[i] Number of gravity domains: 321314 (252022 unique domains) (HERE THERE WERE OVER 1 MILLION!)
[i] Number of exact blacklisted domains: 60
[i] Number of regex blacklist filters: 26
[i] Number of exact whitelisted domains: 61
[i] Number of regex whitelist filters: 1
[✓] Flushing DNS cache
[✓] Cleaning up stray matter

[✓] FTL is listening on port 53
[✓] UDP (IPv4)
[✓] TCP (IPv4)
[✓] UDP (IPv6)
[✓] TCP (IPv6)

[✓] Pi-hole blocking is enabled

Wrap the output with three backticks to start and end the block.

```
LOG OUTPUT

```

We need to look at the whole gravity output. Use the format Dan provided. From which version did you upgrade? Is there any output about invalid domains?

Thanks so much for all the effort here. :slight_smile: I know, two failed lists. But would those two lists contain over 1 million domains? I even think at least one was malfunctioning before.

Here is pihole -g

 [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [i] Using libz compression

  [i] Target: https://adaway.org/hosts.txt
  [✓] Status: No changes detected
  [i] Analyzed 7563 domains

  [i] Target: https://v.firebog.net/hosts/AdguardDNS.txt
  [✓] Status: No changes detected
  [i] Analyzed 43545 domains

  [i] Target: https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 42292 domains
  [i] List stayed unchanged

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
  [✓] Status: No changes detected
  [i] Analyzed 2701 domains

  [i] Target: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext
  [✓] Status: No changes detected
  [i] Analyzed 3685 domains

  [i] Target: https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts
  [✓] Status: Retrieval successful
  [i] Analyzed 15264 domains
  [i] List stayed unchanged

  [i] Target: https://v.firebog.net/hosts/Easyprivacy.txt
  [✓] Status: No changes detected
  [i] Analyzed 12880 domains

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
  [✓] Status: Retrieval successful
  [i] Analyzed 1286 domains
  [i] List stayed unchanged

  [i] Target: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 378 domains
  [i] List stayed unchanged

  [i] Target: https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt
  [✓] Status: No changes detected
  [i] Analyzed 14039 domains

  [i] Target: https://raw.githubusercontent.com/Kees1958/W3C_annual_most_used_survey_blocklist/master/TOP_EU_US_Ads_Trackers_HOST
  [✗] Status: Not found
  [✗] List download failed: no cached list available

  [i] Target: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 6620 domains
  [i] List stayed unchanged

  [i] Target: https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt
  [✓] Status: No changes detected
  [i] Analyzed 918 domains

  [i] Target: https://v.firebog.net/hosts/Prigent-Crypto.txt
  [✓] Status: No changes detected
  [i] Analyzed 14148 domains

  [i] Target: https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
  [✓] Status: No changes detected
  [i] Analyzed 2046 domains

  [i] Target: https://phishing.army/download/phishing_army_blocklist_extended.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 53734 domains
  [i] List has been updated

  [i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 327 domains
  [i] List stayed unchanged

  [i] Target: https://v.firebog.net/hosts/Shalla-mal.txt
  [✓] Status: No changes detected
  [i] Analyzed 19239 domains

  [i] Target: https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 8148 domains
  [i] List stayed unchanged

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
  [✓] Status: Retrieval successful
  [i] Analyzed 2202 domains
  [i] List stayed unchanged

  [i] Target: https://urlhaus.abuse.ch/downloads/hostfile/
  [✓] Status: Retrieval successful
  [i] Analyzed 1473 domains
  [i] List has been updated

  [i] Target: https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser
  [✓] Status: No changes detected
  [i] Analyzed 3496 domains

  [i] Target: https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 41846 domains
  [i] List has been updated

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
  [✓] Status: Retrieval successful
  [i] Analyzed 57 domains
  [i] List stayed unchanged

  [i] Target: https://v.firebog.net/hosts/static/w3kbl.txt
  [✓] Status: No changes detected
  [i] Analyzed 357 domains

  [i] Target: https://raw.githubusercontent.com/CamelCase11/UnifiedHosts/master/hosts.all
  [✗] Status: Not found
  [✗] List download failed: no cached list available

  [i] Target: https://raw.githubusercontent.com/Sekhan/TheGreatWall/master/TheGreatWall.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 132 domains
  [i] List stayed unchanged

  [i] Target: https://raw.githubusercontent.com/deep-bhatt/huawei-block-list/master/huawei-block-host.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 83 domains
  [i] List stayed unchanged

  [i] Target: https://v.firebog.net/hosts/Easylist.txt
  [✓] Status: No changes detected
  [i] Analyzed 16161 domains

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
  [✓] Status: Retrieval successful
  [i] Analyzed 9 domains
  [i] List stayed unchanged

  [i] Target: https://v.firebog.net/hosts/Admiral.txt
  [✓] Status: No changes detected
  [i] Analyzed 636 domains

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
  [✓] Status: No changes detected
  [i] Analyzed 2735 domains

  [i] Target: https://v.firebog.net/hosts/Prigent-Ads.txt
  [✓] Status: No changes detected
  [i] Analyzed 3670 domains

  [✓] Storing downloaded domains in new gravity database
  [✓] Building tree
  [✓] Swapping databases
  [✓] The old database remains available.
  [i] Number of gravity domains: 321670 (252316 unique domains)
  [i] Number of exact blacklisted domains: 60
  [i] Number of regex blacklist filters: 26
  [i] Number of exact whitelisted domains: 61
  [i] Number of regex whitelist filters: 1
  [✓] Flushing DNS cache
  [✓] Cleaning up stray matter

  [✓] FTL is listening on port 53
     [✓] UDP (IPv4)
     [✓] TCP (IPv4)
     [✓] UDP (IPv6)
     [✓] TCP (IPv6)

  [✓] Pi-hole blocking is enabled

Why not? Esp. if it's called /UnifiedHosts/master/hosts.all

Pi-hole can't do anything about removed adlists. It would have used a previously downloaded list, but in your case it seems it was never successfully downloaded:

[✗] List download failed: no cached list available

There is also the possibility that your other adlists removed a huge chunk of domains resulting in a drop of domains in your gravity database.

I forgot: I upgraded from the last available version. I always have the latest one.

Two invalid domains, yes. But that is no explanation for over 1 million less accounts.

I am relaxed by now. Maybe it just doesn't matter. Internet still works nicely. Maybe blocking was never so important.

Maybe the drop can be (at least partially) be explained by a recent change in the way Pi-hole presents the number of domains on the web interface. Now the raw numbers are delivered to the browser, letting the formatting happening inside the browser including localization. Depending on your country, the number separators might have changed.

Instead of around 30% of blocked domains, I am now down to below 10%.

I will just fly and let it happen.

I've dropped your gravity output in a file gravity.output.txt:

pi@ph5b:~ $ awk '/http/{print$ 3}' gravity.output.txt | xargs curl -sSL | sort | uniq | wc -l
294977

Numbers seem to match up with yours if not taking into account that I didn't check for valid domain names, which I believe Pi-hole does, and the lists might have changed in the meantime:

Post the output of the following command from the Pi terminal and I'll see how many domains end up in my gravity file:

sqlite3 /etc/pihole/gravity.db "SELECT address FROM adlist where enabled=1"

1 Like
pi@ph5b:~ $ awk '/http/{print$ 3}' gravity.output.txt
https://adaway.org/hosts.txt
https://v.firebog.net/hosts/AdguardDNS.txt
https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext
https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts
https://v.firebog.net/hosts/Easyprivacy.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt
https://raw.githubusercontent.com/Kees1958/W3C_annual_most_used_survey_blocklist/master/TOP_EU_US_Ads_Trackers_HOST
https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt
https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt
https://v.firebog.net/hosts/Prigent-Crypto.txt
https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
https://phishing.army/download/phishing_army_blocklist_extended.txt
https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
https://v.firebog.net/hosts/Shalla-mal.txt
https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
https://urlhaus.abuse.ch/downloads/hostfile/
https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser
https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
https://v.firebog.net/hosts/static/w3kbl.txt
https://raw.githubusercontent.com/CamelCase11/UnifiedHosts/master/hosts.all
https://raw.githubusercontent.com/Sekhan/TheGreatWall/master/TheGreatWall.txt
https://raw.githubusercontent.com/deep-bhatt/huawei-block-list/master/huawei-block-host.txt
https://v.firebog.net/hosts/Easylist.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
https://v.firebog.net/hosts/Admiral.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
https://v.firebog.net/hosts/Prigent-Ads.txt

EDIT: Noticed a typo above, "print$ 3" should be "print $3" but awk doesnt seem to mind :wink:

pi@ph5b:~ $ awk '/http/{print$ 3}' gravity.output.txt | wc -l
33
pi@ph5b:~ $ awk '/http/{print $3}' gravity.output.txt | wc -l
33
1 Like

We imported new lists. Up to 800.000 blocked domains. But not as much blocking as before. Still, a bit better.

The problem seems how to know what lists to use and especially to update every month. And keep looking for new lists.

"The master all" list was just a summary list. It was not the issue. We will not find out what was the problem.

Thanks a lot:)