Pi-hole update reduced number of blocked domains

Markus_Hope · January 13, 2022, 8:58am

That update reduced my number of blocked domains from 1,5 Million to 250.000. Same at a friend's. Something changed severly! Hardly anything is bloced anymore!

yubiuser · January 13, 2022, 10:05am

Nothing in the latest updated touched the code regarding the adlists.

Please upload a debug log and post just the token URL that is generated after the log is uploaded by running the following command from the Pi-hole host terminal:

pihole -d

or do it through the Web interface:

Tools > Generate Debug Log
__

Also, please post the output of pihole -g

Markus_Hope · January 13, 2022, 10:11am

Well, as I am not the only one having lost most blocked daomains after Update, the problem must be there. I did not change anything at my lists.

I attached some logs. Thanks!

https://tricorder.pi-hole.net/X9FQrWGU/

I will post the output of pihole -g later. The content has many links. New users are not allowed to post more than 5 links in this forum.

Markus_Hope · January 13, 2022, 10:14am

Here the output of pihole -g (part of it, Original has too many links). It receives a lot less domains than before! Same lists produced way more before. Or did all lists recently get reduced?

[i] Neutrino emissions detected...
[✓] Pulling blocklist source list into range

(......)

[✓] Storing downloaded domains in new gravity database
[✓] Building tree
[✓] Swapping databases
[✓] The old database remains available.
[i] Number of gravity domains: 321314 (252022 unique domains) (HERE THERE WERE OVER 1 MILLION!)
[i] Number of exact blacklisted domains: 60
[i] Number of regex blacklist filters: 26
[i] Number of exact whitelisted domains: 61
[i] Number of regex whitelist filters: 1
[✓] Flushing DNS cache
[✓] Cleaning up stray matter

[✓] FTL is listening on port 53
[✓] UDP (IPv4)
[✓] TCP (IPv4)
[✓] UDP (IPv6)
[✓] TCP (IPv6)

[✓] Pi-hole blocking is enabled

DanSchaper · January 13, 2022, 10:53am

Wrap the output with three backticks to start and end the block.

```
LOG OUTPUT

```

yubiuser · January 13, 2022, 11:19am

We need to look at the whole gravity output. Use the format Dan provided. From which version did you upgrade? Is there any output about invalid domains?

Markus_Hope · January 13, 2022, 7:28pm

Thanks so much for all the effort here. I know, two failed lists. But would those two lists contain over 1 million domains? I even think at least one was malfunctioning before.

Here is pihole -g

 [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [i] Using libz compression

  [i] Target: https://adaway.org/hosts.txt
  [✓] Status: No changes detected
  [i] Analyzed 7563 domains

  [i] Target: https://v.firebog.net/hosts/AdguardDNS.txt
  [✓] Status: No changes detected
  [i] Analyzed 43545 domains

  [i] Target: https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 42292 domains
  [i] List stayed unchanged

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
  [✓] Status: No changes detected
  [i] Analyzed 2701 domains

  [i] Target: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&amp;showintro=0&amp;mimetype=plaintext
  [✓] Status: No changes detected
  [i] Analyzed 3685 domains

  [i] Target: https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts
  [✓] Status: Retrieval successful
  [i] Analyzed 15264 domains
  [i] List stayed unchanged

  [i] Target: https://v.firebog.net/hosts/Easyprivacy.txt
  [✓] Status: No changes detected
  [i] Analyzed 12880 domains

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
  [✓] Status: Retrieval successful
  [i] Analyzed 1286 domains
  [i] List stayed unchanged

  [i] Target: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 378 domains
  [i] List stayed unchanged

  [i] Target: https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt
  [✓] Status: No changes detected
  [i] Analyzed 14039 domains

  [i] Target: https://raw.githubusercontent.com/Kees1958/W3C_annual_most_used_survey_blocklist/master/TOP_EU_US_Ads_Trackers_HOST
  [✗] Status: Not found
  [✗] List download failed: no cached list available

  [i] Target: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 6620 domains
  [i] List stayed unchanged

  [i] Target: https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt
  [✓] Status: No changes detected
  [i] Analyzed 918 domains

  [i] Target: https://v.firebog.net/hosts/Prigent-Crypto.txt
  [✓] Status: No changes detected
  [i] Analyzed 14148 domains

  [i] Target: https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
  [✓] Status: No changes detected
  [i] Analyzed 2046 domains

  [i] Target: https://phishing.army/download/phishing_army_blocklist_extended.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 53734 domains
  [i] List has been updated

  [i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 327 domains
  [i] List stayed unchanged

  [i] Target: https://v.firebog.net/hosts/Shalla-mal.txt
  [✓] Status: No changes detected
  [i] Analyzed 19239 domains

  [i] Target: https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 8148 domains
  [i] List stayed unchanged

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
  [✓] Status: Retrieval successful
  [i] Analyzed 2202 domains
  [i] List stayed unchanged

  [i] Target: https://urlhaus.abuse.ch/downloads/hostfile/
  [✓] Status: Retrieval successful
  [i] Analyzed 1473 domains
  [i] List has been updated

  [i] Target: https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser
  [✓] Status: No changes detected
  [i] Analyzed 3496 domains

  [i] Target: https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 41846 domains
  [i] List has been updated

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
  [✓] Status: Retrieval successful
  [i] Analyzed 57 domains
  [i] List stayed unchanged

  [i] Target: https://v.firebog.net/hosts/static/w3kbl.txt
  [✓] Status: No changes detected
  [i] Analyzed 357 domains

  [i] Target: https://raw.githubusercontent.com/CamelCase11/UnifiedHosts/master/hosts.all
  [✗] Status: Not found
  [✗] List download failed: no cached list available

  [i] Target: https://raw.githubusercontent.com/Sekhan/TheGreatWall/master/TheGreatWall.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 132 domains
  [i] List stayed unchanged

  [i] Target: https://raw.githubusercontent.com/deep-bhatt/huawei-block-list/master/huawei-block-host.txt
  [✓] Status: Retrieval successful
  [i] Analyzed 83 domains
  [i] List stayed unchanged

  [i] Target: https://v.firebog.net/hosts/Easylist.txt
  [✓] Status: No changes detected
  [i] Analyzed 16161 domains

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
  [✓] Status: Retrieval successful
  [i] Analyzed 9 domains
  [i] List stayed unchanged

  [i] Target: https://v.firebog.net/hosts/Admiral.txt
  [✓] Status: No changes detected
  [i] Analyzed 636 domains

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
  [✓] Status: No changes detected
  [i] Analyzed 2735 domains

  [i] Target: https://v.firebog.net/hosts/Prigent-Ads.txt
  [✓] Status: No changes detected
  [i] Analyzed 3670 domains

  [✓] Storing downloaded domains in new gravity database
  [✓] Building tree
  [✓] Swapping databases
  [✓] The old database remains available.
  [i] Number of gravity domains: 321670 (252316 unique domains)
  [i] Number of exact blacklisted domains: 60
  [i] Number of regex blacklist filters: 26
  [i] Number of exact whitelisted domains: 61
  [i] Number of regex whitelist filters: 1
  [✓] Flushing DNS cache
  [✓] Cleaning up stray matter

  [✓] FTL is listening on port 53
     [✓] UDP (IPv4)
     [✓] TCP (IPv4)
     [✓] UDP (IPv6)
     [✓] TCP (IPv6)

  [✓] Pi-hole blocking is enabled

yubiuser · January 14, 2022, 8:13am

Why not? Esp. if it's called /UnifiedHosts/master/hosts.all

Pi-hole can't do anything about removed adlists. It would have used a previously downloaded list, but in your case it seems it was never successfully downloaded:

[✗] List download failed: no cached list available

There is also the possibility that your other adlists removed a huge chunk of domains resulting in a drop of domains in your gravity database.

Markus_Hope · January 14, 2022, 8:37am

I forgot: I upgraded from the last available version. I always have the latest one.

Two invalid domains, yes. But that is no explanation for over 1 million less accounts.

I am relaxed by now. Maybe it just doesn't matter. Internet still works nicely. Maybe blocking was never so important.

yubiuser · January 14, 2022, 9:36am

Maybe the drop can be (at least partially) be explained by a recent change in the way Pi-hole presents the number of domains on the web interface. Now the raw numbers are delivered to the browser, letting the formatting happening inside the browser including localization. Depending on your country, the number separators might have changed.

github.com/pi-hole/web

Localize summary numbers

pi-hole:devel ← pi-hole:tweak/localized_numbers

opened 10:43AM - 26 Dec 21 UTC

DL6ER

+12 -24

**By submitting this pull request, I confirm the following:** - [X] I have r…ead and understood the [contributors guide](https://github.com/pi-hole/pi-hole/blob/master/CONTRIBUTING.md), as well as this entire template. - [X] I have made only one major change in my proposed changes. - [X] I have commented my proposed changes within the code. - [X] I have tested my proposed changes, and have included unit tests where possible. - [X] I am willing to help maintain this change if there are issues with it later. - [X] I give this submission freely and claim no ownership. - [X] It is compatible with the [EUPL 1.2 license](https://opensource.org/licenses/EUPL-1.1) - [X] I have squashed any insignificant commits. ([`git rebase`](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)) --- **What does this PR aim to accomplish?:** See title **How does this PR accomplish the above?:** Ensure number formatting is taking place in browser-scope to use client- instead of browser-based number localization. This relies on the locale being correctly applied in the user's browser. **Screenshots** ![ezgif-5-ebd4f24c10](https://user-images.githubusercontent.com/16748619/147405737-7249d824-4294-449b-b721-e7246ef728af.gif) **What documentation changes (if any) are needed to support this PR?:** None

Markus_Hope · January 14, 2022, 4:52pm

Instead of around 30% of blocked domains, I am now down to below 10%.

I will just fly and let it happen.

deHakkelaar · January 15, 2022, 12:26am

I've dropped your gravity output in a file gravity.output.txt:

pi@ph5b:~ $ awk '/http/{print$ 3}' gravity.output.txt | xargs curl -sSL | sort | uniq | wc -l
294977

Numbers seem to match up with yours if ~~not~~ taking into account that I didn't check for valid domain names, which I believe Pi-hole does, and the lists might have changed in the meantime:

jfb · January 15, 2022, 1:25am

Post the output of the following command from the Pi terminal and I'll see how many domains end up in my gravity file:

sqlite3 /etc/pihole/gravity.db "SELECT address FROM adlist where enabled=1"

deHakkelaar · January 15, 2022, 1:32am

pi@ph5b:~ $ awk '/http/{print$ 3}' gravity.output.txt
https://adaway.org/hosts.txt
https://v.firebog.net/hosts/AdguardDNS.txt
https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&amp;showintro=0&amp;mimetype=plaintext
https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts
https://v.firebog.net/hosts/Easyprivacy.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt
https://raw.githubusercontent.com/Kees1958/W3C_annual_most_used_survey_blocklist/master/TOP_EU_US_Ads_Trackers_HOST
https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt
https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt
https://v.firebog.net/hosts/Prigent-Crypto.txt
https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
https://phishing.army/download/phishing_army_blocklist_extended.txt
https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
https://v.firebog.net/hosts/Shalla-mal.txt
https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
https://urlhaus.abuse.ch/downloads/hostfile/
https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser
https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
https://v.firebog.net/hosts/static/w3kbl.txt
https://raw.githubusercontent.com/CamelCase11/UnifiedHosts/master/hosts.all
https://raw.githubusercontent.com/Sekhan/TheGreatWall/master/TheGreatWall.txt
https://raw.githubusercontent.com/deep-bhatt/huawei-block-list/master/huawei-block-host.txt
https://v.firebog.net/hosts/Easylist.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
https://v.firebog.net/hosts/Admiral.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
https://v.firebog.net/hosts/Prigent-Ads.txt

EDIT: Noticed a typo above, "print$ 3" should be "print $3" but awk doesnt seem to mind

pi@ph5b:~ $ awk '/http/{print$ 3}' gravity.output.txt | wc -l
33

pi@ph5b:~ $ awk '/http/{print $3}' gravity.output.txt | wc -l
33

Markus_Hope · January 15, 2022, 7:41am

We imported new lists. Up to 800.000 blocked domains. But not as much blocking as before. Still, a bit better.

The problem seems how to know what lists to use and especially to update every month. And keep looking for new lists.

"The master all" list was just a summary list. It was not the issue. We will not find out what was the problem. I just need to live with less blocked domains.

Thanks a lot:)

system · February 5, 2022, 7:41am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.