Ad lists are empty

sawsanders · November 26, 2024, 1:35pm

I ran sudo pihole -up my v6 Pi-hole install yesterday and I realized today that my blocked domains are only 170 compared to a million normally.

I ran sudo pihole -g from the command line and all but one of my lists are showing empty. They're downloading just fine on my backup pi (also running v6 but not updated from yesterday's merges).

I've tried deleting all the offline lists in the /etc/pihole directory and running gravity again, but that didn't change anything.

Pi-hole works just fine too. It's just not blocking many ads.

https://tricorder.pi-hole.net/OCoQaVM5/

DL6ER · November 26, 2024, 1:39pm

What's the output of

sudo pihole -g

in the terminal (or, on web https://pi.hole/admin/gravity)?

sawsanders · November 26, 2024, 1:40pm

$ sudo pihole -g
  [✓] DNS resolution is available

  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [✓] Creating new gravity databases
  [i] Using libz compression

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/static/w3kbl.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://adaway.org/hosts.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/AdguardDNS.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/Admiral.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/Easylist.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/Easyprivacy.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/Prigent-Ads.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/Prigent-Crypto.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://phishing.army/download/phishing_army_blocklist_extended.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
  [✓] Status: Retrieval successful
  [✓] Parsed 172 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)
  [i] List stayed unchanged

  [i] Target: https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://urlhaus.abuse.ch/downloads/hostfile/
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/chadmayfield/my-pihole-blocklists/master/lists/pi_blocklist_porn_all.list
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/guardicore/labs_campaigns/master/Autodiscover/autodiscover-tlds.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/AssoEchap/stalkerware-indicators/master/generated/hosts
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/RPiList-Malware.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/hosts/doh.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [✓] Building tree
  [i] Number of gravity domains: 172 (172 unique domains)
  [i] Number of exact denied domains: 7
  [i] Number of regex denied filters: 52
  [i] Number of exact allowed domains: 98
  [i] Number of regex allowed filters: 19
  [✓] Optimizing database
  [✓] Swapping databases
  [✓] The old database remains available
  [✓] Cleaning up stray matter

  [✓] Done.

DL6ER · November 26, 2024, 1:43pm

And if you run

sudo rm /etc/pihole/list.*
sudo pihole -g

?

sawsanders · November 26, 2024, 1:45pm

No change. I had tried that before too.

pi@pi5:~$ sudo rm /etc/pihole/list.*
pi@pi5:~$ sudo pihole -g
  [✓] DNS resolution is available

  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [✓] Creating new gravity databases
  [i] Using libz compression

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/static/w3kbl.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://adaway.org/hosts.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/AdguardDNS.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/Admiral.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/Easylist.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/Easyprivacy.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/Prigent-Ads.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/Prigent-Crypto.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://phishing.army/download/phishing_army_blocklist_extended.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
  [✓] Status: Retrieval successful
  [✓] Parsed 172 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://urlhaus.abuse.ch/downloads/hostfile/
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/chadmayfield/my-pihole-blocklists/master/lists/pi_blocklist_porn_all.list
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/guardicore/labs_campaigns/master/Autodiscover/autodiscover-tlds.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/AssoEchap/stalkerware-indicators/master/generated/hosts
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/RPiList-Malware.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/hosts/doh.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [✓] Building tree
  [i] Number of gravity domains: 172 (172 unique domains)
  [i] Number of exact denied domains: 7
  [i] Number of regex denied filters: 52
  [i] Number of exact allowed domains: 98
  [i] Number of regex allowed filters: 19
  [✓] Optimizing database
  [✓] Swapping databases
  [✓] The old database remains available
  [✓] Cleaning up stray matter

  [✓] Done.

sawsanders · November 26, 2024, 1:46pm

The offline list files show empty as well:

EDIT: Interestingly, root owns the hash file on the only list that is working. The rest are owned by pihole...

$ ls -l /etc/pihole/
total 475492
-rw-rw---- 1 pihole pihole        65 Sep  8 12:48 adlists.list
-rw-rw---- 1 pihole pihole        65 Sep  8 12:47 adlists.list.old
-rw-r----- 1 pihole pihole        44 Nov 26 05:21 cli_pw
drwxr-xr-x 2 pihole pihole      4096 Nov 21 05:49 config_backups
-rw-rw---- 1 pihole pihole         0 Sep  8 12:47 custom.list.bck
-rw-rw---- 1 pihole pihole       229 Sep  8 12:52 dhcp.leases
-rw-rw---- 1 pihole pihole       651 Nov 25 21:34 dns-servers.conf
-rw-rw---- 1 pihole pihole      6257 Oct 25 06:41 dnsmasq.conf
-rw-rw---- 1 pihole pihole        12 Oct 26 11:42 ftlbranch
-rw-rw-r-- 1 pihole pihole    126976 Nov 26 05:44 gravity.db
-rw-rw-r-- 1 pihole pihole    126976 Nov 26 05:40 gravity_old.db
drwxr-xr-x 2 pihole pihole      4096 Sep  8 12:52 hosts
-rw-rw---- 1 pihole pihole       408 Nov 25 21:34 install.log
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.118.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.119.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.120.adaway.org.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.121.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.122.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.123.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.124.s3.amazonaws.com.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.125.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.126.pgl.yoyo.org.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.127.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.128.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.129.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.130.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.132.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.133.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.134.hostfiles.frogeye.fr.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.136.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.137.osint.digitalside.it.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.138.s3.amazonaws.com.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.139.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.141.bitbucket.org.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.142.phishing.army.domains
-rw-r--r-- 1 pihole pihole      2538 Nov 26 05:44 list.143.gitlab.com.domains
-rw-r--r-- 1 root   root          82 Nov 26 05:44 list.143.gitlab.com.domains.sha1
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.145.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.146.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.147.urlhaus.abuse.ch.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.148.zerodot1.gitlab.io.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.149.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.150.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.151.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.154.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.155.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole         0 Nov 26 05:44 list.157.raw.githubusercontent.com.domains
-rw-rw---- 1 pihole pihole        65 Sep  8 12:47 local.list
-rw-r--r-- 1 root   root         412 Sep  8 12:48 logrotate
-rw-rw---- 1 pihole pihole   3215360 Nov 25 10:03 macvendor.db
drwxr-xr-x 2 pihole pihole      4096 Sep  8 12:47 migration_backup
drwxr-xr-x 2 pihole pihole      4096 Sep  8 12:48 migration_backup_v6
-rw-rw---- 1 pihole pihole       127 Sep  8 12:47 pihole-FTL.conf.bck
-rw-r----- 1 pihole pihole 481484800 Nov 25 21:35 pihole-FTL.db
-rw-r----- 1 pihole pihole     32768 Nov 26 05:46 pihole-FTL.db-shm
-rw-r----- 1 pihole pihole   1759272 Nov 26 05:46 pihole-FTL.db-wal
-rw-rw---- 1 pihole pihole     52790 Nov 21 05:49 pihole.toml
-rw-rw---- 1 pihole pihole       359 Sep  8 12:47 setupVars.conf.old
-rw-rw---- 1 pihole pihole       713 Sep  8 12:48 tls.crt
-rw-rw---- 1 pihole pihole      1001 Sep  8 12:48 tls.pem
-rw-rw---- 1 pihole pihole       733 Sep  8 12:48 tls_ca.crt
-rw-r--r-- 1 pihole pihole       365 Nov 26 05:21 versions

v3574c · November 26, 2024, 3:51pm

Same here...

[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling-porn/hosts
  [✓] Status: Retrieval successful
  [✓] Parsed 207582 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)
  [i] List stayed unchanged

  [i] Target: https://o0.pages.dev/Lite/domains.txt
  [✓] Status: Retrieval successful
  [✓] Parsed 132088 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)
  [i] List stayed unchanged

  [i] Target: https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/Easylist.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/Easyprivacy.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/AdguardDNS.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/Prigent-Ads.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/Prigent-Crypto.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/RPiList-Malware.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [i] Target: https://v.firebog.net/hosts/RPiList-Phishing.txt
  [✓] Status: No changes detected
  [✓] Parsed 0 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

DL6ER · November 26, 2024, 4:49pm

This is okay, it is a symptom rather than a cause. You are running pihole -g as root and newly created files are expected to be created as user root. However, Pi-hole only creates/updates the file in case there were changes so that's the reason why only one of your lists got an accompanying .sha1 file.

So much for now, I just arrived home and will come back to you in a bit. Consider this active-work-in-progress

buntspexxt · November 26, 2024, 5:00pm

Can you try

curl -i -v -k https://v.firebog.net/hosts/static/w3kbl.txt

and post the result please

DL6ER · November 26, 2024, 5:05pm

I am pretty sure I know what is causing this.

Please run

sudo pihole checkout core fix/gravity_heisenberg

and see if this fixes the issue for you.

sawsanders · November 26, 2024, 5:13pm

Alright, it worked. Gravity downloaded and processed all the lists!

I see some errors in the output however. All lists show an error such as this:

chown: cannot access '/etc/pihole/list.157.raw.githubusercontent.com.domains': No such file or directory
chmod: cannot access '/etc/pihole/list.157.raw.githubusercontent.com.domains': No such file or directory
  [i] Target: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/hosts/doh.txt
  [✓] Status: Retrieval successful
  [✓] Parsed 1684 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

I will try another gravity run and see if it is an artifact of the original problem.

EDIT: Confirmed. The errors are gone on the subsequent gravity run.

DL6ER · November 26, 2024, 5:17pm

Okay, but they shouldn't be there. Please delete the list.* files again and checkout again my custom fix branch above. I did just push another commit there that should prevent the errors from showing up.

DL6ER · November 26, 2024, 5:20pm

github.com/pi-hole/pi-hole

Fix empty adlists

pi-hole:development ← pi-hole:fix/gravity_heisenberg

opened 05:11PM - 26 Nov 24 UTC

DL6ER

+16 -9

# What does this implement/fix? Do not `touch` list files as this creates the…m. This causes issues down the line in the Heisenberg compensator. This fixes an unintended side-effect of #5819 --- **Related issue or feature (if applicable):** https://discourse.pi-hole.net/t/ad-lists-are-empty/74207 **Pull request in [docs](https://github.com/pi-hole/docs) with documentation (if applicable):** N/A --- **By submitting this pull request, I confirm the following:** 1. I have read and understood the [contributors guide](https://docs.pi-hole.net/guides/github/contributing/), as well as this entire template. I understand which branch to base my commits and Pull Requests against. 2. I have commented my proposed changes within the code. 3. I am willing to help maintain this change if there are issues with it later. 4. It is compatible with the [EUPL 1.2 license](https://opensource.org/licenses/EUPL-1.1) 5. I have squashed any insignificant commits. ([`git rebase`](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)) ## Checklist: - [x] The code change is tested and works locally. - [x] I based my code and PRs against the repositories `developmental` branch. - [x] I [signed off](https://docs.pi-hole.net/guides/github/how-to-signoff/) all commits. Pi-hole enforces the [DCO](https://docs.pi-hole.net/guides/github/dco/) for all contributions - [x] I [signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) all my commits. Pi-hole requires signatures to verify authorship - [x] I have read the above and my PR is ready for review.

is the proposed fix

sawsanders · November 26, 2024, 5:23pm

Ok, that cleared up the errors!

Also all the list* files are now owned by pihole even though I ran gravity as root (I'm guessing that is a good thing).

ls -l /etc/pihole/
total 986892
-rw-rw---- 1 pihole pihole        65 Sep  8 12:48 adlists.list
-rw-rw---- 1 pihole pihole        65 Sep  8 12:47 adlists.list.old
-rw-r----- 1 pihole pihole        44 Nov 26 09:19 cli_pw
drwxr-xr-x 2 pihole pihole      4096 Nov 21 05:49 config_backups
-rw-rw---- 1 pihole pihole         0 Sep  8 12:47 custom.list.bck
-rw-rw---- 1 pihole pihole       229 Sep  8 12:52 dhcp.leases
-rw-rw---- 1 pihole pihole       651 Nov 26 09:19 dns-servers.conf
-rw-rw---- 1 pihole pihole      6257 Oct 25 06:41 dnsmasq.conf
-rw-rw---- 1 pihole pihole        12 Oct 26 11:42 ftlbranch
-rw-rw-r-- 1 pihole pihole 222932992 Nov 26 09:20 gravity.db
-rw-rw---- 1 pihole pihole 222932992 Nov 26 09:15 gravity_old.db
drwxr-xr-x 2 pihole pihole      4096 Sep  8 12:52 hosts
-rw-rw---- 1 pihole pihole       408 Nov 26 09:19 install.log
-rw-rw-r-- 1 pihole pihole      1172 Nov 26 09:19 list.118.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole        97 Nov 26 09:19 list.118.raw.githubusercontent.com.domains.sha1
-rw-rw-r-- 1 pihole pihole      6898 Nov 26 09:19 list.119.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole        85 Nov 26 09:19 list.119.v.firebog.net.domains.sha1
-rw-rw-r-- 1 pihole pihole    138405 Nov 26 09:19 list.120.adaway.org.domains
-rw-rw-r-- 1 pihole pihole        82 Nov 26 09:19 list.120.adaway.org.domains.sha1
-rw-rw-r-- 1 pihole pihole    987169 Nov 26 09:19 list.121.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole        85 Nov 26 09:19 list.121.v.firebog.net.domains.sha1
-rw-rw-r-- 1 pihole pihole     15470 Nov 26 09:19 list.122.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole        85 Nov 26 09:19 list.122.v.firebog.net.domains.sha1
-rw-rw-r-- 1 pihole pihole   1013616 Nov 26 09:19 list.123.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole        97 Nov 26 09:19 list.123.raw.githubusercontent.com.domains.sha1
-rw-rw-r-- 1 pihole pihole     43529 Nov 26 09:19 list.124.s3.amazonaws.com.domains
-rw-rw-r-- 1 pihole pihole        88 Nov 26 09:19 list.124.s3.amazonaws.com.domains.sha1
-rw-rw-r-- 1 pihole pihole    537129 Nov 26 09:19 list.125.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole        85 Nov 26 09:19 list.125.v.firebog.net.domains.sha1
-rw-rw-r-- 1 pihole pihole     58503 Nov 26 09:19 list.126.pgl.yoyo.org.domains
-rw-rw-r-- 1 pihole pihole        84 Nov 26 09:19 list.126.pgl.yoyo.org.domains.sha1
-rw-rw-r-- 1 pihole pihole       204 Nov 26 09:19 list.127.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole        97 Nov 26 09:19 list.127.raw.githubusercontent.com.domains.sha1
-rw-rw-r-- 1 pihole pihole    383653 Nov 26 09:19 list.128.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole        97 Nov 26 09:19 list.128.raw.githubusercontent.com.domains.sha1
-rw-rw-r-- 1 pihole pihole    877305 Nov 26 09:19 list.129.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole        85 Nov 26 09:19 list.129.v.firebog.net.domains.sha1
-rw-rw-r-- 1 pihole pihole     60227 Nov 26 09:19 list.130.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole        85 Nov 26 09:19 list.130.v.firebog.net.domains.sha1
-rw-rw-r-- 1 pihole pihole     49889 Nov 26 09:19 list.132.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole        97 Nov 26 09:19 list.132.raw.githubusercontent.com.domains.sha1
-rw-rw-r-- 1 pihole pihole      9366 Nov 26 09:19 list.133.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole        97 Nov 26 09:19 list.133.raw.githubusercontent.com.domains.sha1
-rw-rw-r-- 1 pihole pihole    345292 Nov 26 09:19 list.134.hostfiles.frogeye.fr.domains
-rw-rw-r-- 1 pihole pihole        92 Nov 26 09:19 list.134.hostfiles.frogeye.fr.domains.sha1
-rw-rw-r-- 1 pihole pihole    500518 Nov 26 09:19 list.136.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole        97 Nov 26 09:19 list.136.raw.githubusercontent.com.domains.sha1
-rw-rw-r-- 1 pihole pihole      2429 Nov 26 09:19 list.137.osint.digitalside.it.domains
-rw-rw-r-- 1 pihole pihole        92 Nov 26 09:19 list.137.osint.digitalside.it.domains.sha1
-rw-rw-r-- 1 pihole pihole     44078 Nov 26 09:19 list.138.s3.amazonaws.com.domains
-rw-rw-r-- 1 pihole pihole        88 Nov 26 09:19 list.138.s3.amazonaws.com.domains.sha1
-rw-rw-r-- 1 pihole pihole    309519 Nov 26 09:19 list.139.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole        85 Nov 26 09:19 list.139.v.firebog.net.domains.sha1
-rw-rw-r-- 1 pihole pihole     43844 Nov 26 09:19 list.141.bitbucket.org.domains
-rw-rw-r-- 1 pihole pihole        85 Nov 26 09:19 list.141.bitbucket.org.domains.sha1
-rw-rw-r-- 1 pihole pihole   5329969 Nov 26 09:19 list.142.phishing.army.domains
-rw-rw-r-- 1 pihole pihole        85 Nov 26 09:19 list.142.phishing.army.domains.sha1
-rw-rw-r-- 1 pihole pihole      2538 Nov 26 09:19 list.143.gitlab.com.domains
-rw-rw-r-- 1 pihole pihole        82 Nov 26 09:19 list.143.gitlab.com.domains.sha1
-rw-rw-r-- 1 pihole pihole    146607 Nov 26 09:19 list.145.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole        97 Nov 26 09:19 list.145.raw.githubusercontent.com.domains.sha1
-rw-rw-r-- 1 pihole pihole     40645 Nov 26 09:19 list.146.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole        97 Nov 26 09:19 list.146.raw.githubusercontent.com.domains.sha1
-rw-rw-r-- 1 pihole pihole      3840 Nov 26 09:19 list.147.urlhaus.abuse.ch.domains
-rw-rw-r-- 1 pihole pihole        88 Nov 26 09:19 list.147.urlhaus.abuse.ch.domains.sha1
-rw-rw-r-- 1 pihole pihole     68862 Nov 26 09:19 list.148.zerodot1.gitlab.io.domains
-rw-rw-r-- 1 pihole pihole        90 Nov 26 09:19 list.148.zerodot1.gitlab.io.domains.sha1
-rw-rw-r-- 1 pihole pihole  50401261 Nov 26 09:20 list.149.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole        97 Nov 26 09:20 list.149.raw.githubusercontent.com.domains.sha1
-rw-rw-r-- 1 pihole pihole   1750448 Nov 26 09:20 list.150.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole        97 Nov 26 09:20 list.150.raw.githubusercontent.com.domains.sha1
-rw-rw-r-- 1 pihole pihole    228867 Nov 26 09:20 list.151.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole        97 Nov 26 09:20 list.151.raw.githubusercontent.com.domains.sha1
-rw-rw-r-- 1 pihole pihole     19339 Nov 26 09:20 list.154.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole        97 Nov 26 09:20 list.154.raw.githubusercontent.com.domains.sha1
-rw-rw-r-- 1 pihole pihole  15467845 Nov 26 09:20 list.155.v.firebog.net.domains
-rw-rw-r-- 1 pihole pihole        85 Nov 26 09:20 list.155.v.firebog.net.domains.sha1
-rw-rw-r-- 1 pihole pihole     30384 Nov 26 09:20 list.157.raw.githubusercontent.com.domains
-rw-rw-r-- 1 pihole pihole        97 Nov 26 09:20 list.157.raw.githubusercontent.com.domains.sha1
-rw-rw---- 1 pihole pihole        65 Sep  8 12:47 local.list
-rw-r--r-- 1 root   root         412 Sep  8 12:48 logrotate
-rw-rw---- 1 pihole pihole   3215360 Nov 25 10:03 macvendor.db
drwxr-xr-x 2 pihole pihole      4096 Sep  8 12:47 migration_backup
drwxr-xr-x 2 pihole pihole      4096 Sep  8 12:48 migration_backup_v6
-rw-rw---- 1 pihole pihole       127 Sep  8 12:47 pihole-FTL.conf.bck
-rw-r----- 1 pihole pihole 482050048 Nov 26 09:19 pihole-FTL.db
-rw-r----- 1 pihole pihole     32768 Nov 26 09:21 pihole-FTL.db-shm
-rw-r----- 1 pihole pihole    135992 Nov 26 09:21 pihole-FTL.db-wal
-rw-rw---- 1 pihole pihole     52790 Nov 21 05:49 pihole.toml
-rw-rw---- 1 pihole pihole       359 Sep  8 12:47 setupVars.conf.old
-rw-rw---- 1 pihole pihole       713 Sep  8 12:48 tls.crt
-rw-rw---- 1 pihole pihole      1001 Sep  8 12:48 tls.pem
-rw-rw---- 1 pihole pihole       733 Sep  8 12:48 tls_ca.crt
-rw-r--r-- 1 pihole pihole       376 Nov 26 09:20 versions

DL6ER · November 26, 2024, 5:24pm

Yes, it is. Thank you for confirming the fix so quickly and thoroughly at the same time!

austinmm6 · November 26, 2024, 6:05pm

I was having the same problem. I went from 1.3 million to 180,000 blocked domains. The checkout command definitely fixed it. I confirmed this accidentally by switching back to development too soon

v3574c · November 26, 2024, 7:38pm

Fix confirmed working on two Pi-Hole v6 instances. Thanks!

   528 -rw-rw-r--  1 pihole pihole    537129 Nov 26 19:14 list.13.v.firebog.net.domains
     4 -rw-rw-r--  1 pihole pihole        84 Nov 26 19:14 list.13.v.firebog.net.domains.sha1
   860 -rw-rw-r--  1 pihole pihole    877305 Nov 26 19:14 list.14.v.firebog.net.domains
     4 -rw-rw-r--  1 pihole pihole        84 Nov 26 19:14 list.14.v.firebog.net.domains.sha1
   968 -rw-rw-r--  1 pihole pihole    987169 Nov 26 19:14 list.15.v.firebog.net.domains
     4 -rw-rw-r--  1 pihole pihole        84 Nov 26 19:14 list.15.v.firebog.net.domains.sha1
    60 -rw-rw-r--  1 pihole pihole     60227 Nov 26 19:14 list.18.v.firebog.net.domains
     4 -rw-rw-r--  1 pihole pihole        84 Nov 26 19:14 list.18.v.firebog.net.domains.sha1
   304 -rw-rw-r--  1 pihole pihole    309519 Nov 26 19:14 list.19.v.firebog.net.domains
     4 -rw-rw-r--  1 pihole pihole        84 Nov 26 19:14 list.19.v.firebog.net.domains.sha1
 15108 -rw-rw-r--  1 pihole pihole  15467845 Nov 26 19:14 list.20.v.firebog.net.domains
     4 -rw-rw-r--  1 pihole pihole        84 Nov 26 19:14 list.20.v.firebog.net.domains.sha1
 10192 -rw-rw-r--  1 pihole pihole  10434870 Nov 26 19:14 list.21.v.firebog.net.domains
     4 -rw-rw-r--  1 pihole pihole        84 Nov 26 19:14 list.21.v.firebog.net.domains.sha1
  3908 -rw-rw-r--  1 pihole pihole   3999421 Nov 26 19:14 list.2.raw.githubusercontent.com.domains
     4 -rw-rw-r--  1 pihole pihole        95 Nov 26 19:14 list.2.raw.githubusercontent.com.domains.sha1

DL6ER · November 26, 2024, 8:12pm

The branch has been merged into development.

Please run

sudo pihole checkout core development

to get back on track and receive future updates. The branch we used here for testing will not receive any further updates.

Thank you all for helping confirming the fix!