FTL goes offline, Blocklists Not Added Back (Again)

MisterMister · March 5, 2020, 7:26pm

It's me again. I'm not sure what else I could troubleshoot, this has been happening daily. Again.

Every time these blocklists are queried, FTL goes offline. When it comes back up, a number of blocklists are missing. In other words, I should have 900K domains on my blocklists, but when system crashes, it comes back with 125K domains on the blocklists. (I'm aware the list maintainers handle these and make changes, but that's a HUGE difference). When I restart the system, the 900k domains comes back.

Thanks for any help.

Please follow the below template, it will help us to help you!

Expected Behaviour:

900k domains on Blocklists

Actual Behaviour:

FTL goes offline, FTL comes back with 125k domains

Debug Token:

https://tricorder.pi-hole.net/f91kh7ybbs

jfb · March 5, 2020, 7:35pm

What is the full output of the following commands from the Pi terminal:

cat /var/log/pihole_updateGravity.log

pihole -g -f

MisterMister · March 5, 2020, 7:44pm

pihole -g -f

 >  [✓] Deleting existing list cache
>   [i] Pi-hole blocking is enabled
>   [i] Neutrino emissions detected...
>   [✓] Pulling blocklist source list into range
> 
>   [i] Target: raw.githubusercontent.com (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: mirror1.malwaredomains.com (justdomains)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: sysctl.org (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: s3.amazonaws.com (simple_tracking.txt)
>   [✗] Status: s3.amazonaws.com is blocked by Whitelist. Using DNS on 208.67.222.222 to download https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: s3.amazonaws.com (simple_ad.txt)
>   [✗] Status: s3.amazonaws.com is blocked by Whitelist. Using DNS on 208.67.222.222 to download https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: hosts-file.net (ad_servers.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (Easyprivacy.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 208.67.222.222 to download https://v.firebog.net/hosts/Easyprivacy.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (Prigent-Ads.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 208.67.222.222 to download https://v.firebog.net/hosts/Prigent-Ads.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: gitlab.com (notrack-blocklist.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (spy.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: www.github.developerdan.com (ads-and-tracking-extended.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: hostfiles.frogeye.fr (firstparty-trackers-hosts.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: hostfiles.frogeye.fr (multiparty-trackers-hosts.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (android-tracking.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (SmartTV.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (AmazonFireTV.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (Airelle-trc.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 208.67.222.222 to download https://v.firebog.net/hosts/Airelle-trc.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: s3.amazonaws.com (simple_malvertising.txt)
>   [✗] Status: s3.amazonaws.com is blocked by Whitelist. Using DNS on 208.67.222.222 to download https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: hosts-file.net (exp.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: hosts-file.net (emd.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: mirror.cedia.org.ec (immortal_domains.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: www.malwaredomainlist.com (hosts.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: bitbucket.org (Mandiant_APT1_Report_Appendix_D.txt)
>   [✗] Status: bitbucket.org is blocked by Blacklist. Using DNS on 208.67.222.222 to download https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (Prigent-Malware.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 208.67.222.222 to download https://v.firebog.net/hosts/Prigent-Malware.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (Prigent-Phishing.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 208.67.222.222 to download https://v.firebog.net/hosts/Prigent-Phishing.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: phishing.army (phishing_army_blocklist_extended.txt)
>   [✗] Status: phishing.army is blocked by Blacklist. Using DNS on 208.67.222.222 to download https://phishing.army/download/phishing_army_blocklist_extended.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: gitlab.com (notrack-malware.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (Shalla-mal.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 208.67.222.222 to download https://v.firebog.net/hosts/Shalla-mal.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: gitlab.com (urlhaus-filter-hosts.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (url-lists.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (Airelle-hrsk.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 208.67.222.222 to download https://v.firebog.net/hosts/Airelle-hrsk.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: adaway.org (hosts.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (AdguardDNS.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 208.67.222.222 to download https://v.firebog.net/hosts/AdguardDNS.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (adservers.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (Easylist.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 208.67.222.222 to download https://v.firebog.net/hosts/Easylist.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: pgl.yoyo.org (serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: hosts-file.net (grm.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: reddestdream.github.io (minimalhosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (w3kbl.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 208.67.222.222 to download https://v.firebog.net/hosts/static/w3kbl.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (BillStearns.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 208.67.222.222 to download https://v.firebog.net/hosts/BillStearns.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: www.dshield.org (suspiciousdomains_Low.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: www.dshield.org (suspiciousdomains_Medium.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: www.dshield.org (suspiciousdomains_High.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: www.joewein.net (dom-bl-base.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (spammers.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: hostsfile.org (hosts.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: someonewhocares.org (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (main-blacklist.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: winhelp2002.mvps.org (hosts.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: hosts.nfz.moe (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (SNAFU.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: ssl.bblck.me (hosts-file.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (Kowabit.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 208.67.222.222 to download https://v.firebog.net/hosts/Kowabit.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (ads-and-tracking.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: www.github.developerdan.com (facebook-extended.txt)
>   [✓] Status: Retrieval successful
> 
>   [✓] Consolidating blocklists
>   [✓] Extracting domains from blocklists
>   [i] Number of domains being pulled in by gravity: 1603645
>   [✓] Removing duplicate domains
>   [i] Number of unique domains trapped in the Event Horizon: 1174816
>   [i] Number of whitelisted domains: 104
>   [i] Number of blacklisted domains: 241
>   [i] Number of regex filters: 91
>   [✓] Parsing domains into hosts format
>   [✓] Cleaning up stray matter
> 
>   [✓] Force-reloading DNS service
>   [✓] DNS service is running
>   [✓] Pi-hole blocking is Enabled

MisterMister · March 5, 2020, 7:47pm

'cat /var/log/pihole_updateGravity.log` gives me:

> [i] Pi-hole blocking is enabled
>   [i] Neutrino emissions detected...
>   [✓] Pulling blocklist source list into range
> 
>   [i] Target: raw.githubusercontent.com (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: mirror1.malwaredomains.com (justdomains)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: sysctl.org (hosts)
>   [✓] Status: No changes detected
> 
>   [i] Target: s3.amazonaws.com (simple_tracking.txt)
>   [✗] Status: s3.amazonaws.com is blocked by Whitelist. Using DNS on 8.8.8.8 to download https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
>   [✓] Status: No changes detected
> 
>   [i] Target: s3.amazonaws.com (simple_ad.txt)
>   [✗] Status: s3.amazonaws.com is blocked by Whitelist. Using DNS on 8.8.8.8 to download https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
>   [✓] Status: No changes detected
> 
>   [i] Target: hosts-file.net (ad_servers.txt)
>   [✓] Status: No changes detected
> 
>   [i] Target: v.firebog.net (Easyprivacy.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 8.8.8.8 to download https://v.firebog.net/hosts/Easyprivacy.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (Prigent-Ads.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 8.8.8.8 to download https://v.firebog.net/hosts/Prigent-Ads.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: gitlab.com (notrack-blocklist.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (spy.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (w3tech_hostfile.txt)
>   [✗] Status: Not found
>   [✗] List download failed: using previously cached list
> 
>   [i] Target: www.github.developerdan.com (ads-and-tracking-extended.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: hostfiles.frogeye.fr (firstparty-trackers-hosts.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: hostfiles.frogeye.fr (multiparty-trackers-hosts.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (android-tracking.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (SmartTV.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (AmazonFireTV.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (Airelle-trc.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 8.8.8.8 to download https://v.firebog.net/hosts/Airelle-trc.txt
>   [✓] Status: No changes detected
> 
>   [i] Target: s3.amazonaws.com (simple_malvertising.txt)
>   [✗] Status: s3.amazonaws.com is blocked by Whitelist. Using DNS on 8.8.8.8 to download https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: hosts-file.net (exp.txt)
>   [✓] Status: No changes detected
> 
>   [i] Target: hosts-file.net (emd.txt)
>   [✓] Status: No changes detected
> 
>   [i] Target: mirror.cedia.org.ec (immortal_domains.txt)
>   [✓] Status: No changes detected
> 
>   [i] Target: www.malwaredomainlist.com (hosts.txt)
>   [✓] Status: No changes detected
> 
>   [i] Target: bitbucket.org (Mandiant_APT1_Report_Appendix_D.txt)
>   [✗] Status: bitbucket.org is blocked by Blacklist. Using DNS on 8.8.8.8 to download https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
>   [✓] Status: No changes detected
> 
>   [i] Target: v.firebog.net (Prigent-Malware.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 8.8.8.8 to download https://v.firebog.net/hosts/Prigent-Malware.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (Prigent-Phishing.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 8.8.8.8 to download https://v.firebog.net/hosts/Prigent-Phishing.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: phishing.army (phishing_army_blocklist_extended.txt)
>   [✗] Status: phishing.army is blocked by Blacklist. Using DNS on 8.8.8.8 to download https://phishing.army/download/phishing_army_blocklist_extended.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: gitlab.com (notrack-malware.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (Shalla-mal.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 8.8.8.8 to download https://v.firebog.net/hosts/Shalla-mal.txt
>   [✓] Status: No changes detected
> 
>   [i] Target: raw.githubusercontent.com (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: www.squidblacklist.org (dg-malicious.acl)
>   [✗] Status: Connection Timed Out (Cloudflare)
>   [✗] List download failed: using previously cached list
> 
>   [i] Target: gitlab.com (urlhaus-filter-hosts.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (url-lists.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (Airelle-hrsk.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 8.8.8.8 to download https://v.firebog.net/hosts/Airelle-hrsk.txt
>   [✓] Status: No changes detected
> 
>   [i] Target: adaway.org (hosts.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (AdguardDNS.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 8.8.8.8 to download https://v.firebog.net/hosts/AdguardDNS.txt
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (adservers.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: v.firebog.net (Easylist.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 8.8.8.8 to download https://v.firebog.net/hosts/Easylist.txt
>   [✓] Status: No changes detected
> 
>   [i] Target: pgl.yoyo.org (serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext)
>   [✓] Status: No changes detected
> 
>   [i] Target: raw.githubusercontent.com (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: www.squidblacklist.org (dg-ads.acl)
>   [✗] Status: Connection Timed Out (Cloudflare)
>   [✗] List download failed: using previously cached list
> 
>   [i] Target: raw.githubusercontent.com (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: hosts-file.net (grm.txt)
>   [✓] Status: No changes detected
> 
>   [i] Target: reddestdream.github.io (minimalhosts)
>   [✓] Status: No changes detected
> 
>   [i] Target: v.firebog.net (w3kbl.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 8.8.8.8 to download https://v.firebog.net/hosts/static/w3kbl.txt
>   [✓] Status: No changes detected
> 
>   [i] Target: v.firebog.net (BillStearns.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 8.8.8.8 to download https://v.firebog.net/hosts/BillStearns.txt
>   [✓] Status: No changes detected
> 
>   [i] Target: www.dshield.org (suspiciousdomains_Low.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: www.dshield.org (suspiciousdomains_Medium.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: www.dshield.org (suspiciousdomains_High.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: www.joewein.net (dom-bl-base.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (spammers.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: hostsfile.org (hosts.txt)
>   [✓] Status: No changes detected
> 
>   [i] Target: someonewhocares.org (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (main-blacklist.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: winhelp2002.mvps.org (hosts.txt)
>   [✓] Status: No changes detected
> 
>   [i] Target: hosts.nfz.moe (hosts)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: raw.githubusercontent.com (SNAFU.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: ssl.bblck.me (hosts-file.txt)
>   [✓] Status: No changes detected
> 
>   [i] Target: v.firebog.net (Kowabit.txt)
>   [✗] Status: v.firebog.net is blocked by Blacklist. Using DNS on 8.8.8.8 to download https://v.firebog.net/hosts/Kowabit.txt
>   [✓] Status: No changes detected
> 
>   [i] Target: raw.githubusercontent.com (ads-and-tracking.txt)
>   [✓] Status: Retrieval successful
> 
>   [i] Target: (^|\.)buffpanel\.com$ ((^|\.)buffpanel\.com$)
>   [✗] Status: Connection Refused
>   [✗] List download failed: no cached list available
> 
>   [i] Target: (^|\.)redshell\.io$ ((^|\.)redshell\.io$)
>   [✗] Status: Connection Refused
>   [✗] List download failed: no cached list available
> 
>   [i] Target: (^|\.)treasuredata\.com$ ((^|\.)treasuredata\.com$)
>   [✗] Status: Connection Refused
>   [✗] List download failed: no cached list available
> 
>   [i] Target: (^|\.)unity(|3d)\.com$ ((^|\.)unity(|3d)\.com$)
>   [✗] Status: Connection Refused
>   [✗] List download failed: no cached list available
> 
>   [i] Target: (^|\.)unityads(|\.co)\.com$ ((^|\.)unityads(|\.co)\.com$)
>   [✗] Status: Connection Refused
>   [✗] List download failed: no cached list available
> 
>   [i] Target: www.github.developerdan.com (facebook-extended.txt)
>   [✓] Status: Retrieval successful
> 
>   [✓] Consolidating blocklists
>   [✓] Extracting domains from blocklists
>   [i] Number of domains being pulled in by gravity: 1747702
>   [✓] Removing duplicate domains
>   [i] Number of unique domains trapped in the Event Horizon: 1237531
>   [i] Number of whitelisted domains: 103
>   [i] Number of blacklisted domains: 228
>   [i] Number of regex filters: 91
>   [✓] Parsing domains into hosts format
>   [✓] Cleaning up stray matter
> 
>   [✓] DNS service is running
>   [✓] Pi-hole blocking is Enabled
>

MisterMister · March 5, 2020, 7:51pm

MisterMister:

[i] Target: (^|.)buffpanel.com$ ((^|.)buffpanel.com$)
[✗] Status: Connection Refused
[✗] List download failed: no cached list available

[i] Target: (^|.)redshell.io$ ((^|.)redshell.io$)
[✗] Status: Connection Refused
[✗] List download failed: no cached list available

[i] Target: (^|.)treasuredata.com$ ((^|.)treasuredata.com$)
[✗] Status: Connection Refused
[✗] List download failed: no cached list available

[i] Target: (^|.)unity(|3d).com$ ((^|.)unity(|3d).com$)
[✗] Status: Connection Refused
[✗] List download failed: no cached list available

[i] Target: (^|.)unityads(|.co).com$ ((^|.)unityads(|.co).com$)
[✗] Status: Connection Refused
[✗] List download failed: no cached list available

As a side note, I removed these, or I thought I did. What else do I need to do delete these once and for all?

jfb · March 5, 2020, 8:08pm

You did remove them. The log you printed was for the weekly gravity update run on Sunday morning. The gravity update you just ran did not contain those improper entries.

Please elaborate. Is it every time gravity updates that you have a problem? Or, what is the specific sequence of events that leads to FTL crashing?

This error appears in your debug log - what is the line in question?

[2020-03-05 09:12:20.092 927] ERROR compiling regex on line 24: Trailing backslash (5)

MisterMister · March 5, 2020, 8:41pm

I suppose when Gravity updates. (wow, I feel stupid, I didn't know Gravity updates on its own? But most cases it's when I see a list of blocklists in the query log right before it goes offline and return with much less number of blocked domains).

I don't know what regex line 24 is referring to.

edit: for clarification

edit 2: I updated Gravity just now and now I'm back down to 125k domains.

jfb · March 5, 2020, 9:35pm

What is the output of that gravity update?

This is line 24 of the regex list. Post the output of the following command (will be many lines):

cat -n /etc/pihole/regex.list

MisterMister · March 5, 2020, 9:45pm

edit: (Apologies for the poor formatting earlier in this thread)

the output:

(ads|logs|cloudservices).roku.com$
(^|\.)scribe\.logs\.roku\.com$
(^|\.)cloudservices\.roku\.com$
(^|\.)dishanalytics\.com$
(^|\.)facebook\.[A-Za-z0-9]+$
(^|\.)fb\.[A-Za-z0-9]+$
(^|\.)fbcdn\.[A-Za-z0-9]+$
(^|\.)fbsbx\.com\.online-metrix\.net$
(^|\.)whatsapp\.com$
^(.+\.)?amp\.cloudflare\.com$
^(.+\.)?cdn\.ampproject\.org$
^([a-z0-9]+[.])*(ads|captive|cloudservices|logs)\.roku\.com
^https?://([A-Za-z0-9.-]*\.)?clicks\.beap\.bc\.yahoo\.com/
^https?://([A-Za-z0-9.-]*\.)?scorecardresearch(\.\w{2}\.\w{2}|\.\w{2,4})/
^https?://([A-Za-z0-9.-]*\.)?doubleclick(\.\w{2}\.\w{2}|\.\w{2,4})/
(^|\.)bugsnag\.com$
(^|\.)bugsnag\.com$
(^|\.)sqm\.telemetry\.microsoft\.com$
(^|\.)www-roku\.mgo\.com$
(^|\.)ade\.googlesyndication\.com$
(^|\.)pagead2\.googlesyndication\.com$
(^|\.)marketing\.net$
^marketing\.net\
(^|\.)atc\.eulerian\.net$
(^|\.) litix\.io$
(^|\.)2mdn\.net$
(^|\.)adrise\.tv$
(^|\.)adsrvr\.org$
(^|\.)aragoncreek\.com$
(^|\.)bfmio\.com$
(^|\.)bidswitch\.net$
(^|\.)conviva\.com$
(^|\.)demdex\.net$
(^|\.)digitru\.st$
(^|\.)ewscloud\.com$
(^|\.)ifood\.tv$
(^|\.)imrworldwide\.com$
(^|\.)innovid\.com$
(^|\.)irchan\.com$
(^|\.)junctiontv\.net$
(^|\.)kargo\.com$
(^|\.)lightcast\.com$
(^|\.)mtvnservices\.com$
(^|\.)myspotlight\.tv$
(^|\.)omtrdc\.net$
(^|\.)sharethrough\.com$
(^|\.)titantv\.com$
(^|\.)tremorhub\.com$
(^|\.)wsi\.com$
(^|\.)xpressunblock\.com$
(^|\.)youboranqs01\.com$
(^|\.|-)dns-leaks[-_\.]
(^|\.)ad\.doubleclick\.net$
(^|\.)ade\.googlesyndication\.com$
(ads|captive|logs).roku.com$
^([a-z0-9]+[.])*(ads|captive|cloudservices|logs)\.roku\.com
(^|\.)bugsnag\.com$
^https?://([A-Za-z0-9.-]*\.)?clicks\.beap\.bc\.yahoo\.com/
^https?://([A-Za-z0-9.-]*\.)?secure\.footprint\.net/
^https?://([A-Za-z0-9.-]*\.)?match\.com/
^https?://([A-Za-z0-9.-]*\.)?surveylink/
^https?://([A-Za-z0-9.-]*\.)?info\.yahoo\.com/
^https?://([A-Za-z0-9.-]*\.)?scorecardresearch\.com/
^https?://([A-Za-z0-9.-]*\.)?advertising\.com/
^https?://([A-Za-z0-9.-]*\.)?s\.yimg\.com/cv/ae/us/audience/
^https?://([A-Za-z0-9.-]*\.)?googlesyndication(\.\w{2}\.\w{2}|\.\w{2,4})/
^(.+[-_.])??telemetry[-.]
^(.+[-_.])??ad[sxv]?[0-9]*[-_.]
^adtrack(er|ing)?[0-9]*[-.]
^analytics?[-.]
^beacons?[0-9]*[-.]
^stat(s|istics)?[0-9]*[-.]
^(.+\.)??(.*(v10|v20)[-.a-z0-9]*(events|vortex).+|.+(\.1drv|\.llnw)|.*([-.]win[-.]).+|[a-z0-9]+(\.ms[a-z]+)|.*(ad|aria|data|spynet[a-z0-9]*|telemetry|vortex|watson).+microsoft(.+)?|.*(wns|telemetry).+windows(azure)?)\.(com|net)$
(.*\.|^)((think)?with)?google($|((adservices|apis|mail|static|syndication|tagmanager|tagservices|usercontent|zip|-analytics)($|\..+)))
(ads|captive|logs).roku.com$
(ads|logs|cloudservices).roku.com$
(^|\.)s3\.amazonaws\.com$
(^|\.)Airties\.com$
(^|\.|-)dns-leaks[-_\.]
(^|\.)youboranqs01\.com$
(^|\.)xpressunblock\.com$
(^|\.)myspotlight\.tv$
(^|\.)mtvnservices\.com$
(^|\.)digitru\.st$
(^|\.)adsrvr\.org$
(^|\.)dp2analytics\.ampsvc\.com$
(^|\.)fama\.io$
^(.+[-_.])??xn--
.+(g00).+
(^|\.)Fwmwm\.net$```

jfb · March 5, 2020, 9:51pm

This is the line that is throwing the compile error:

(^|\.)atc\.eulerian\.net$

There are also a number of lines that are improperly written. https should never be part of a domain name (particularly leading), as it is a protocol and not part of the domain name. All the lines that lead with "^https" need to be cleaned up or deleted.

DanSchaper · March 5, 2020, 9:53pm

It's possible that the line preceding it is the problem.

^marketing\.net\

But that line is redundant as preceding that is (^|\.)marketing\.net$

MisterMister · March 5, 2020, 9:55pm

Okay, I will get right on it. Thanks for all your help!

jfb · March 5, 2020, 9:56pm

Regex tester here:

Note that because of the version of Regex run by Pi-hole (POSIX ERE), some regex that pass the test there won't work in Pi-hole, but it's a start to see if your regex functions as intended.

MisterMister · March 5, 2020, 10:05pm

This is very helpful. Thank you!

system · March 26, 2020, 10:05pm

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.