access docs.google.com (https://docs.google.com)
Pi-Hole is returning address for docs.google.com as 0.0.0.0
https://tricorder.pi-hole.net/qxst69i1dk!
Thank you very much.
With almost 9.8 million domains blocked and 305 subscribed blocklists, this is likely in your gravity list.
pihole -q -adlist -all docs.google.com will show you which lists contain this domain.
sudo grep docs.google.com /etc/pihole/gravity.list will check to see if the domain is currently in gravity
Following is the output. Any idea why the white-listing is not working?
Is there a command to delete this from the blocklist entries and not the entire blocklist itself.
Match found in Whitelist
docs.google.com
www.docs.google.com
Match found in https://raw.githubusercontent.com/EnergizedProtection/block/master/ultimate/formats/hosts.txt:
79.docs.google.com
dalk4zrp4jp3q.cloudfront.netdocs.google.come.thirdpresence.com
Match found in https://dbl.oisd.nl/:
dalk4zrp4jp3q.cloudfront.netdocs.google.come.thirdpresence.com
docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67748530257.website-hosting-thailand.com
docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67753358589.website-hosting-thailand.com
www.docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67748530257.website-hosting-thailand.com
www.docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67753358589.website-hosting-thailand.com
Match found in https://hosts-file.net/psh.txt:
docs.google.com-filesattachment.themineralqueen.com
www.docs.google.com-filesattachment.themineralqueen.com
Match found in http://phishing.mailscanner.info/phishing.bad.sites.conf:
79.docs.google.com
Match found in https://urlhaus.abuse.ch/downloads/text/:
docs.google.com
Match found in https://smokingwheels.github.io/Pi-hole/allhosts:
docs.google.com-filesattachment.themineralqueen.com
docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67748530257.website-hosting-thailand.com
docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67753358589.website-hosting-thailand.com
docs.google.com.lonf.dk
www.docs.google.com-filesattachment.themineralqueen.com
www.docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67748530257.website-hosting-thailand.com
www.docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67753358589.website-hosting-thailand.com
www.docs.google.com.lonf.dk
Match found in https://ncloud.zaclys.com/index.php/s/MPYSjLkb8cE8gPN/download:
www.docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67748530257.website-hosting-thailand.com
.docs.google.com
w.docs.google.com-filesattachment.themineralqueen.com
www.docs.google.com.lonf.dk
www.docs.google.com-filesattachment.themineralqueen.com
www.docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67753358589.website-hosting-thailand.com
docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67753358589.website-hosting-thailand.com
docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67748530257.website-hosting-thailand.com
w.docs.google.com.lonf.dk
docs.google.com.lonf.dk
docs.google.com-filesattachment.themineralqueen.com
Match found in http://data.phishtank.com/data/online-valid.csv:
5860114,https://docs.google.com/document/d/1IGf88eawenXPw2wRjHq0Sz1fO_NdzHPp,http://www.phishtank.com/phish_detail.php?phish_id=5860114,2018-11-29T15:02:35+00:00,yes,2018-12-11T04:25:04+00:00,yes,Other
5817739,https://docs.google.com/document/d/1zKs-G2rmRPI4od2kaAtem3IDzdA1pbYCwX4R5QZYIWw/edit,http://www.phishtank.com/phish_detail.php?phish_id=5817739,2018-10-17T07:54:09+00:00,yes,2018-10-23T20:57:32+00:00,yes,Other
5817738,https://docs.google.com/document/d/1zKs-G2rmRPI4od2kaAtem3IDzdA1pbYCwX4R5QZYIWw,http://www.phishtank.com/phish_detail.php?phish_id=5817738,2018-10-17T07:54:08+00:00,yes,2018-10-26T00:18:17+00:00,yes,Other
5799037,https://docs.google.com/forms/d/e/1FAIpQLSfttS12wZ9DLNWcmDvgE38oWrINS8y7cxnYT_fTNdAcEf03GA/viewform?usp=send_form,http://www.phishtank.com/phish_detail.php?phish_id=5799037,2018-09-29T01:21:04+00:00,yes,2018-09-30T20:43:13+00:00,yes,Other
5792604,https://docs.google.com/document/d/1zDX5RheFiN6fdspDuzEifl1TPYspMzKIgrtTXC5fGO8/edit,http://www.phishtank.com/phish_detail.php?phish_id=5792604,2018-09-24T04:17:24+00:00,yes,2018-09-24T21:12:16+00:00,yes,Other
5714738,https://docs.google.com/forms/d/e/1FAIpQLSfpz-d1xY39MgiAGUJwdOz9Y67SZt_2cgyv58h3M8SqvhL6oQ/viewform,http://www.phishtank.com/phish_detail.php?phish_id=5714738,2018-07-05T14:18:45+00:00,yes,2018-07-05T14:22:36+00:00,yes,Other
5630606,https://docs.google.com/forms/d/e/1FAIpQLSdw7WWLMe5NBmdW5bohTmHDl3Uyh4bKaoCq0P7I7wbcuSoWeg/viewform,http://www.phishtank.com/phish_detail.php?phish_id=5630606,2018-05-18T20:50:56+00:00,yes,2018-05-22T13:32:24+00:00,yes,Other
5630605,https://docs.google.com/forms/d/e/1FAIpQLSdw7WWLMe5NBmdW5bohTmHDl3Uyh4bKaoCq0P7I7wbcuSoWeg/viewform?usp=send_form,http://www.phishtank.com/phish_detail.php?phish_id=5630605,2018-05-18T20:50:36+00:00,yes,2018-05-27T01:59:21+00:00,yes,Other
5605988,https://docs.google.com/forms/d/e/1FAIpQLSedi778KA_QpFW-6VseK02w_qghyAY4KrmhVAHDl-qBzUQytg/viewform,http://www.phishtank.com/phish_detail.php?phish_id=5605988,2018-04-29T00:47:51+00:00,yes,2018-05-14T14:27:19+00:00,yes,Other
5370021,http://optico-koszalin.pl/wp-includes/js/docs.google.com/9feab007af9cd8aa78c31ef99bbf7e7d/,http://www.phishtank.com/phish_detail.php?phish_id=5370021,2017-12-07T06:44:40+00:00,yes,2017-12-09T16:06:47+00:00,yes,Google
3496688,https://79.docs.google.com/,http://www.phishtank.com/phish_detail.php?phish_id=3496688,2015-09-29T03:21:04+00:00,yes,2015-12-08T23:28:26+00:00,yes,Other
3496687,http://79.docs.google.com/,http://www.phishtank.com/phish_detail.php?phish_id=3496687,2015-09-29T03:21:03+00:00,yes,2015-10-12T14:32:14+00:00,yes,Other
Match found in https://raw.githubusercontent.com/botherder/targetedthreats/master/targetedthreats.csv:
"domain","docs.google.com.publicvm.com","Misc","China","https://targetedthreats.net/"
Match found in https://raw.githubusercontent.com/Neo23x0/signature-base/master/iocs/otx-c2-iocs.txt:
https://docs.google.com/document/u/1/d/e/2PACX-1vR2TWm68bLidO3e2X0wTCqs0609vo5RX
What was the output of the second command searching the gravity list?
This is what putting the domain in the whitelist does. When gravity is rebuilt, whitelisted domains are removed from gravity. We need to check your gravity list to see if this domain is still in there.
Output of sudo grep docs.google.com /etc/pihole/gravity.list
.docs.google.com
79.docs.google.com
dalk4zrp4jp3q.cloudfront.netdocs.google.come.thirdpresence.com
docs.google.com-filesattachment.themineralqueen.com
docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67748530257.website-hosting-thailand.com
docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67753358589.website-hosting-thailand.com
docs.google.com.lonf.dk
w.docs.google.com-filesattachment.themineralqueen.com
w.docs.google.com.lonf.dk
www.docs.google.com-filesattachment.themineralqueen.com
www.docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67748530257.website-hosting-thailand.com
www.docs.google.com-session.id3432534641f6a850a564167e47e1fdd0fdacef8342d42f0ad67753358589.website-hosting-thailand.com
www.docs.google.com.lonf.dkAlso, I can see docs.google.com in my whitelist. When I try to add it again it says:
[i] docs.google.com already exists in whitelist, no need to add!
[i] docs.google.com does not exist in blacklist, no need to remove!
add this exact domain to your whitelist
.docs.google.com
I am getting the following error:
Failure! Something went wrong, see output below:
.docs.google.com is not a valid domain
Can i use nano to open the file and delete the entry?
Do i need to disable service(s) before doing so?
Is the gravity.list file having some integrity monitoring which may break if I edit manually?
The cause of this problem is that you are using some block lists that are not in hosts format and Pi-Hole is importing some of the domains that have a leading ".". These should not appear in a hosts format blocklist.
You have a few options:
Get rid of this blocklist and rebuild gravity - https://ncloud.zaclys.com/index.php/s/MPYSjLkb8cE8gPN/download
Try adding the .docs.google.com domain to your whitelist by editing file /etc/pihole/whitelist.txt, then rebuilding gravity.
The permanent fix is to look carefully at the blocklists to which you subscribe and eliminate those that aren't properly formatted.
For example, post the output of these commands, and see the junk in there.
tail -n20 /etc/pihole/gravity.list
head -n20 /etc/pihole/gravity.list
sudo grep https: /etc/pihole/gravity.list | tail -n30
Looking at some of your 305 subscribed blocklists, I find many that aren't in hosts format. You are basically importing junk into your gravity list, and this is causing you problems.
https://sslbl.abuse.ch/blacklist/sslblacklist.csv
https://feodotracker.abuse.ch/downloads/ipblocklist.csv
http://data.phishtank.com/data/online-valid.csv
...
and there are many more.
This list in particular is fairly representative of some on your block list. It clearly states not to use as a blocklist, and it does not contain domain names - it is a list of IP's that are of no use to Pi-Hole.
http://www.dshield.org/ipsascii.html?limit=10000
# ipsascii.html
# DO NOT USE AS BLOCKLIST
# you may obtain longer/shorter lists by adding '?limit=1234' to the URL
# "1234" is the number of IPs returned. You can specify a number between 10
# and 10,000.
# NOTE: Please use your API instead. See https://isc.sans.edu/api
#
# IMPORTANT: Include your e-mail address in the user agent if you are using
# a script to access this page.
#
# created: Fri, 15 Mar 2019 20:34:43 +0000# Date in GMT. YYYY-MM-DD format.
077.072.085.026 220891 9442 2018-12-27 2019-02-15
185.176.027.118 246484 9395 2019-02-02 2019-03-15
185.254.123.012 88391 8854 2019-01-10 2019-02-18
193.032.160.035 112718 8745 2019-01-18 2019-03-15
If I may ask, what is your purpose in blocking almost 10 million domains? And, how are you screening the block lists to which you subscribe? Have you looked at the contents of any of those lists, or are you just finding them listed online and adding them to your blocklists?
Thank you for all the prompt replies. I have deleted lists that aren't giving data in the format required or data that is required (IPs as opposed to hostname, which DNS is queried for).
I took the lists I could find from the internet (most are from threads on discourse.pi-hole.net).
I have manually ran an update post deletion of the list. Is there a specific command to "rebuild" gravity list?
Thank you.
pihole -g rebuilds gravity.
Ah thank you very much. it is the same switch as updategravity.
I will post output of tail, head and grep commands in sometime. I have deleted lost of lists too. You gave me a text file, how do I import it to my list?
I didn't give you a file to import into any lists.
https://ncloud.zaclys.com/index.php/s/MPYSjLkb8cE8gPN/download
This link invoked a download which is a text file with ~400,000 domains. All having loopback address in front of them.
tail -n20 /etc/pihole/gravity.list
|http:*=$popup,domain=grammarist.com
|https:$domain=adfarm1.adition.com
|https:$popup,third-party,domain=eztv.ag|flashx.tv|imagerar.com|imgbox.com|mywebtv.info|sendvid.com|thevideo.me|uptobox.com|uptostream.com|yts.ag
|https:$popup,third-party,domain=fmovies.is
|https:$subdocument,third-party,domain=2ad.in|adf.ly|adfoc.us|adjet.biz|adv.li|ay.gy|j.gs|linkbucksmedia.com|q.gs|sh.st|thevideo.me|tvbximak.com|u.bb
|https:$third-party,xmlhttprequest,domain=123movies-proxy.ru|123movies.cz|123movies.gs|123movies.is|123movies.live|123movies.net|123movies.net.ru|123movies.ru|123movies.vc|123movieshd.net|123movieshd.tv|dropapk.com|estream.to|hdmusic99.in|otorrents.com|solarmovie.sc|wholecloud.net
|javascript:*settimeout*location.href$popup
|javascript^$popup,domain=1337x.to|biology-online.org|bitvid.sx|cloudtime.to|eztv.ag|eztv.tf|eztv.yt|flashx.tv|gorillavid.in|letwatch.us|movpod.in|nowvideo.sx|nowvideo.to|wholecloud.net
▒眠症改善.com
▒眠症改善.jpn.com
▒氧水.com
▒イフラボ.com
▒・ダーマラボ99.net
▒ethèrwallét.com
ètherwället.com
ƒluwdu.cn
дольщикиспб.рф
лоу-пак.com
шляхтен.рф
ラジオショッピング.jp
head -n20 /etc/pihole/gravity.list
!akamai.smartadserver.com
!cdn1.smartadserver.com
!diff.smartadserver.com
!diff1.smartadserver.com
!diff2.smartadserver.com
!diff3.smartadserver.com
!diff4.smartadserver.com
!diff5.smartadserver.com
!eqx.smartadserver.com
!im2.smartadserver.com
!itx5-publicidad.smartadserver.com
!itx5.smartadserver.com
!mobile.smartadserver.com
!rtb-csync.smartadserver.com
!smartadserver.com
!tcy.smartadserver.com
!ww1003.smartadserver.com
!ww1270.smartadserver.com
!ww129.smartadserver.com
sudo grep https: /etc/pihole/gravity.list | tail -n30
a[onmousedown^="this.href='https:
|https:$domain=adfarm1.adition.com
|https:$popup,third-party,domain=eztv.ag|flashx.tv|imagerar.com|imgbox.com|mywebtv.info|sendvid.com|thevideo.me|uptobox.com|uptostream.com|yts.ag
|https:$popup,third-party,domain=fmovies.is
|https:$subdocument,third-party,domain=2ad.in|adf.ly|adfoc.us|adjet.biz|adv.li|ay.gy|j.gs|linkbucksmedia.com|q.gs|sh.st|thevideo.me|tvbximak.com|u.bb
|https:$third-party,xmlhttprequest,domain=123movies-proxy.ru|123movies.cz|123movies.gs|123movies.is|123movies.live|123movies.net|123movies.net.ru|123movies.ru|123movies.vc|123movieshd.net|123movieshd.tv|dropapk.com|estream.to|hdmusic99.in|otorrents.com|solarmovie.sc|wholecloud.net
None of those are valid domains, nor valid entries that will work with Pi-hole.
The link was to the block list that contained the invalid domain .docs.google.com that is causing your problem. This is one of many lists you are using that have improper domains, and this list should be removed from your blocklist.
The head and tail of your gravity list shows that your gravity list contains improperly formatted domains that won't work with Pi-Hole. We only looked at the very ends of the gravity list, since that's where odd characters show up. The search for "https" in the gravity list shows further evidence - the text "https:" should not appear in a hosts format list.
The solution for you - completely remove all your block lists. Put back the seven original blocklists (link below), then selectively add properly formatted block lists one at a time. There is no value in randomly grabbing junk lists from the internet - this is preventing your Pi-Hole from working properly as evidenced by your post.
https://discourse.pi-hole.net/t/how-can-i-restore-pi-holes-default-ad-lists/4683/3?u=ramset
Properly formatted lists are found here:
Great! I agree dumping lists that the system does not parse (understand) is making the system fragile instead of strong.
is there a single command to delete all the lists? Or do I have to delete them manually through the web administration portal?