Hi -
I realize that this is a common question but i can't seem to figure out why this domain is being blocked.
OCSP.Apple.Com is on the whitelist, but is still being blocked
matt@matt-ThinkPad-T61:~$ pihole -q ocsp.apple.com
Match found in Whitelist
ocsp.apple.com
Match found in list.1.reddestdream.github.io.domains:
ocsp.apple.com
After you added the ocsp.apple.com domain to your whitelist, did you rebuild gravity? I would check this - look in the whitelist and make sure there are no trailing spaces after the oscp.apple.com. Then, if there are, fix that, save and exit that file.
Rebuild gravity with pihole -g --skip-download. This should remove that domain from the gravity list. Check afterwards to see if the domain still exists in gravity.
One unrelated note from your debug log: you are running unbound as local resolver, but have DNSSEC enabled in Pi-Hole. The developers recommend that since unbound is handling DNSSEC, that DNSSEC should not be enabled in Pi-Hole. Turn it off in Pi-Hole and you will still have the DNSSEC in unbound and fewer potential problems with the Pi-Hole display of DNSSEC data from unbound.
I checked the whitelist, but there doesn't appear to be any extra characters there.
I actually whitelisted it about a week ago, and have rebuild gravity a number of times since then, but I did it again just now. Still same result.
[BTW, i have lots of other domains that are whitelisted that work fine, for some reason it seems to just be this one apple domain that doesn't work.]
matt@matt-ThinkPad-T61:/etc/pihole$ pihole -g --skip-download
[i] Pi-hole blocking is enabled
[i] Using cached Event Horizon list...
[i] 2914674 unique domains trapped in the Event Horizon
[i] Number of blacklisted domains: 13
[i] Number of regex filters: 122
[✓] Force-reloading DNS service
[✓] DNS service is running
[✓] Pi-hole blocking is Enabled
When you look at the whitelist display on the Web Admin GUI, do you see the oscp.apple.com domain? If so, try this. Remove the domain and save. Then add the domain back in and save.
I suspect it was a line break or something of that nature. Your grep showed the words but not how they fit into a line. I assume you deleted the domain, then re-entered it? That would fix any line break issues. Hard to tell without having the original whitelist.txt file in hand.
@anon55913113 - interesting that you have jumped in here - I actually am using your postclean scripts to reduce my gravity.list and instead create regex entries.
I have been tinkering with this a little bit and I think that the whitelist functions as intended when the DNS block is created by an item in the regex.list. However, if the DNS block is actually only from the gravity.list (as was ocsp.apple.com in setup) then the whitelist doesn't allow access. My success in getting access to this specific site was shortlived, because as soon as I downloaded blocklists and ran postclean, the site was re-added to gravity and I was blocked again. Ironically, the fix I had to make was to add this site to as a single regex entry, and then it works fine.
I think that if the whitelist.txt entries were removed from gravity as a final postclean.sh processing step it would eliminate this issue. I was staring at your code trying to figure out if it could do it myself, but I'd be happy if you were able to implement it correctly.
Is there one place where post updates to the code? I had to hunt around some various threads previously to see what improvements you had made, and it'd be nice if it were in (at least) one place.
Matt
@anon55913113 I don't believe this is the case. Pi-Hole checks in gravity (the master list of blocked domains) when determining domains to block (ignore regex for this example). The whitelist is a separate list that makes domains "gravity-proof". Each time gravity is rebuilt, the whitelisted entries are removed from gravity, if they exist in gravity.
Quick test I ran - cleared out all adlists, whitelist, blacklist, regex list. Rebuilt gravity to zero, cleared the unbound cache. I added a single blocklist from the web, with not a lot of entries and domains that were easy to spell.
pi@PiZero:/etc/pihole $ pihole -g
[i] Pi-hole blocking is enabled
[i] Neutrino emissions detected...
[✓] Pulling blocklist source list into range
[i] Target: blocklist.site (fakenews)
[✓] Status: No changes detected
[✓] Consolidating blocklists
[✓] Extracting domains from blocklists
[i] Number of domains being pulled in by gravity: 669
[✓] Removing duplicate domains
[i] Number of unique domains trapped in the Event Horizon: 669
[i] Number of whitelisted domains: 0
[i] Number of blacklisted domains: 0
[✓] Parsing domains into hosts format
[✓] Cleaning up stray matter
[✓] Force-reloading DNS service
[✓] DNS service is running
[✓] Pi-hole blocking is Enabled
pi@PiZero:/etc/pihole $ sudo service unbound restart
Found a domain in the list that was valid, tested with this domain. Verified that the domain was only in gravity and not in the whitelist
pi@PiZero:/etc/pihole $ pihole -q -adlist breitbart
Match found in https://blocklist.site/app/dl/fakenews:
www.breitbart.com
pi@PiZero:/etc/pihole $ sudo grep breitbart gravity.list
www.breitbart.com
pi@PiZero:/etc/pihole $ cat whitelist.txt
pi@PiZero:/etc/pihole $ dig +short www.breitbart.com
0.0.0.0
Now add the domain to the whitelist, don't rebuild gravity. Domain is in both the whitelist and gravity.
pi@PiZero:/etc/pihole $ sudo nano whitelist.txt
pi@PiZero:/etc/pihole $ cat whitelist.txt
www.breitbart.com
pi@PiZero:/etc/pihole $ pihole -q -adlist breitbart
Match found in Whitelist
www.breitbart.com
Match found in https://blocklist.site/app/dl/fakenews:
www.breitbart.com
pi@PiZero:/etc/pihole $ sudo grep breitbart gravity.list
www.breitbart.com
pi@PiZero:/etc/pihole $ dig +short www.breitbart.com
0.0.0.0
Rebuild gravity - this removes the whitelisted domain from the gravity list. Verify that the domain is not in gravity.
pi@PiZero:/etc/pihole $ pihole -g
[i] Pi-hole blocking is enabled
[i] Neutrino emissions detected...
[✓] Pulling blocklist source list into range
[i] Target: blocklist.site (fakenews)
[✓] Status: No changes detected
[✓] Consolidating blocklists
[✓] Extracting domains from blocklists
[i] Number of domains being pulled in by gravity: 669
[✓] Removing duplicate domains
[i] Number of unique domains trapped in the Event Horizon: 669
[i] Number of whitelisted domains: 1
[i] Number of blacklisted domains: 0
[✓] Parsing domains into hosts format
[✓] Cleaning up stray matter
[✓] Force-reloading DNS service
[✓] DNS service is running
[✓] Pi-hole blocking is Enabled
pi@PiZero:/etc/pihole $ pihole -q -adlist breitbart
Match found in Whitelist
www.breitbart.com
Match found in https://blocklist.site/app/dl/fakenews:
www.breitbart.com
pi@PiZero:/etc/pihole $ sudo grep breitbart gravity.list
pi@PiZero:/etc/pihole $ dig +short www.breitbart.com
35.241.35.213
This is the current code from the master branch of Pi-Hole. Regardless of any local changes you have made to your local code, or additional scripts that you run to pre and post process the gravity list, this is the code used by Pi-Hole as distributed.
Where are you getting the idea that the code I linked is "legacy" code?
The code shows how whitelist entries are removed from gravity.
If it were the case that the whitelist is referenced prior to gravity during a domain resolution, what would be the purpose of removing any whitelisted domains from gravity at any time? And why, when a domain is added to the whitelist through the GUI, is gravity rebuilt by list.sh?
Can we get back to helping the original poster with their issues? If you'd like to see anything improved or changed in the core Pi-hole code just send us a PR with your changes and we'll take a look.
No. Gravity is not being corrupted. What you see Pi-Hole doing is not being correctly interpreted.
Since the developers are not aware of any errors in the code and gravity is working the way it was intended, no PR exists. If you find that the code is not working as intended, then a PR is the path to resolution.