Adding to whitelist, still not allowed until whitelist button clicked

Ryan_Ellis · April 16, 2020, 1:04pm

Please follow the below template, it will help us to help you!

Expected Behaviour:

Add a site to a whitelist in UI, see site listed in whitelist, site should not be blocked due to blacklists / gravity.

Actual Behaviour:

ad.doubleclick.net is set in whitelist, and was blocked until I clicked the whitelist button in the query log ... then whitelist was working properly and domain was able to resolve. This happened in the previous version too. I have to click that whitelist button in the query log for domains to actually whitelist.

Debug Token:

5f130poo2c

This isn't a new setup, been running over a year. But not being able to access google shopping results has run its course.

jfb · April 16, 2020, 1:34pm

Let's take a closer look at this. I see ad.doubleclick.net in your whitelist. Is this domain currently whitelisted? What is the output of dig +short ad.doubleclick.net

Now let's take a random domain that is in your blacklist.

dig yuxyz.com

Now, in the admin UI, add that domain to the whitelist. Wait at least 2 seconds for the TTL to expire.

dig yuxyz.com

Ryan_Ellis · April 16, 2020, 1:50pm

ad.doubleclick.net is now whitelisted because I clicked the "Whitelist" button in the query log, It already existed in the whitelist prior to the UI button click. The only way things whitelist successfully in the UI is clicking the whitelist button in the query log.

All that being said, I ran through your test:

Adding yuxyz.com to the whitelist did make the domain resolve.

Aka I'm not able to reproduce right now so I can't help. Maybe the latest version fixed. Guess I'll be back.

But seems very similar to this previous issue:
https://discourse.pi-hole.net/t/pihole-whitelist-not-working-properly/7790/18

jfb · April 16, 2020, 2:42pm

Note that when you whitelist an ad-serving domain like you have, you whitelist that for all sites that use that domain. You will be allowing more than just Google shopping results.

Also note that Google shopping results have the sponsored posts at the top, and those are typically served through ad-serving domains. If you scroll down a bit in the results, the same information will be available directly from non ad-serving domains.

In the upcoming Pi-hole V5, per-client blocking is a feature. You can whitelist that domain for a subset of clients only. This may help in your situation.

Lastly, it is unusual to see so many domains in a blacklist. Did you add an entire blocklist here?

Ryan_Ellis · April 16, 2020, 5:29pm

Yep I'm aware that the pi hole just zeros out the ip for dns requests of FQDNs on the blocklist.

Yes, the shopping results are paid results.

Paid ads are an auction for only the ones that are bidding on the search term (paying google).

Bidding on keywords doesn't get you into organic results.

The organic results below are absolutely not the same as paid results. The organic results are a completely different algorithm (and you can't bid on results).

I'm a marketer so it's important to be clear that paid vs free results are not the same results, and the best results for buying-intent keywords are going to be at the top.

As an advertiser, you don't continually bid on keywords unless they convert (unless you like wasting ad spend). That means people are buying from "shopping results/ads" when searching these queries, which means the results are relevant... and advertisers continue to spend.

But ... I also have a heavy IT background ... so in terms of blacklist ... blocklists are added and pulled weekly (cron) from a ton of sources, de-duped, and added to /etc/pihole/blacklist.txt

(I realize there are some duplicated efforts in terms of updating lists, but I'm adding new blacklists as they publish - via cron)

jfb · April 16, 2020, 5:49pm

Why do you add these to your blacklist file and not let Pi-hole import these lists with the existing cron and pihole-g routines and store these in the existing gravity list?

Adding the domains from the "list" produces a lot more unique domains than you have in your blacklist and gravity lists combined. Your blacklist contains 46,439 entries, gravity has 91,609. The "list" produces 342,648 unique domains when processed by a gravity update, quite a larger number.

Some of the lists in the "list" are not valid for Pi-hole as shown below:

  [✓] Preparing new gravity database
  [i] Target: https://easylist-downloads.adblockplus.org/adwarefilters.txt
  [✓] Status: Retrieval successful
  [i] Received 124 domains, 124 domains invalid!
      Sample of invalid domains:
      - 2.0]
      - ||74055djs.info^
      - ||aceadsys.net^
      - ||addon.downloadterms.com^
      - ||admdftjs.info^$third-party

  [i] Target: https://easylist-downloads.adblockplus.org/fanboy-annoyance.txt
  [✓] Status: Retrieval successful
  [i] Received 919 domains, 887 domains invalid!
      Sample of invalid domains:
      - 2.0]
      - easylist.subscription@gmail.com
      - -facebook.js$domain=~king.com
      - -share-button.$~xmlhttprequest
      - -share-buttons.$~xmlhttprequest

  [i] Target: https://easylist-downloads.adblockplus.org/fanboy-social.txt
  [✓] Status: Retrieval successful
  [i] Received 235 domains, 219 domains invalid!
      Sample of invalid domains:
      - 2.0]
      - easylist.subscription@gmail.com
      - -facebook.js$domain=~king.com
      - -share-button.$~xmlhttprequest
      - -share-buttons.$~xmlhttprequest

  [i] Target: http://www.kiboke-studio.hr/i-dont-care-about-cookies/abp/
  [✓] Status: Retrieval successful
  [i] Received 87 domains, 19 domains invalid!
      Sample of invalid domains:
      - 2.0]
      - cookieconsent.js$domain=~blackboard.com|~kayak.pl
      - quantcast.mgr.consensu.org$domain=~sourceforge.net|~vi.nl|~joe.ie|~cnews.fr
      - consentmanager.mgr.consensu.org$domain=~sourceforge.net
      - policy.app.cookieinformation.com$domain=~kartor.eniro.se|~kart.gulesider.no|~kort.degulesider.dk|~map.krak.dk

  [i] Target: https://easylist-downloads.adblockplus.org/malwaredomains_full.txt
  [✓] Status: Retrieval successful
  [i] Received 26858 domains, 26858 domains invalid!
      Sample of invalid domains:
      - 1.1]
      - ||amazon.co.uk.security-check.ga^
      - ||autosegurancabrasil.com^
      - ||dadossolicitado-antendimento.sad879.mobi^
      - ||hitnrun.com.my^

  [i] Target: https://raw.github.com/liamja/Prebake/master/obtrusive.txt
  [✓] Status: Retrieval successful
  [i] Received 13 domains, 9 domains invalid!
      Sample of invalid domains:
      - 1.1]
      - (cookies@prebake.eu).
      - ||consent.truste.com
      - ||cookieconsent.com^$third-party
      - eu_cookie_compliance.js*$script

  [i] Target: https://raw.githubusercontent.com/Dawsey21/Lists/master/adblock-list.txt
  [✓] Status: Retrieval successful
  [i] Received 6371 domains, 6370 domains invalid!
      Sample of invalid domains:
      - 2.0]
      - admin@spam404.com
      - ||telechargerdes.com^
      - ||rarshare.com^
      - ||planetside2-hacks.com^

Ryan_Ellis · April 25, 2020, 7:33pm

ad.doubleclick.net became blocked again.

https://share.getcloudapp.com/kpuLbe4B

had to click the whitelist button in the Query log again to whitelist.

The only issue at hand is:

Domains added to whitelist aren't honored via GUI.

Unless the invalid domains, lists, cron updating pihole-g, etc are contributing to the pi hole failing to operate on its most foundational principle of operating off a whitelist/blacklist .... any other those other "issues" are just adding complexity any distracting from the root issue.

Thanks!

jfb · April 25, 2020, 7:35pm

Let's take a look at a current debug log. Your old one expired last week.

pihole -d

or do it through the Web interface:

Tools > Generate Debug Log

system · May 16, 2020, 7:47pm

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.