False blocked DNS-Names

MaDa · January 23, 2020, 2:31pm

Problem with Beta 5.0:
some DNS names are blocked without being in a blocklist or blacklisted

www.spiegel.de
api.onedrive.com
ocsp.digicert.com

Whitelisting directly from query log was also not possible, only manually

jfb · January 23, 2020, 3:05pm

Edited after posting to correct spelling of the domain names.

What is the output from the Pi command line of the following:

pihole -q -adlist spiegel.de

dig +short www.spiegel.de

dig +short spiegel.de

pihole -q -adlist link11.de

MaDa · January 23, 2020, 3:18pm

# pihole -q -adlist spiegel.de
 Match found in exact whitelist
   spiegel.de
   www.spiegel.de
 Match found in https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts:
   c.spiegel.de
   count.spiegel.de
   spiegel-de.spiegel.de
 Match found in http://sysctl.org/cameleon/hosts:
   c.spiegel.de
   count.spiegel.de
 Match found in https://hosts-file.net/ad_servers.txt:
   c.spiegel.de
 Match found in https://v.firebog.net/hosts/Kowabit.txt:
   c.spiegel.de
   count.spiegel.de
 Match found in http://v.firebog.net/hosts/Easyprivacy.txt:
   count.spiegel.de

# pihole -q -adlist link11.de
 Match found in exact blacklist
   aacfb9d106f4.link11.de

remember www.spiegel.de still whitelisted...

# dig +short www.spiegel.de
aacfb9d106f4.link11.de.
128.65.210.185
128.65.210.181
128.65.210.184
128.65.210.182
128.65.210.183
128.65.210.180

# dig +short spiegel.de
128.65.210.8

jfb · January 23, 2020, 3:29pm

With the new CNAME blocking in beta 5.0, blocked domains are checked all along the CNAME path. In your case, the CNAME is blocked, so the request is blocked.

dig www.spiegel.de

;; ANSWER SECTION:

www.spiegel.de. 3579 IN CNAME aacfb9d106f4.link11.de.
aacfb9d106f4.link11.de. 3579 IN A 128.65.210.180
aacfb9d106f4.link11.de. 3579 IN A 128.65.210.185
aacfb9d106f4.link11.de. 3579 IN A 128.65.210.183
aacfb9d106f4.link11.de. 3579 IN A 128.65.210.184
aacfb9d106f4.link11.de. 3579 IN A 128.65.210.182
aacfb9d106f4.link11.de. 3579 IN A 128.65.210.181

MaDa · January 23, 2020, 3:35pm

okay... and what is the conclusion?

DL6ER · January 23, 2020, 6:09pm

It means that one of your block lists contains a domain that appeared in the chain of queries leading to www.spiegel.de:

Exactly whitelisting from the Query Log should have helped, can you reproduce that it does not add the domain to the whitelist? Otherwise, it might have just been your system/browser that cached the negative response for some time leading to a false positive.

MaDa · January 23, 2020, 10:31pm

Thank you very much for the explanation. I would prefer to receive information about which of the FQDN generates the hit in a black- or blocklist. In this case 'www.spiege.de' and additionally deep 'aacfb9d106f4.link11.de' should be signaled.

Additionally... my dig command don't produce not the same output. I have to use
dig www.spiegel.de CNAME

MaDa · January 23, 2020, 10:46pm

mysterious manner the received CNAME odc-routekey-meta-geo.onedrive.akadns.net could not found in one of the blocklists

# pihole -q -adlist odc-routekey-meta-geo.onedrive.akadns.net
  [i] No results found for odc-routekey-meta-geo.onedrive.akadns.net within the blocklists

jfb · January 23, 2020, 10:59pm

What is the OS on which you are running this command? Here is what I see on Raspbian Buster:

dig www.spiegel.de CNAME @127.0.0.1

; <<>> DiG 9.11.5-P4-5.1-Raspbian <<>> www.spiegel.de CNAME @127.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5791
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;www.spiegel.de. IN CNAME

;; ANSWER SECTION:
www.spiegel.de. 2826 IN CNAME aacfb9d106f4.link11.de.
;; Query time: 897 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Jan 23 16:56:38 CST 2020
;; MSG SIZE rcvd: 77

dig www.spiegel.de @127.0.0.1

; <<>> DiG 9.11.5-P4-5.1-Raspbian <<>> www.spiegel.de @127.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46695
;; flags: qr rd ra; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;www.spiegel.de. IN A

;; ANSWER SECTION:
www.spiegel.de. 2822 IN CNAME aacfb9d106f4.link11.de.
aacfb9d106f4.link11.de. 2822 IN A 128.65.210.181
aacfb9d106f4.link11.de. 2822 IN A 128.65.210.182
aacfb9d106f4.link11.de. 2822 IN A 128.65.210.184
aacfb9d106f4.link11.de. 2822 IN A 128.65.210.185
aacfb9d106f4.link11.de. 2822 IN A 128.65.210.180
aacfb9d106f4.link11.de. 2822 IN A 128.65.210.183

;; Query time: 7 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Jan 23 16:56:42 CST 2020
;; MSG SIZE  rcvd: 173

MaDa · January 23, 2020, 11:04pm

Linux Mint (Ubuntu)

# dig www.spiegel.de @127.0.0.1

; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> www.spiegel.de @127.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16980
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;www.spiegel.de.			IN	A

;; Query time: 80 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Jan 24 00:04:13 CET 2020
;; MSG SIZE  rcvd: 43

Centos 7

dig www.spiegel.de @127.0.0.1

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> www.spiegel.de @127.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50126
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;www.spiegel.de.			IN	A

;; Query time: 4 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Jan 24 00:05:35 CET 2020
;; MSG SIZE  rcvd: 43

MacBook-Pro  ~  dig www.spiegel.de                          ✔  2623  00:06:47

; <<>> DiG 9.10.6 <<>> www.spiegel.de
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58689
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;www.spiegel.de.			IN	A

;; Query time: 71 msec
;; SERVER: 192.168.3.5#53(192.168.3.5)
;; WHEN: Fri Jan 24 00:06:57 CET 2020
;; MSG SIZE  rcvd: 43

jfb · January 23, 2020, 11:08pm

That is not the only CNAME in the path:

dig api.onedrive.com @127.0.0.1

;; ANSWER SECTION:

api.onedrive.com. 3599 IN CNAME odc-routekey-meta-geo.onedrive.akadns.net.

odc-routekey-meta-geo.onedrive.akadns.net. 3600 IN CNAME odc-routekey-meta-brs.onedrive.akadns.net.

odc-routekey-meta-brs.onedrive.akadns.net. 3600 IN CNAME odc-common-us-meta.onedrive.akadns.net.l-0003.dc-msedge.net.l-0003.l-msedge.net.

odc-common-us-meta.onedrive.akadns.net.l-0003.dc-msedge.net.l-0003.l-msedge.net. 3600 IN CNAME l-0003.l-msedge.net.

l-0003.l-msedge.net. 3600 IN A 13.107.42.12

MaDa · January 23, 2020, 11:09pm

I see, my mistake. Sorry

MaDa · January 23, 2020, 11:10pm

Do you have an idea how pihole could support the deeper analysis of such CNAME blockings? Do you think it is realistic request for?

jfb · January 23, 2020, 11:11pm

I think the FTL developer is already working on this. Stay tuned.

MaDa · January 23, 2020, 11:13pm

and is this behavior reproducible?

jfb · January 23, 2020, 11:17pm

I am able to whitelist directly from the query log in V 5.0 beta. I picked the first blocked domain on my query log, whitelisted it. It showed up on the whitelist and in a search of the domain in black/white/blocklists:

pihole -q -adlist vendorlist.consensu.org
 Match found in exact whitelist
   vendorlist.consensu.org
 Match found in https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts:
   vendorlist.consensu.org

Was the problem you were having:

The domain would not show up in your whitelist at all, or
After you whitelisted it, it was still blocked for a time?

MaDa · January 23, 2020, 11:19pm

Yes, this was the behavior.

But I tested it now again and it works ... strange.

DL6ER · January 24, 2020, 4:17pm

Maybe this is a ping

Storing this extra information, the domain that was the reason for blocking the entire query, is not really straightforward. Reserving extra space in the queries structure inside FTL would increase the memory requirements although not by much when we do it cleverly (hint: we would).

We already have a change that just needs to get reviewed here:

github.com/pi-hole/web

Add extra label when blocking happend during deep CNAME inspection

pi-hole:release/v5.0 ← pi-hole:new/query_log_show_CNAME_blocked

opened 10:37PM - 21 Jan 20 UTC

DL6ER

+5 -0

**By submitting this pull request, I confirm the following:** - [X] I have r…ead and understood the [contributors guide](https://github.com/pi-hole/pi-hole/blob/master/CONTRIBUTING.md), as well as this entire template. - [X] I have made only one major change in my proposed changes. - [X] I have commented my proposed changes within the code. - [X] I have tested my proposed changes, and have included unit tests where possible. - [X] I am willing to help maintain this change if there are issues with it later. - [X] I give this submission freely and claim no ownership. - [X] It is compatible with the [EUPL 1.2 license](https://opensource.org/licenses/EUPL-1.1) - [X] I have squashed any insignificant commits. ([`git rebase`](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)) --- **What does this PR aim to accomplish?:** Illustrate when blocking happened during a (possibly deep) CNAME inspection: ![Screenshot from 2020-01-21 23-34-16](https://user-images.githubusercontent.com/16748619/72849404-df7d7400-3ca6-11ea-8b2b-d3ce48942c43.png) Whitelisting makes sense even if the inspection was deep as explicitly whitelisted domains along the path mark the entire path as permitted. **How does this PR accomplish the above?:** Add extra label when blocking happend during deep CNAME inspection **What documentation changes (if any) are needed to support this PR?:** None

Even though this does not show the exact domain that was blocked along the path, it will signal that there was something CNAME-related going on. And it is not really necessary to know which domain exactly was the reason as whitelisting on a high level works perfectly fine. A query to a whitelisted domain will always by permitted, even if it contains a domain that would normally get blocked.

FTL employs the rule "the whitelist always wins" everywhere.

DL6ER · January 29, 2020, 10:56pm

Over the days, I got convinced that knowing the domain that was the actual blocking reason is meaningful. See, e.g.,

MaDa · January 30, 2020, 7:50am

Over the days, I got convinced that knowing the domain that was the actual blocking reason is meaningful. See, e.g.,

Thank you @DL6ER