Some domains are intermittently blocked even though they are on the whitelist

General Beta 5.0
1

Problem with Beta 5.0:
I've seen cases where even though a domain is explicitly on my whitelist, I see in the query log and on my devices when they try to access said domain that it is blocked due to deep CNAME inspection. The example I'll give is of "steamcdn-a.akamaihd.net". This domain was under a exact whitelist entry for the past 24 hours (i.e. from my understanding, for all the times in the screenshot, it should have been allowed, despite linking to a blocked CNAME). This doesn't even happen 100% of the time - sometimes the domain will be allowed and then a little while later it will be blocked again, as shown in the screenshot.

If I run "pihole restartdns" the domains seem to be correctly allowed for a little while.

05%20AM
05%20AM2830×1232 311 KB

Log snippet from allowed:
/var/log/pihole.log

11953 Jan 26 07:57:46 dnsmasq[765]: query[A] steamcdn-a.akamaihd.net from 192.168.1.10
11954 Jan 26 07:57:46 dnsmasq[765]: forwarded steamcdn-a.akamaihd.net to 127.0.0.1
11955 Jan 26 07:57:46 dnsmasq[765]: reply steamcdn-a.akamaihd.net is <CNAME>
11956 Jan 26 07:57:46 dnsmasq[765]: reply steamcdn-a.akamaihd.net.edgesuite.net is <CNAME>
11957 Jan 26 07:57:46 dnsmasq[765]: reply a1843.g1.akamai.net is 23.200.236.194
11958 Jan 26 07:57:46 dnsmasq[765]: reply a1843.g1.akamai.net is 23.200.236.201

Log snippet from (erroneously) blocked:
/var/log/pihole.log

13897 Jan 26 09:00:30 dnsmasq[765]: query[A] steamcdn-a.akamaihd.net from 192.168.1.10
13898 Jan 26 09:00:30 dnsmasq[765]: forwarded steamcdn-a.akamaihd.net to 127.0.0.1
13899 Jan 26 09:00:30 dnsmasq[765]: reply steamcdn-a.akamaihd.net is <CNAME>
13900 Jan 26 09:00:30 dnsmasq[765]: reply steamcdn-a.akamaihd.net.edgesuite.net is <CNAME>

/var/log/pihole-FTL.log (I think this part may be relevant?)

222 [2020-01-26 09:00:08.095 765] Regex blacklist (ID 98) ".*\.g[0-9]+\..*" matches "a1843.g1.akamai.net"

Debug Token:
https://tricorder.pi-hole.net/rhwdkd8wzn

2

I see that too sometimes.

3

What are the outputs from the Pi terminal of the following commands:

dig steamcdn-a.akamaihd.net

dig steamcdn-a.akamaihd.net.edgesuite.net

dig a1843.g1.akamai.net

pihole -q -adlist steamcdn Edited to remove the "-"

pihole -q -adlist akamai.net

4

At this time, it is currently working (not blocked), but:

$ dig steamcdn-a.akamaihd.net

; <<>> DiG 9.11.5-P4-5.1-Debian <<>> steamcdn-a.akamaihd.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58062
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;steamcdn-a.akamaihd.net.	IN	A

;; ANSWER SECTION:
steamcdn-a.akamaihd.net. 0	IN	CNAME	steamcdn-a.akamaihd.net.edgesuite.net.
steamcdn-a.akamaihd.net.edgesuite.net. 0 IN CNAME a1843.g1.akamai.net.
a1843.g1.akamai.net.	0	IN	A	184.28.82.144
a1843.g1.akamai.net.	0	IN	A	184.28.82.192

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Jan 26 17:00:19 EST 2020
;; MSG SIZE  rcvd: 162

$ dig steamcdn-a.akamaihd.net.edgesuite.net

; <<>> DiG 9.11.5-P4-5.1-Debian <<>> steamcdn-a.akamaihd.net.edgesuite.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25963
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;steamcdn-a.akamaihd.net.edgesuite.net. IN A

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Jan 26 17:01:06 EST 2020
;; MSG SIZE  rcvd: 66

$ dig a1843.g1.akamai.net

; <<>> DiG 9.11.5-P4-5.1-Debian <<>> a1843.g1.akamai.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58333
;; flags: qr aa rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;a1843.g1.akamai.net.		IN	A

;; ANSWER SECTION:
a1843.g1.akamai.net.	2	IN	A	0.0.0.0

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Jan 26 17:02:07 EST 2020
;; MSG SIZE  rcvd: 53

$ pihole -q -adlist steamcdn
 Match found in exact whitelist
   steamcdn-a.akamaihd.net

$ pihole -q -adlist akamai.net
 Match found in exact whitelist
   a1697.g1.akamai.net
   a1737.g1.akamai.net
  [i] Over 100 results found for akamai.net
        This can be overridden using the -all option

Output with the -all option is at https://hastebin.com/raw/polebusaze

I'll post again with the results of the 3 dig commands if I notice the domain is blocked again.

EDIT: Not even a minute later I opened Steam again and it was blocked:

https://hastebin.com/raw/qoduweyeno

5

There was a update in pihole -up today

I dont see any false block any more, not sure if it got fixed.

6

I am up to date.

$ pihole -up
  [i] Checking for updates...
  [i] Pi-hole Core:	up to date
  [i] Web Interface:	up to date
  [i] FTL:		up to date
  [i] Warning: You are using FTL from a custom branch (release/v5.0) and might be missing future releases.

  [✓] Everything is up to date!

EDIT: I was thinking, and perhaps a possible cause was me running the "dig" command on the intermediate CNAME steamcdn-a.akamaihd.net.edgesuite.net causing something to go in the cache and making Pi-hole think steamcdn-a.akamaihd.net should be blocked?

7

I tried to reproduce this locally but haven't been able to get the wrong/strange behavior.

Please add

DEBUG_QUERIES=true
DEBUG_FLAGS=true

to your /etc/pihole/pihole-FTL.conf and run pihole restartdns. This will put FTL into (very verbose!) debugging mode. Please provide snippets from the file /var/log/pihole-FTL.log when it works and when it doesn't

1 Like
8

Working correctly

[2020-01-27 15:35:27.050 29386] **** new UDP query[A] "steamcdn-a.akamaihd.net" from 127.0.0.1 (ID 104, FTL 21187, src/dnsmasq/forward.c:1571)
[2020-01-27 15:35:27.050 29386] steamcdn-a.akamaihd.net is not known
[2020-01-27 15:35:27.050 29386] **** forwarded steamcdn-a.akamaihd.net to 127.0.0.1 (ID 104, src/dnsmasq/forward.c:558)
[2020-01-27 15:35:27.050 29386] steamcdn-a.akamaihd.net is known as not to be blocked (whitelisted)
[2020-01-27 15:35:27.050 29386] CNAME steamcdn-a.akamaihd.net
[2020-01-27 15:35:27.050 29386] **** got reply steamcdn-a.akamaihd.net is (CNAME) (ID 104, src/dnsmasq/cache.c:487)
[2020-01-27 15:35:27.050 29386]      Flags: F_FORWARD F_CNAME 
[2020-01-27 15:35:27.050 29386] steamcdn-a.akamaihd.net.edgesuite.net is not known
[2020-01-27 15:35:27.050 29386] Query is permitted as at least one whitelist entry matched
[2020-01-27 15:35:27.050 29386] CNAME steamcdn-a.akamaihd.net ---> steamcdn-a.akamaihd.net.edgesuite.net
[2020-01-27 15:35:27.051 29386] **** got reply steamcdn-a.akamaihd.net.edgesuite.net is (CNAME) (ID 104, src/dnsmasq/cache.c:487)
[2020-01-27 15:35:27.051 29386]      Flags: F_FORWARD F_CNAME 
[2020-01-27 15:35:27.051 29386] a1843.g1.akamai.net is not known
[2020-01-27 15:35:27.051 29386] Query is permitted as at least one whitelist entry matched
[2020-01-27 15:35:27.051 29386] CNAME steamcdn-a.akamaihd.net.edgesuite.net ---> a1843.g1.akamai.net
[2020-01-27 15:35:27.051 29386] **** got reply a1843.g1.akamai.net is 23.200.236.211 (ID 104, src/dnsmasq/cache.c:487)

Working incorrectly

[2020-01-27 15:37:15.271 29386] **** new UDP query[A] "steamcdn-a.akamaihd.net" from 127.0.0.1 (ID 230, FTL 21209, src/dnsmasq/forward.c:1571)
[2020-01-27 15:37:15.271 29386] steamcdn-a.akamaihd.net is known as not to be blocked (whitelisted)
[2020-01-27 15:37:15.271 29386] **** forwarded steamcdn-a.akamaihd.net to 127.0.0.1 (ID 230, src/dnsmasq/forward.c:558)
[2020-01-27 15:37:15.271 29386] steamcdn-a.akamaihd.net is known as not to be blocked (whitelisted)
[2020-01-27 15:37:15.271 29386] CNAME steamcdn-a.akamaihd.net
[2020-01-27 15:37:15.271 29386] **** got reply steamcdn-a.akamaihd.net is (CNAME) (ID 230, src/dnsmasq/cache.c:487)
[2020-01-27 15:37:15.272 29386]      Flags: F_FORWARD F_CNAME 
[2020-01-27 15:37:15.272 29386] steamcdn-a.akamaihd.net.edgesuite.net is known as not to be blocked
[2020-01-27 15:37:15.272 29386] CNAME steamcdn-a.akamaihd.net ---> steamcdn-a.akamaihd.net.edgesuite.net
[2020-01-27 15:37:15.272 29386] **** got reply steamcdn-a.akamaihd.net.edgesuite.net is (CNAME) (ID 230, src/dnsmasq/cache.c:487)
[2020-01-27 15:37:15.272 29386]      Flags: F_FORWARD F_CNAME 
[2020-01-27 15:37:15.272 29386] a1843.g1.akamai.net is known as regex blacklisted
[2020-01-27 15:37:15.272 29386] CNAME steamcdn-a.akamaihd.net.edgesuite.net ---> a1843.g1.akamai.net

I've found that after running the dig command on steamcdn-a.akamaihd.net.edgesuite.net (not explicitly whitelisted, so blocked because a1843.g1.akamai.net is blocked through regex), then steamcdn-a.akamaihd.net will be blocked 100% of the time (presumably until I restart the resolver).

9

Oh, this is a false-positive cache hit...I thought I had ruled out such scenarios, but apparently I haven't thought about all possible cases. A partially cached CNAME inspection stored as partial path is apparently able to slip the explicit whitelisting rules.

I will prepare a fix, many thanks for reporting and providing the necessary details!

1 Like
10

@anon13884536 Please try

pihole checkout ftl fix/deeply_whitelisted_domain

and see if this fixes the issue you're seeing here. I fixed the twisted logic that (presumably) lead to this strange cache malfunctions.

If my fix doesn't help, I will set up a system for reproducing your issue locally, tomorrow.

11

Your fix seems to work beautifully, many thanks for the quick work!

1 Like
13

Well, with the exception of one oddity, related to the web interface:

If I ran dig on steamcdn-a.akamaihd.net.edgesuite.net beforehand, then dig steamcdn-a.akamaihd.net, the query log in the admin panel incorrectly shows that the status for steamcdn-a.akamaihd.net was blocked, even though it was allowed.

55%20PM
55%20PM2210×72 7.3 KB

(successful dig)

; <<>> DiG 9.11.5-P4-5.1-Debian <<>> steamcdn-a.akamaihd.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58731
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;steamcdn-a.akamaihd.net.	IN	A

;; ANSWER SECTION:
steamcdn-a.akamaihd.net. 122	IN	CNAME	steamcdn-a.akamaihd.net.edgesuite.net.
steamcdn-a.akamaihd.net.edgesuite.net. 20278 IN	CNAME a1843.g1.akamai.net.
a1843.g1.akamai.net.	15	IN	A	23.200.236.208
a1843.g1.akamai.net.	15	IN	A	23.200.236.194
a1843.g1.akamai.net.	15	IN	A	23.200.236.201
a1843.g1.akamai.net.	15	IN	A	23.200.236.200

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Jan 27 17:09:47 EST 2020
;; MSG SIZE  rcvd: 194
14

Ah, yes, thanks for catching this! This should now be addresses as well. The binary will be available in a few minutes if you want to try again. is ready for another checkout.

1 Like
15

Looks good now, thank you!

1 Like
16

Thanks for your confirmation. The fix has been merged, please go back onto the main track to stay updated with possible new changes

pihole checkout ftl release/v5.0
1 Like