Cloudflare DoH: Netflix problems on SmartTV

Please follow the below template, it will help us to help you!

Expected Behaviour:

Netflix starts playing and pi-hole works

Actual Behaviour:

Netflix can`t start play and Pi-hole stops working and the internet is dropping off because of dns is not working. After I exit Netflix pi-hole starts to respond again and internet (DNS) starts to work again. Using FTLDNS and DoH. (Using Netflix from an Samsung SmartTV).

I also disabled Pi-Hole in 5min, but the pi-hole stops repsonding.

From the logs I can see this, still the status is under disabled for 5min:
Apr 7 06:24:25 dnsmasq[1219]: 52 10.0.0.67/34174 query[A] nrdp.nccp.netflix.com from 10.0.0.67
Apr 7 06:24:25 dnsmasq[1219]: 52 10.0.0.67/34174 forwarded nrdp.nccp.netflix.com to 127.0.0.1

There is no blockfilter to this domain (There is more *.netflix.com domains that is forwarded to localhost

Debug Token:

3dygujj398

So you play Netflix and no other device can get out to the Internet?

Yes after I starts Netflix on my SamsungTV Pihole stops responding, so after I quit Netflix then Pihole starts to respond. Netflix work fine on regular web-browser and mobile-app.

What does your query log show during this problem?

Why does it forward to 127.0.0.1?

edit: I found out what "DoH" means - would be nice to write out less common acronyms before using them next time :wink:

2 Likes

I'm seeing a similar issue - also running FTLDNS and it stops responding after there's a netflix query. I'm using cloudflared proxy dns running on port 8053 of the same machine and my server=192.168.50.20#8053 in a dmsmasq.d config file. I've also tried with server=127.0.0.1#54 which is a dnscrypt-proxy and it fails the same way. If I change my server to just 1.1.1.1 it doesn't have this problem.

I also did a test of using server=208.67.222.222#5353 to see if the issue is limited to servers with an alternate port specified and I didn't have a problem.

This is what it looks like running pihole-FTL debug when it stops responding:

dnsmasq: 13 192.168.30.30/51473 query[A] api-global.netflix.com from 192.168.30.30
[2018-04-07 13:16:43.130] **** new query query[A] api-global.netflix.com 192.168.30.30 (ID 13)
dnsmasq: 13 192.168.30.30/51473 forwarded api-global.netflix.com to 192.168.50.20
[2018-04-07 13:16:43.132] **** forwarded api-global.netflix.com to 192.168.50.20 (ID 13)
dnsmasq: 13 192.168.30.30/51473 reply api-global.netflix.com is <CNAME>
[2018-04-07 13:16:43.166] **** got reply api-global.netflix.com is (CNAME) (TTL 60, ID 13)
[2018-04-07 13:16:43.167]      Flags: F_FORWARD F_CNAME
dnsmasq: 13 192.168.30.30/51473 reply api-global.geo.netflix.com is <CNAME>
[2018-04-07 13:16:43.174] **** got reply api-global.geo.netflix.com is (CNAME) (TTL 60, ID 13)
[2018-04-07 13:16:43.175]      Flags: F_FORWARD F_CNAME
dnsmasq: 13 192.168.30.30/51473 reply api-global.latency.prodaa.netflix.com is 34.202.114.174
[2018-04-07 13:16:43.183] **** got reply api-global.latency.prodaa.netflix.com is 34.202.114.174 (TTL 60, ID 13)
[2018-04-07 13:16:43.184]      Flags: F_FORWARD F_IPV4
dnsmasq: 13 192.168.30.30/51473 reply api-global.latency.prodaa.netflix.com is 34.204.238.125
[2018-04-07 13:16:43.191] **** got reply api-global.latency.prodaa.netflix.com is 34.204.238.125 (TTL 60, ID 13)
[2018-04-07 13:16:43.192]      Flags: F_FORWARD F_IPV4
dnsmasq: 13 192.168.30.30/51473 reply api-global.latency.prodaa.netflix.com is 34.225.135.220
[2018-04-07 13:16:43.200] **** got reply api-global.latency.prodaa.netflix.com is 34.225.135.220 (TTL 60, ID 13)
[2018-04-07 13:16:43.201]      Flags: F_FORWARD F_IPV4
dnsmasq: 13 192.168.30.30/51473 reply api-global.latency.prodaa.netflix.com is 52.5.61.180
[2018-04-07 13:16:43.208] **** got reply api-global.latency.prodaa.netflix.com is 52.5.61.180 (TTL 60, ID 13)
[2018-04-07 13:16:43.209]      Flags: F_FORWARD F_IPV4
dnsmasq: 13 192.168.30.30/51473 reply api-global.latency.prodaa.netflix.com is 52.6.75.180
[2018-04-07 13:16:43.216] **** got reply api-global.latency.prodaa.netflix.com is 52.6.75.180 (TTL 60, ID 13)
[2018-04-07 13:16:43.218]      Flags: F_FORWARD F_IPV4
dnsmasq: 13 192.168.30.30/51473 reply api-global.latency.prodaa.netflix.com is 52.73.224.7
[2018-04-07 13:16:43.223] **** got reply api-global.latency.prodaa.netflix.com is 52.73.224.7 (TTL 60, ID 13)
[2018-04-07 13:16:43.224]      Flags: F_FORWARD F_IPV4
dnsmasq: 13 192.168.30.30/51473 reply api-global.latency.prodaa.netflix.com is 54.85.157.92
[2018-04-07 13:16:43.229] **** got reply api-global.latency.prodaa.netflix.com is 54.85.157.92 (TTL 60, ID 13)
[2018-04-07 13:16:43.230]      Flags: F_FORWARD F_IPV4
dnsmasq: 13 192.168.30.30/51473 reply api-global.latency.prodaa.netflix.com is 107.23.209.173
[2018-04-07 13:16:43.235] **** got reply api-global.latency.prodaa.netflix.com is 107.23.209.173 (TTL 60, ID 13)
[2018-04-07 13:16:43.235]      Flags: F_FORWARD F_IPV4
dnsmasq: 14 192.168.30.30/50380 query[A] api-global.netflix.com from 192.168.30.30
dnsmasq: 14 192.168.30.30/50380 cached api-global.netflix.com is <CNAME>
dnsmasq: 14 192.168.30.30/50380 cached api-global.geo.netflix.com is <CNAME>
dnsmasq: 14 192.168.30.30/50380 cached api-global.latency.prodaa.netflix.com is 107.23.209.173
[2018-04-07 13:16:43.249] **** got cache answer for api-global.latency.prodaa.netflix.com / 107.23.209.173 / <unknown> (TTL 60, ID 14)
[2018-04-07 13:16:43.250]      Flags: F_FORWARD F_IPV4
dnsmasq: 14 192.168.30.30/50380 cached api-global.latency.prodaa.netflix.com is 54.85.157.92
[2018-04-07 13:16:43.255] **** got cache answer for api-global.latency.prodaa.netflix.com / 54.85.157.92 / <unknown> (TTL 60, ID 14)
[2018-04-07 13:16:43.256]      Flags: F_FORWARD F_IPV4
dnsmasq: 14 192.168.30.30/50380 cached api-global.latency.prodaa.netflix.com is 52.73.224.7
[2018-04-07 13:16:43.261] **** got cache answer for api-global.latency.prodaa.netflix.com / 52.73.224.7 / <unknown> (TTL 60, ID 14)
[2018-04-07 13:16:43.261]      Flags: F_FORWARD F_IPV4
dnsmasq: 14 192.168.30.30/50380 cached api-global.latency.prodaa.netflix.com is 52.6.75.180
[2018-04-07 13:16:43.266] **** got cache answer for api-global.latency.prodaa.netflix.com / 52.6.75.180 / <unknown> (TTL 60, ID 14)
[2018-04-07 13:16:43.267]      Flags: F_FORWARD F_IPV4
dnsmasq: 14 192.168.30.30/50380 cached api-global.latency.prodaa.netflix.com is 52.5.61.180
[2018-04-07 13:16:43.272] **** got cache answer for api-global.latency.prodaa.netflix.com / 52.5.61.180 / <unknown> (TTL 60, ID 14)
[2018-04-07 13:16:43.273]      Flags: F_FORWARD F_IPV4
dnsmasq: 14 192.168.30.30/50380 cached api-global.latency.prodaa.netflix.com is 34.225.135.220
[2018-04-07 13:16:43.278] **** got cache answer for api-global.latency.prodaa.netflix.com / 34.225.135.220 / <unknown> (TTL 60, ID 14)
[2018-04-07 13:16:43.279]      Flags: F_FORWARD F_IPV4
dnsmasq: 14 192.168.30.30/50380 cached api-global.latency.prodaa.netflix.com is 34.204.238.125
[2018-04-07 13:16:43.284] **** got cache answer for api-global.latency.prodaa.netflix.com / 34.204.238.125 / <unknown> (TTL 60, ID 14)
[2018-04-07 13:16:43.285]      Flags: F_FORWARD F_IPV4
dnsmasq: 14 192.168.30.30/50380 cached api-global.latency.prodaa.netflix.com is 34.202.114.174
[2018-04-07 13:16:43.290] **** got cache answer for api-global.latency.prodaa.netflix.com / 34.202.114.174 / <unknown> (TTL 60, ID 14)
[2018-04-07 13:16:43.290]      Flags: F_FORWARD F_IPV4
[2018-04-07 13:17:00.198] Notice: Queries stored in DB: 7 (took 125.7 ms)
[2018-04-07 13:18:00.140] Notice: Queries stored in DB: 0 (took 94.3 ms)
[2018-04-07 13:19:00.156] Notice: Queries stored in DB: 0 (took 107.9 ms)

i`m running 4 TVs all streaming Netflix, FTLDNS does not crash.

Okay, so this very much sounds like it is no FTLDNS problem at all, but a problem of Cloudflare's DoH and/or dnscrypt. FTLDNS doesn't care (as much as dnsmasq didn't, either) if you are sending your queries to 1.1.1.1, 8.8.8.8, 9.9.9.9 or wherever on whatever port.

Looking at @Jason_A's log excerpt, it seems like there are no new incoming queries to the Pi-hole after 13:16:43. However, FTLDNS is still running, as you can see in the messages that it stored queries in the database.

  • Does it only stop responding to the Netflix queries of your TV or does it stop responding to all queries for all devices in your network?
    Even if it would be FTLDNS's fault, it isn't clear to me how FTLDNS should even know the time you switched off your TV. The only thing I could imagine here is that somehow Cloudflare+DoH doesn't answer / answers wrong to the netflix queries and your TV might then be sending thousands of queries per second to your Pi-hole keeping it 100% busy.
  • Can you confirm something like this from the statistics on your dashboard?
  • Also, could you please try using another DNS upstream provider (preferably not 1.1.1.1) to see if you can confirm that it is a problem on their side?

@Tntdruid is one of them also a Samsung device? I have no (Smart)TV, so I cannot test this at all.

1 Like

@DL6ER Yes, all 4 are Samsung TVs

Okay, but you are not using this DoH or dnscrypt, right? Which upstream DNS servers are you using?

@DL6ER No i dont use dnscrypt

PIHOLE_DNS_1=1.1.1.1
PIHOLE_DNS_2=1.0.0.1
PIHOLE_DNS_3=2606:4700:4700::1111
PIHOLE_DNS_4=2606:4700:4700::1001

Cloudflare

@Tntdruid, any chance you could try setting up DoH (as per this guide) and seeing if you experience the same issues?

For the pi-hole part, just blank out the PIHOLE_DNS_X lines, create /etc/dnsmasq.d/50-cloudflared.conf (containing SERVER=127.0.0.1#5053), and then run pihole -r.

I have done this and have no issues with Netflix, but then again, I am not running a Samsung Smart TV

Edit: starting to see this using Netflix on XboxOne

@PromoFaux Looke like DoH mess stuff up, going to remove it again.

Yeah, probably wise. Dom and I are trying to debug it currently. You can actually still make queries over DoH, it's just FTLDNS does not like this particular query via DoH for some reason...

OK, it's not FTLDNS, it's dnsmasq. Can reproduce the same thing on a system with only dnsmasq

1 Like

Confirmed, dnsmasq stalls at

#0  0x00007ffff7b15700 in __read_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000555555566533 in read_write (fd=fd@entry=13, packet=packet@entry=0x7fffffffe334 "\002[\001", size=size@entry=1, rw=rw@entry=1) at util.c:654
#2  0x000055555557029c in tcp_request (confd=13, now=1523191598, local_addr=0x7fffffffe410, netmask=..., auth_dns=0) at forward.c:1713
#3  0x0000555555575999 in check_dns_listeners (now=1523191598) at dnsmasq.c:1745
#4  0x000055555555d019 in main (argc=<optimized out>, argv=<optimized out>) at dnsmasq.c:1061

I will contact Simon Kelly about it - I'm afraid there will be no quick solution to this... Please don't use DoH meanwhile!

3 Likes

This might not necessarily help anything... But I have seen no issues with cloudflare DNS using DNS over TLS (via getdns and stubby). Maybe it would be useful to try the same use case but with the DNS over TLS to see if the prolem persists?

https://dnsprivacy.org/wiki/pages/viewpage.action?pageId=3145786

Do you need to have netflix to trigger the problem or does a CURL to a certain netflix domain trigger the bug as well? For reproduction purposes?

It just so happens i'm currently testing DoH on firefox, which internally makes use of the same cloudflare solution.

have you tried enabling DNSSEC and tried to reproduce?

The exact point at which dnsmasq crapped out was after these two queries:

Apr  8 13:30:49 dnsmasq[16025]: 3 192.168.0.6/53524 query[A] api-global.netflix.com from 192.168.0.6
Apr  8 13:30:49 dnsmasq[16025]: 3 192.168.0.6/53524 forwarded api-global.netflix.com to 127.0.0.1
Apr  8 13:30:49 dnsmasq[16025]: 3 192.168.0.6/53524 reply api-global.netflix.com is <CNAME>
Apr  8 13:30:49 dnsmasq[16025]: 3 192.168.0.6/53524 reply api-global.geo.netflix.com is <CNAME>
Apr  8 13:30:49 dnsmasq[16025]: 3 192.168.0.6/53524 reply api-global.latency.prodaa.netflix.com is 52.5.237.4
Apr  8 13:30:49 dnsmasq[16025]: 3 192.168.0.6/53524 reply api-global.latency.prodaa.netflix.com is 52.44.197.215
Apr  8 13:30:49 dnsmasq[16025]: 3 192.168.0.6/53524 reply api-global.latency.prodaa.netflix.com is 52.45.38.64
Apr  8 13:30:49 dnsmasq[16025]: 3 192.168.0.6/53524 reply api-global.latency.prodaa.netflix.com is 52.45.118.37
Apr  8 13:30:49 dnsmasq[16025]: 3 192.168.0.6/53524 reply api-global.latency.prodaa.netflix.com is 52.54.15.52
Apr  8 13:30:49 dnsmasq[16025]: 3 192.168.0.6/53524 reply api-global.latency.prodaa.netflix.com is 52.54.22.121
Apr  8 13:30:49 dnsmasq[16025]: 3 192.168.0.6/53524 reply api-global.latency.prodaa.netflix.com is 52.54.174.212
Apr  8 13:30:49 dnsmasq[16025]: 3 192.168.0.6/53524 reply api-global.latency.prodaa.netflix.com is 52.54.242.21
Apr  8 13:30:49 dnsmasq[16025]: 4 192.168.0.6/50363 query[A] api-global.netflix.com from 192.168.0.6
Apr  8 13:30:49 dnsmasq[16025]: 4 192.168.0.6/50363 cached api-global.netflix.com is <CNAME>
Apr  8 13:30:49 dnsmasq[16025]: 4 192.168.0.6/50363 cached api-global.geo.netflix.com is <CNAME>
Apr  8 13:30:49 dnsmasq[16025]: 4 192.168.0.6/50363 cached api-global.latency.prodaa.netflix.com is 52.54.242.21
Apr  8 13:30:49 dnsmasq[16025]: 4 192.168.0.6/50363 cached api-global.latency.prodaa.netflix.com is 52.54.174.212
Apr  8 13:30:49 dnsmasq[16025]: 4 192.168.0.6/50363 cached api-global.latency.prodaa.netflix.com is 52.54.22.121
Apr  8 13:30:49 dnsmasq[16025]: 4 192.168.0.6/50363 cached api-global.latency.prodaa.netflix.com is 52.54.15.52
Apr  8 13:30:49 dnsmasq[16025]: 4 192.168.0.6/50363 cached api-global.latency.prodaa.netflix.com is 52.45.118.37
Apr  8 13:30:49 dnsmasq[16025]: 4 192.168.0.6/50363 cached api-global.latency.prodaa.netflix.com is 52.45.38.64
Apr  8 13:30:49 dnsmasq[16025]: 4 192.168.0.6/50363 cached api-global.latency.prodaa.netflix.com is 52.44.197.215
Apr  8 13:30:49 dnsmasq[16025]: 4 192.168.0.6/50363 cached api-global.latency.prodaa.netflix.com is 52.5.237.4
Apr  8 13:53:07 dnsmasq[25121]: 119 192.168.0.6/52192 query[A] ichnaea.netflix.com from 192.168.0.6
Apr  8 13:53:07 dnsmasq[25121]: 119 192.168.0.6/52192 forwarded ichnaea.netflix.com to 127.0.0.1
Apr  8 13:53:07 dnsmasq[25121]: 119 192.168.0.6/52192 reply ichnaea.netflix.com is <CNAME>
Apr  8 13:53:07 dnsmasq[25121]: 119 192.168.0.6/52192 reply ichnaea.geo.netflix.com is <CNAME>
Apr  8 13:53:07 dnsmasq[25121]: 119 192.168.0.6/52192 reply ichnaea.latency.prodaa.netflix.com is 34.195.89.28
Apr  8 13:53:07 dnsmasq[25121]: 119 192.168.0.6/52192 reply ichnaea.latency.prodaa.netflix.com is 34.197.160.123
Apr  8 13:53:07 dnsmasq[25121]: 119 192.168.0.6/52192 reply ichnaea.latency.prodaa.netflix.com is 34.203.138.40
Apr  8 13:53:07 dnsmasq[25121]: 119 192.168.0.6/52192 reply ichnaea.latency.prodaa.netflix.com is 34.206.109.253
Apr  8 13:53:07 dnsmasq[25121]: 119 192.168.0.6/52192 reply ichnaea.latency.prodaa.netflix.com is 54.81.133.206
Apr  8 13:53:07 dnsmasq[25121]: 119 192.168.0.6/52192 reply ichnaea.latency.prodaa.netflix.com is 54.83.180.65
Apr  8 13:53:07 dnsmasq[25121]: 119 192.168.0.6/52192 reply ichnaea.latency.prodaa.netflix.com is 54.85.134.97
Apr  8 13:53:07 dnsmasq[25121]: 119 192.168.0.6/52192 reply ichnaea.latency.prodaa.netflix.com is 54.87.182.219
Apr  8 13:53:07 dnsmasq[25121]: 120 192.168.0.6/50382 query[A] ichnaea.netflix.com from 192.168.0.6
Apr  8 13:53:07 dnsmasq[25121]: 120 192.168.0.6/50382 cached ichnaea.netflix.com is <CNAME>
Apr  8 13:53:07 dnsmasq[25121]: 120 192.168.0.6/50382 cached ichnaea.geo.netflix.com is <CNAME>
Apr  8 13:53:07 dnsmasq[25121]: 120 192.168.0.6/50382 cached ichnaea.latency.prodaa.netflix.com is 54.87.182.219
Apr  8 13:53:07 dnsmasq[25121]: 120 192.168.0.6/50382 cached ichnaea.latency.prodaa.netflix.com is 54.85.134.97
Apr  8 13:53:07 dnsmasq[25121]: 120 192.168.0.6/50382 cached ichnaea.latency.prodaa.netflix.com is 54.83.180.65
Apr  8 13:53:07 dnsmasq[25121]: 120 192.168.0.6/50382 cached ichnaea.latency.prodaa.netflix.com is 54.81.133.206
Apr  8 13:53:07 dnsmasq[25121]: 120 192.168.0.6/50382 cached ichnaea.latency.prodaa.netflix.com is 34.206.109.253
Apr  8 13:53:07 dnsmasq[25121]: 120 192.168.0.6/50382 cached ichnaea.latency.prodaa.netflix.com is 34.203.138.40
Apr  8 13:53:07 dnsmasq[25121]: 120 192.168.0.6/50382 cached ichnaea.latency.prodaa.netflix.com is 34.197.160.123
Apr  8 13:53:07 dnsmasq[25121]: 120 192.168.0.6/50382 cached ichnaea.latency.prodaa.netflix.com is 34.195.89.28

thanks. I've changed my dns to the 5053 provider. ran curl on both addresses to trigger the DNS lookup (verified it triggered response via tcpdump)

but dnsmasq does not seem to mind or crash

I do run "proxy-dnssec" (as only other config option). Can anyone confirm that curl to those two addresses triggers the problem on their systems?

--Edit: borrowed a laptop with netflix and ran it across the DNS. see it run through 15 (?) netflix domains and a bunch of akamai stuff, but i can't seem to trigger the problem

--edit2: it seems for me the actual cloudflare app crashed that time and dnsmasq reloaded to defaults :slight_smile: it is strange. but it does not seem to be any single url on its own