Confirming Chromecast is blackholed/sinkholed

I've posted results from a few minutes of the suggested command sudo tcpdump -npi eth0 -vvv host 192.168.0.2 and port 53 on my Pi.

The Pi IP address is 192.168.0.2. The Chromecast is 192.168.0.250. I have this

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    202    0        0 eth0
8.8.4.4         192.168.0.2     255.255.255.255 UGH   0      0        0 eth0
8.8.8.8         192.168.0.2     255.255.255.255 UGH   0      0        0 eth0
192.168.0.0     0.0.0.0         255.255.255.0   U     202    0        0 eth0

But the Chromecast IP, 192.168.0.250, is not showing in the tcpdump.

I also have the suggestions from the Chromecast (Don’t) Phone Home thread, so I have <?php header("HTTP/1.0 204 No Content"); ?> and the entry in /etc/hosts.

I don't see any traffic going to 8.8.8.8 nor 8.8.4.4. I have an Arris cable modem so I had to configure DNS to point to the Pi, and I enabled Port Forwarding so UDP port 53 goes to the Pi IP which is 192.168.0.2. arris-port-forwarding arris-router-set-dns Pings from the Pi to the Google Public DNS addresses timeout:

pi@raspberrypi:~ $ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 192.168.0.2 icmp_seq=1 Destination Host Unreachable
From 192.168.0.2 icmp_seq=2 Destination Host Unreachable

pi@raspberrypi:~ $ ping 8.8.4.4
PING 8.8.4.4 (8.8.4.4) 56(84) bytes of data.
From 192.168.0.2 icmp_seq=1 Destination Host Unreachable

A nslookup via my Mac's Network Utility to 8.8.8.8 results in:

Lookup has started…
8.8.4.4 -> google-public-dns-b.google.com

Cloudflare and Comodo are the selected DNS options in pi-hole. I have this iptables rule too:

iptables -L -n -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DNAT       udp  --  8.8.4.4              0.0.0.0/0            udp dpt:53 to:10.141.165.1
DNAT       udp  --  8.8.8.8              0.0.0.0/0            udp dpt:53 to:10.141.165.1
DNAT       tcp  --  192.168.1.100        0.0.0.0/0            tcp dpt:53 to:10.141.165.1
DNAT       udp  --  0.0.0.0/0            8.8.8.8              udp dpt:53 to:10.141.165.1
DNAT       udp  --  0.0.0.0/0            8.8.4.4              udp dpt:53 to:10.141.165.1

And some more nslookup's from my Mac:

nslookup
> server 8.8.8.8
Default server: 8.8.8.8
Address: 8.8.8.8#53
> ping google.com
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
Name:	google.com
Address: 172.217.3.110
Name:	google.com
Address: 2607:f8b0:4006:818::200e

When visiting google.com from my Mac, IP 192.168.0.171 I see these logs:

Apr 26 00:29:07 dnsmasq[2168]: query[A] www.gstatic.com from 192.168.0.171
Apr 26 00:29:07 dnsmasq[2168]: forwarded www.gstatic.com to 1.0.0.1
Apr 26 00:29:07 dnsmasq[2168]: reply www.gstatic.com is 172.217.10.227
Apr 26 00:29:07 dnsmasq[2168]: query[A] adservice.google.com from 192.168.0.171
Apr 26 00:29:07 dnsmasq[2168]: /etc/pihole/gravity.list adservice.google.com is 0.0.0.0
Apr 26 00:29:07 dnsmasq[2168]: query[A] apis.google.com from 192.168.0.171
Apr 26 00:29:07 dnsmasq[2168]: forwarded apis.google.com to 1.0.0.1
Apr 26 00:29:07 dnsmasq[2168]: reply apis.google.com is <CNAME>
Apr 26 00:29:07 dnsmasq[2168]: reply plus.l.google.com is 172.217.7.14
Apr 26 00:29:08 dnsmasq[2168]: query[A] a1507.b.akamai.net from 192.168.0.171
Apr 26 00:29:08 dnsmasq[2168]: forwarded a1507.b.akamai.net to 1.0.0.1
Apr 26 00:29:08 dnsmasq[2168]: reply a1507.b.akamai.net is 104.70.178.115
Apr 26 00:29:08 dnsmasq[2168]: reply a1507.b.akamai.net is 104.70.178.104
Apr 26 00:29:08 dnsmasq[2168]: query[A] pancake.cdn-apple.com.akadns.net from 192.168.0.171
Apr 26 00:29:08 dnsmasq[2168]: forwarded pancake.cdn-apple.com.akadns.net to 1.0.0.1
Apr 26 00:29:08 dnsmasq[2168]: reply pancake.cdn-apple.com.akadns.net is <CNAME>
Apr 26 00:29:08 dnsmasq[2168]: reply pancake.g.aaplimg.com is 17.253.97.202
Apr 26 00:29:08 dnsmasq[2168]: reply pancake.g.aaplimg.com is 17.253.97.204

Is there anything else I can check to see this is working as desired? How can I use dig and/or nslookup to test this and what should I look for in the Query Logs?

I don't have an Arris modem, but I would strongly suggest to verify that you haven't accidently opened port 53 to the outside world, by running the test on this page, select server port test, press scan ports, the result should be Timed Out. Opening port 53 to the outside world is a very bad idea.

First select a domain that is blocked. Assuming doubleclick.net is in one of your blocklists, you can verify this by entering the command pihole -q doubleclick.net on your pihole server, the result probably is something like [i] Over 100 results found for doubleclick.net, depending on the blocklists you use, it should be used at least once.

From a windows computer, using pihole's DNS services, enter the command:
nslookup doubleclick.net 8.8.8.8

Assuming you use null blocking (pihole's default, read about it here) the result should be:

nslookup doubleclick.net 8.8.8.8
Server:  google-public-dns-a.google.com
Address:  8.8.8.8

Name:    doubleclick.net
Addresses:  ::
          0.0.0.0

If the redirection is configured correctly on your router, the request to 8.8.8.8 is redirected to your pihole. Since doubleclick.net is in your blocklist (assumption above) the reply is 0.0.0.0 (assuming blocking mode NULL blocking), indicating it is NOT 8.8.8.8 that replies, but your pihole. If you get the real address for doubleclick.net (something like216.58.212.174) your router configuration (NAT) isn't correct.

Repeat this for the secondary google dns server, e.g. nslookup doubleclick.net 8.8.4.4

Screenshot:

Indeed:

pihole -q doubleclick.net
  [i] Over 100 results found for doubleclick.net
        This can be overridden using the -all option

Looks like I have the following seems I should change it:

cat /etc/pihole/pihole-FTL.conf
PRIVACYLEVEL=0

Here we go, from a Mac at least:

nslookup
>8.8.8.8
8.8.8.8.in-addr.arpa name = google-public-dns-a.google.com.
Authoritative answers can be found from:
> doubleclick.net
Server: 192.168.0.2
Address: 192.168.0.2#53
Name: doubleclick.net
Address: 0.0.0.0
Name: doubleclick.net
Address: ::

And:

nslookup
> 8.8.4.4
4.4.8.8.in-addr.arpa	name = google-public-dns-b.google.com.

Authoritative answers can be found from:
> doubleclick.net
Server:		192.168.0.2
Address:	192.168.0.2#53

Name:	doubleclick.net
Address: 0.0.0.0
Name:	doubleclick.net
Address: ::

Thank you for the thorough response! Will be very useful in my final paper for an IOT class!

I don't have a mac, so I can't duplicate it, but it looks wrong. From the looks of it, you're using pihole to resolve. When you enter 8.8.8.8 without the server command, you're simply doing a reverse lookup, e.g. find the DNS name for the IP address, using the default DNS server (192.168.0.2 witch is probably your pihole)

Using windows, if you want to instruct nslookup to use a different server to resolve, you need to enter the server command.

nslookup
Default Server:  raspberry57.localdomain
Address:  192.168.2.57

> server 8.8.8.8
Default Server:  google-public-dns-a.google.com
Address:  8.8.8.8

> doubleclick.net
Server:  google-public-dns-a.google.com
Address:  8.8.8.8

Name:    doubleclick.net
Addresses:  ::
          0.0.0.0

Notice the Default Server: output from nslookup.

If redirection (NAT) is really working, you should find the following at the end of /var/log/pihole.log:

Apr 28 09:47:34 dnsmasq[17071]: query[A] doubleclick.net from x.x.x.x
Apr 28 09:47:34 dnsmasq[17071]: /etc/pihole/gravity.list doubleclick.net is 0.0.0.0

x.x.x.x being the IP address from your MAC.

edit
warning: you'll see the exact same entries in the pihole log if you're using the pihole as the resolver, when using nslookup, so you have to ensure you really make nslookup use 8.8.8.8 for your test.
/edit

In Query Log I was seeing this when I did the nslookup from a Mac:
image

But on a Windows PC I see this, which seems to imply it's not working:
image

Perhaps I'm misunderstanding what the "server" command does in nslookup. On My Mac if I do nslookup followed by server (without an IP) it clearly defaults to the Pi Hole IP . For more testing, I added aol.com manually to the blacklist via the GUI.

nslookup
> server
Default server: 192.168.0.2
Address: 192.168.0.2#53
> aol.com
Server:		192.168.0.2
Address:	192.168.0.2#53

Name:	aol.com
Address: 0.0.0.0
Name:	aol.com
Address: ::
> doubleclick.net
Server:		192.168.0.2
Address:	192.168.0.2#53

Name:	doubleclick.net
Address: 0.0.0.0
Name:	doubleclick.net
Address: ::
> doubleclick.net
Server:		192.168.0.2
Address:	192.168.0.2#53

Name:	doubleclick.net
Address: 0.0.0.0
Name:	doubleclick.net
Address: ::

From the /var/log/pihole.log file it appears to be to be the desired results:

Apr 28 19:06:25 dnsmasq[576]: query[A] aol.com from 192.168.0.171
Apr 28 19:06:25 dnsmasq[576]: /etc/pihole/regex.list aol.com is 0.0.0.0
Apr 28 19:06:25 dnsmasq[576]: query[AAAA] aol.com from 192.168.0.171
Apr 28 19:06:25 dnsmasq[576]: /etc/pihole/regex.list aol.com is 0.0.0.0
Apr 28 19:06:30 dnsmasq[576]: query[A] doubleclick.net from 192.168.0.171
Apr 28 19:06:30 dnsmasq[576]: /etc/pihole/gravity.list doubleclick.net is 0.0.0.0
Apr 28 19:06:30 dnsmasq[576]: query[AAAA] doubleclick.net from 192.168.0.171
Apr 28 19:06:30 dnsmasq[576]: /etc/pihole/gravity.list doubleclick.net is 0.0.0.0

Now again on my Mac if I follow your suggestion to specify 8.8.8.8 along with the server command I get the following and no logs in /var/log/pihole.log, which implies I'm bypassing the Pi Hole using the server command:

nslookup
> server 8.8.8.8
Default server: 8.8.8.8
Address: 8.8.8.8#53
> 
> aol.com
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
Name:	aol.com
Address: 124.108.115.87
Name:	aol.com
Address: 66.218.87.12
Name:	aol.com
Address: 67.195.231.10
Name:	aol.com
Address: 106.10.218.150
Name:	aol.com
Address: 188.125.72.165
> doubleclick.net
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
Name:	doubleclick.net
Address: 172.217.12.206
Name:	doubleclick.net
Address: 2607:f8b0:4006:81b::200e

Here's a screen shot of the Pi Hole query log:

Traffic is clearly going through the Pi Hole and I did set BLOCKINGMODE=NULL
in /etc/pihole/pihole-FTL.conf and restarted the Pi Hole.

And here's the nslookup from the Mac GUI Network Utility:
31%20PM

Also here are results from these dig and nslookup's I've seen asked for on other threads here:

nslookup pi-hole.net
Server:		192.168.0.2
Address:	192.168.0.2#53

Non-authoritative answer:
Name:	pi-hole.net
Address: 206.189.252.21
Name:	pi-hole.net
Address: 2604:a880:400:d0::1071:1

nslookup flurry.com
Server:		192.168.0.2
Address:	192.168.0.2#53

Name:	flurry.com
Address: 0.0.0.0
Name:	flurry.com
Address: ::

dig pi-hole.net

; <<>> DiG 9.12.3-P1 <<>> pi-hole.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32907
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;pi-hole.net.			IN	A

;; ANSWER SECTION:
pi-hole.net.		150	IN	A	206.189.252.21

;; Query time: 4 msec
;; SERVER: 192.168.0.2#53(192.168.0.2)
;; WHEN: Sun Apr 28 20:35:17 EDT 2019
;; MSG SIZE  rcvd: 56

dig flurry.com

; <<>> DiG 9.12.3-P1 <<>> flurry.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26990
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;flurry.com.			IN	A

;; ANSWER SECTION:
flurry.com.		2	IN	A	0.0.0.0

;; Query time: 6 msec
;; SERVER: 192.168.0.2#53(192.168.0.2)
;; WHEN: Sun Apr 28 20:35:33 EDT 2019
;; MSG SIZE  rcvd: 55

Would any debug logs help? Since I have a static route and iptables preroiuting NAT, could that be causing issues?

You are obviously getting nowhere with your current configuration, so I suggest you try the following:

  • Disable the port forwarding and the iptable rules you have already applied.
  • Change your Primary DNS Server (DHCP settings) to 8.8.8.8 (yes, NOT the pihole)

Now all DNS requests should go directly to 8.8.8.8, so nothing will appear in the pihole log.
Start configuring the firewall to redirect requests from all systems to IP 8.8.8.8 port 53 to your pihole
If you get it right, requests your system(s) make will start appearing in the pihole log. To look dynamically @ the pihole log, you can use either tools / tail pihole log from the pihole admin console or look at this topic to get a dynamic view.

As soon as all of the DNS requests from all machines appear in the pihole log, you can change your Primary DNS Server (DHCP settings) back to the pihole and duplicate, if appropriate, the rules you have configured for 8.8.4.4.

From what I read in your comments, you appear to be ignoring the fact that most DNS tools will always use the default DNS server (the pihole in your current configuration). Always specify the DNS server to use when using nlsookup or dig.
Some examples:
nslookup -query=A doubleclick.net 208.67.222.222 : gets the IPv4 address for doubleclick.net from server 208.67.222.222 (OpenDNS server)
dig @208.67.222.222 -p 53 A doubleclick.net : gets the IPv4 addresses for doubleclick.net from server 208.67.222.222 (OpenDNS server)
dig @208.67.222.222 -p 53 AAAA doubleclick.net : gets the IPv6 addresses for doubleclick.net from server 208.67.222.222 (OpenDNS server)

FWIW I've always had traffic going through the Pi Hole and logs have been showing all devices go through it. And just to reiterate, the firewall is simply iptables in the Raspberry Pi that the Pi Hole is running on.

Here are some logs showing my Mac using the Pi Hole as DNS right after making the change:

Apr 29 23:05:59 dnsmasq[576]: query[A] caldav.calendar.yahoo.com from 192.168.0.92
Apr 29 23:05:59 dnsmasq[576]: forwarded caldav.calendar.yahoo.com to 1.0.0.1
Apr 29 23:05:59 dnsmasq[576]: reply caldav.calendar.yahoo.com is <CNAME>
Apr 29 23:05:59 dnsmasq[576]: reply ds-geoycpi-uno-deluxe.gycpi.b.yahoodns.net is 69.147.82.61
Apr 29 23:05:59 dnsmasq[576]: reply ds-geoycpi-uno-deluxe.gycpi.b.yahoodns.net is 69.147.82.60
Apr 29 23:06:08 dnsmasq[576]: query[A] docs.google.com from 192.168.0.92
Apr 29 23:06:08 dnsmasq[576]: forwarded docs.google.com to 1.0.0.1
Apr 29 23:06:08 dnsmasq[576]: reply docs.google.com is 172.217.12.142
Apr 29 23:06:11 dnsmasq[576]: query[A] 11.client-channel.google.com from 192.168.0.92
Apr 29 23:06:11 dnsmasq[576]: forwarded 11.client-channel.google.com to 1.0.0.1
Apr 29 23:06:11 dnsmasq[576]: reply 11.client-channel.google.com is 173.194.205.189
Apr 29 23:06:16 dnsmasq[576]: query[A] star-mini.c10r.facebook.com from 192.168.0.92
Apr 29 23:06:16 dnsmasq[576]: forwarded star-mini.c10r.facebook.com to 1.0.0.1
Apr 29 23:06:16 dnsmasq[576]: reply star-mini.c10r.facebook.com is 31.13.71.36
Apr 29 23:07:38 dnsmasq[576]: query[A] 27.client-channel.google.com from 192.168.0.92
Apr 29 23:07:38 dnsmasq[576]: forwarded 27.client-channel.google.com to 1.0.0.1
Apr 29 23:07:38 dnsmasq[576]: reply 27.client-channel.google.com is 173.194.66.189
Apr 29 23:07:52 dnsmasq[576]: query[A] googlemail.l.google.com from 192.168.0.92
Apr 29 23:07:52 dnsmasq[576]: forwarded googlemail.l.google.com to 1.0.0.1
Apr 29 23:07:52 dnsmasq[576]: reply googlemail.l.google.com is 172.217.6.197
Apr 29 23:09:01 dnsmasq[576]: query[A] hangouts.google.com from 192.168.0.92
Apr 29 23:09:01 dnsmasq[576]: forwarded hangouts.google.com to 1.0.0.1
Apr 29 23:09:01 dnsmasq[576]: reply hangouts.google.com is <CNAME>
Apr 29 23:09:01 dnsmasq[576]: reply www3.l.google.com is 172.217.12.142
Apr 29 23:09:02 dnsmasq[576]: query[A] chat-pa.clients6.google.com from 192.168.0.92
Apr 29 23:09:02 dnsmasq[576]: forwarded chat-pa.clients6.google.com to 1.0.0.1
Apr 29 23:09:02 dnsmasq[576]: reply chat-pa.clients6.google.com is 172.217.10.106
Apr 29 23:09:04 dnsmasq[576]: query[A] 21.client-channel.google.com from 192.168.0.92
Apr 29 23:09:04 dnsmasq[576]: forwarded 21.client-channel.google.com to 1.0.0.1
Apr 29 23:09:04 dnsmasq[576]: reply 21.client-channel.google.com is 209.85.232.189
Apr 29 23:09:36 dnsmasq[576]: query[TXT] time-osx.g.aaplimg.com from 192.168.0.92
Apr 29 23:09:36 dnsmasq[576]: forwarded time-osx.g.aaplimg.com to 1.0.0.1
Apr 29 23:09:36 dnsmasq[576]: reply time-osx.g.aaplimg.com is ntp minpoll 9 maxpoll 12 iburst
Apr 29 23:10:35 dnsmasq[576]: query[A] 25.client-channel.google.com from 192.168.0.92
Apr 29 23:10:35 dnsmasq[576]: forwarded 25.client-channel.google.com to 1.0.0.1
Apr 29 23:10:35 dnsmasq[576]: forwarded 25.client-channel.google.com to 1.1.1.1
Apr 29 23:10:35 dnsmasq[576]: forwarded 25.client-channel.google.com to 8.20.247.20
Apr 29 23:10:35 dnsmasq[576]: forwarded 25.client-channel.google.com to 8.26.56.26
Apr 29 23:10:35 dnsmasq[576]: reply 25.client-channel.google.com is 173.194.204.189
Apr 29 23:10:37 dnsmasq[576]: query[A] clients4.google.com from 192.168.0.92
Apr 29 23:10:37 dnsmasq[576]: forwarded clients4.google.com to 1.0.0.1
Apr 29 23:10:37 dnsmasq[576]: reply clients4.google.com is <CNAME>
Apr 29 23:10:37 dnsmasq[576]: reply clients.l.google.com is 172.217.12.206
Apr 29 23:12:00 dnsmasq[576]: query[A] 26.client-channel.google.com from 192.168.0.92
Apr 29 23:12:00 dnsmasq[576]: forwarded 26.client-channel.google.com to 1.0.0.1
Apr 29 23:12:00 dnsmasq[576]: reply 26.client-channel.google.com is 173.194.205.189
Apr 29 23:12:02 dnsmasq[576]: query[A] play.google.com from 192.168.0.92
Apr 29 23:12:02 dnsmasq[576]: forwarded play.google.com to 1.0.0.1
Apr 29 23:12:02 dnsmasq[576]: reply play.google.com is 172.217.10.238
Apr 29 23:13:09 dnsmasq[576]: query[A] www.google.com from 192.168.0.92
Apr 29 23:13:09 dnsmasq[576]: forwarded www.google.com to 1.0.0.1
Apr 29 23:13:09 dnsmasq[576]: query[A] www.arris.com from 192.168.0.92
Apr 29 23:13:09 dnsmasq[576]: forwarded www.arris.com to 1.0.0.1
Apr 29 23:13:09 dnsmasq[576]: reply www.google.com is 172.217.12.164
Apr 29 23:13:09 dnsmasq[576]: reply www.arris.com is 128.136.141.21
Apr 29 23:13:23 dnsmasq[576]: query[A] clients6.google.com from 192.168.0.92
Apr 29 23:13:23 dnsmasq[576]: forwarded clients6.google.com to 1.0.0.1
Apr 29 23:13:23 dnsmasq[576]: reply clients6.google.com is <CNAME>
Apr 29 23:13:23 dnsmasq[576]: reply clients.l.google.com is 172.217.10.110
Apr 29 23:13:23 dnsmasq[576]: query[A] clients.l.google.com from 192.168.0.92
Apr 29 23:13:23 dnsmasq[576]: cached clients.l.google.com is 172.217.10.110
Apr 29 23:13:23 dnsmasq[576]: query[A] people-pa.clients6.google.com from 192.168.0.92
Apr 29 23:13:23 dnsmasq[576]: forwarded people-pa.clients6.google.com to 1.0.0.1
Apr 29 23:13:23 dnsmasq[576]: reply people-pa.clients6.google.com is 172.217.11.10
Apr 29 23:13:28 dnsmasq[576]: query[A] 7.client-channel.google.com from 192.168.0.92
Apr 29 23:13:28 dnsmasq[576]: forwarded 7.client-channel.google.com to 1.0.0.1
Apr 29 23:13:28 dnsmasq[576]: reply 7.client-channel.google.com is 173.194.207.189
Apr 29 23:13:36 dnsmasq[576]: query[A] star-mini.c10r.facebook.com from 192.168.0.92
Apr 29 23:13:36 dnsmasq[576]: forwarded star-mini.c10r.facebook.com to 1.0.0.1
Apr 29 23:13:36 dnsmasq[576]: reply star-mini.c10r.facebook.com is 31.13.71.36

I don't get any errors when I use the iptables commands but perhaps the syntax is still incorrect? At the moment the Chromecast IP is 192.168.0.250. I did what you suggested, made DNS for the Arris 8.8.8.8, and deleted the static routes and deleted the iptables rules.

So for now I have these iptables rules, and yes I realize they are multiple commands to do almost the same thing:

sudo iptables -L -n -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DNAT       udp  -- !192.168.0.2         !192.168.0.2          udp dpt:53 to:192.168.0.2
DNAT       udp  --  192.168.0.250        8.8.8.8              udp dpt:53 to:192.168.0.2
DNAT       udp  --  0.0.0.0/0            8.8.8.8              udp dpt:53 to:192.168.0.2

I used this syntax:
sudo iptables -t nat -A PREROUTING -i br0 -p udp ! --source 192.168.0.2 ! --destination 192.168.0.2 --dport 53 -j DNAT --to 192.168.0.2
then:
sudo iptables -t nat -A PREROUTING -s 192.168.0.250/32 -d 8.8.8.8 -p udp --dport 53 -j DNAT --to 192.168.0.2
and finally:
sudo iptables -t nat -A PREROUTING -d 8.8.8.8 -p udp --dport 53 -j DNAT --to-destination 192.168.0.2

Don't these logs imply the rule is working?

Apr 29 22:49:32 dnsmasq[576]: reply 192.168.0.250 is NXDOMAIN
Apr 29 22:49:32 dnsmasq[576]: query[PTR] 8.8.8.8.in-addr.arpa from 127.0.0.1
Apr 29 22:49:32 dnsmasq[576]: forwarded 8.8.8.8.in-addr.arpa to 1.0.0.1
Apr 29 22:49:32 dnsmasq[576]: reply 8.8.8.8 is google-public-dns-a.google.com

And I really do appreciate you taking the time to walk me though this.

Here is what I think is happening:
You have, on your Arris cable modem, configured port forwarding for UDP port 53 to 192.168.0.2 (your pi)
You have configured, on your Arris cable modem, configured DHCP to handout 192.168.0.1 as the gateway address for all clients.

When any system on your network is making a DNS request to 192.168.0.2 (your pihole), the request will go directly to the pi.
When any system on your network is making a request to any other resolver than your pihole it will use the known routes and send the request to the outside world, using the default gateway to reach the destination. This initial request never comes near the pihole. If this request is an UDP request, the port forward rule on your Arris cable modem will redirect the UDP packet to the pi (192.168.0.2) but if it is a TCP packet, the request will directly go to the resolver, specified in the request. DNS uses UDP and TCP as fallback.

Given your configuration, as far as I can see the details in the screenshot, I don't think your pi is making successful UDP DNS requests anymore, because the Arris modem always bounces them back, unless, you have something like dnscrypt-proxy running on your pi, witch doesn't use port 53 but 443, or any other method that makes the pi NOT use port 53.

All UDP port 53 packets are bounced back to the pi, even the packets from the pi.
All TCP port 53 packets are passing trough, without any interference.

What you are trying to achieve is typically configured on the firewall/router that is specified as the default gateway, so all packets are captured.
Such a rule is called a NAT rule and typically requires the following mandatory parameters:

  • protocol: UDP/TCP (to redirect all DNS requests you need both)
  • source address: all systems EXCEPT the pihole, unless your pihole doesn't use port 53 to resolve DNS requests (dnscrypt-proxy or equivalent)
  • source port: any
  • destination: the address of the servers you don't want to contact (8.8.8.8)
  • destination port: 53
  • redirect target IP: the address of your pihole
  • redirect target port : 53

You may think the solution is to change the port forwarding to UDP/TCP but this is NOT the solution, you will expose the pihole to the public internet, witch is a very bad idea.
Again, I don't have an Arris cable modem, so I don't know what it does, but port forwarding rules are typically used to make a service in your local network available to the public.
Earlier (first reply to this topic) I referred a port scanner to test your exposure. Here is a different one, I wouldn't be surprised if you already have UDP port 53 exposed (bad idea), but again, I don't know what the Arris modem can and cannot do.

In conclusion, if you want to protect your network from the outside word, and use features such as NAT, you need a router/firewall such as pfsense placed between the cable modem and your network, unless of course the Arris modem is already capable of doing this.

This is all based on the screenshots and comments you posted. If your Arris cable modem does something else than I would expect from a cable modem, it might be incorrect.

Port Scanner shows closed:

So I removed the Port Forwarding for now as well.

Here are the Arris firewall options, set to Low for now:

I'm trying DHCP from the Pi Hole now:
58%20AM

Edit I realized only wired connections would work with the Pi Hole as DHCP so I disabled that.

There is also a Enable DNS Relay option that I'm unchecking:
00%20AM

Stil no 8.8.8.8 in the log file:
grep 8.8.8.8 /var/log/pihole.log

It does look like the connectivitycheck.gstatic.com check is getting re-routed though:

Apr 30 08:38:00 dnsmasq[26679]: query[A] connectivitycheck.gstatic.com from 192.168.0.136
Apr 30 08:38:00 dnsmasq[26679]: /etc/hosts connectivitycheck.gstatic.com is 192.168.0.2
Apr 30 08:38:00 dnsmasq[26679]: query[A] www.google.com from 192.168.0.136
Apr 30 08:38:00 dnsmasq[26679]: forwarded www.google.com to 1.1.1.1
Apr 30 08:38:00 dnsmasq[26679]: reply www.google.com is 172.217.7.4
Apr 30 08:38:00 dnsmasq[26679]: query[A] www.google.com from 192.168.0.136
Apr 30 08:38:00 dnsmasq[26679]: cached www.google.com is 172.217.7.4
Apr 30 08:38:01 dnsmasq[26679]: query[A] mtalk.google.com from 192.168.0.136
Apr 30 08:38:01 dnsmasq[26679]: forwarded mtalk.google.com to 1.1.1.1
Apr 30 08:38:01 dnsmasq[26679]: reply mtalk.google.com is <CNAME>
Apr 30 08:38:01 dnsmasq[26679]: reply mobile-gtalk.l.google.com is 209.85.201.188
Apr 30 08:38:01 dnsmasq[26679]: query[ANY] connectivitycheck.gstatic.com from 127.0.0.1
Apr 30 08:38:01 dnsmasq[26679]: /etc/hosts connectivitycheck.gstatic.com is 192.168.0.2
Apr 30 08:38:01 dnsmasq[26679]: query[A] www.google.com from 192.168.0.136
Apr 30 08:38:01 dnsmasq[26679]: cached www.google.com is 172.217.7.4
Apr 30 08:38:02 dnsmasq[26679]: query[NAPTR] us.sprint.rcs.telephony.goog from 192.168.0.136
Apr 30 08:38:02 dnsmasq[26679]: forwarded us.sprint.rcs.telephony.goog to 1.1.1.1
Apr 30 08:38:02 dnsmasq[26679]: query[SRV] _sips._tcp.us.sprint.rcs.telephony.goog from 192.168.0.136
Apr 30 08:38:02 dnsmasq[26679]: forwarded _sips._tcp.us.sprint.rcs.telephony.goog to 1.1.1.1
Apr 30 08:38:02 dnsmasq[26679]: query[A] us.sprint.rcs.telephony.goog from 192.168.0.136
Apr 30 08:38:02 dnsmasq[26679]: forwarded us.sprint.rcs.telephony.goog to 1.1.1.1
Apr 30 08:38:02 dnsmasq[26679]: reply us.sprint.rcs.telephony.goog is 64.9.243.111

8.8.8.8 will never show up in the pihole log file as the destination address, because the address will be replaced in the packets by the rules. 8.8.8.8 might show up in the pihole log if you query google-public-dns-a.google.com

If you want to have a better understanding of what is going on, install wireshark and analyze the DNS packets.

So I did that, and used host 192.168.0.109 (IP of the Chromecast) as a filter. I've uploaded the pcapng file to my Google Drive here. All I see are IGMP, ARP and multicast MDNS queries to IPs like 224.0.0.22 and 224.0.0.251.

I also did a dig:

dig google-public-dns-a.google.com

; <<>> DiG 9.12.3-P1 <<>> google-public-dns-a.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41565
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;google-public-dns-a.google.com.	IN	A

;; ANSWER SECTION:
google-public-dns-a.google.com.	3842 IN	A	8.8.8.8

;; Query time: 5 msec
;; SERVER: 192.168.0.2#53(192.168.0.2)
;; WHEN: Tue Apr 30 22:47:25 EDT 2019
;; MSG SIZE  rcvd: 75

But only see this the Pi Hole logs:

Apr 30 22:47:26 dnsmasq[26679]: query[A] google-public-dns-a.google.com from 192.168.0.129
Apr 30 22:47:26 dnsmasq[26679]: cached google-public-dns-a.google.com is 8.8.8.8
Apr 30 22:47:26 dnsmasq[26679]: query[A] 3.client-channel.google.com from 192.168.0.129
Apr 30 22:47:26 dnsmasq[26679]: forwarded 3.client-channel.google.com to 1.0.0.1
Apr 30 22:47:26 dnsmasq[26679]: reply 3.client-channel.google.com is 209.85.232.189

I see another user posted elsewhere about the MDNS queries.

Edit: capture from a Pi is very similar:

Capture everything, select multiple adapters by CTRL clicking.

Initiate a dns lookup

Stop the capture

Apply display filter (udp.port == 53 || tcp.port == 53)

Result:

The result appears to be coming from 8.8.8.8, but since the reply is 0.0.0.0, it's really the pihole that has answered.
Not all of the DNS traffic is shown in this example, my pihole is on a different physical ethernet network, and thus not captured by wireshark in this example.

As you suspected it doesn't appear 8.8.8.8 is being redirected nor blocked:

nslookup doubleclick.net 8.8.8.8
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
Name:	doubleclick.net
Address: 172.217.12.206
Name:	doubleclick.net
Address: 2607:f8b0:4006:81b::200e

Could it be the syntax error of iptables on my Pi Hole? Or the route command?

sudo iptables -L -n -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DNAT       udp  -- !192.168.0.2         !192.168.0.2          udp dpt:53 to:192.168.0.2
DNAT       udp  --  192.168.0.250        8.8.8.8              udp dpt:53 to:192.168.0.2
DNAT       udp  --  0.0.0.0/0            8.8.8.8              udp dpt:53 to:192.168.0.2
route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    202    0        0 eth0
8.8.8.8         192.168.0.2     255.255.255.255 UGH   0      0        0 eth0
192.168.0.0     0.0.0.0         255.255.255.0   U     202    0        0 eth0

As you suggested I did flush all the routes and iptables and started over. Seems to be the same.

From the wireshark screenshot you posted, I conclude that redirection (port forwarding) as shown in the first entry of this topic, isn't even doing what you expect, e.g. send the packet to 192.168.0.2 (the pihole), so nothing from 192.168.0.129 ever arrives at the pihole. Doesn't matter if your iptables are correct or incorrect, their never used.

I still think, as I was from the first entry in this topic, you cannot achieve redirection with a port forwarding rule, you need NAT rules (protocol - source address - source port - destination - destination port -redirect target IP - redirect target port). With port forwarding only, as defined in you initial screenshot, how would the modem be able to differentiate between a packet from a client (192.168.0.129) and a packet from the pihole (192.168.0.2)? In your screenshot, there is no option to specify the source address, so packets from your pi would also be redirected. If you get the port forwarding rule to work, you'll have no name resolution anymore (UDP requests), unless of course the pi uses something like dnscrypt-proxy (different port) …

The only reason I keep responding to this, is to find out where you'll end up, trying to get this solution to work, maybe I'll learn something new...

I removed the port forwarding option from the Arris cable modem. I definitely know that the Pi Hole is serving all devices on the network as I don't see ads on sites when I use my phone, iPad, or laptop. In fact the Pi Hole is blocking downloads from the Google Play store.

Here are the DNS-related options in the Arris.

46%20AM 21%20AM 04%20AM

The Wireshark capture and Query log here demonstrate my laptop is using the Pi Hole. So is it possible that the Chromecast is bypassing the Pi Hole because I can't create a static NAT route in my cable modem?

Yes. Some Google devices are known to do this.

So is this just a limitation of my cable modem even though I can force DNS of it to use my Pi Hole? That some how the Chromecast bypasses the DNS even with the sinkhole I created in the Pi Hole?

If the device has a DNS IP hard coded into it, it won't use any DNS provided by the router. The router can tell the device that Pi-Hole is your DNS, but the device will ignore it.

This is not related to any function of a cable modem, unless the modem includes a router with ti.

Some routers have a firewall capability where you can redirect DNS traffic to Pi-Hole. If any device is trying to use Google DNS instead of Pi-Hole, the router will see these DNS requests coming from the device to Google, and will redirect those requests to the Pi-Hole.