I don't disagree about the docker setup but as I mentioned before the ansible build I referenced in my original post is how it was built/installed and that happened to be docker. I don't know enough about containers yet to know how to run unbound in one.
I did turn on all of my tracelogging and flow debugging in my juniper and could see the outbound packets heading off to the internet port 53. I also see the firewall accept packet in splunk which is there I send my juniper syslogs to. So I'm pretty confident that it's something on the pi itself.
So I turned off unbound as an upstream and checked the two google IPV4 boxes and it works. Further troubleshooting shows a query coming from my desktop to pi-hole looking for bolt.dropbox. That request then goes out to 8.8.8.8:53 (this is with the two google IPV4 checkboxes off and just 127.0.0.1#5335 turned on) and shortly I see the packet come back in.
pi@inet-pi:~$ sudo tcpdump -nn port 5335 or port 53
07:14:48.699512 IP 10.20.15.132.49194 > 10.20.15.176.53: 59636+ A? bolt.dropbox.com. (34)
07:14:49.703916 IP 10.20.15.132.49194 > 10.20.15.176.53: 59636+ A? bolt.dropbox.com. (34)
07:14:51.707041 IP 10.20.15.132.49194 > 10.20.15.176.53: 59636+ A? bolt.dropbox.com. (34)
07:14:55.711211 IP 10.20.15.132.49194 > 10.20.15.176.53: 59636+ A? bolt.dropbox.com. (34)
07:15:02.643189 IP 10.20.15.176.38102 > 8.8.8.8.53: 61962+ PTR? 8.8.8.8.in-addr.arpa. (38)
07:15:02.659272 IP 8.8.8.8.53 > 10.20.15.176.38102: 61962 1/0/0 PTR dns.google. (62)
07:15:03.714237 IP 10.20.15.132.49194 > 10.20.15.176.53: 59636+ A? bolt.dropbox.com. (34)
07:15:16.671592 IP 10.20.15.176.59564 > 8.8.8.8.53: 59765+ PTR? 4.4.8.8.in-addr.arpa. (38)
07:15:16.688305 IP 8.8.8.8.53 > 10.20.15.176.59564: 59765 1/0/0 PTR dns.google. (62)
I noticed that I never see the port 5335 packet even though it shows up in the pi-hole query log
I am wondering if there should be an entry in the pi-hole's iptables to allow for the port 5335 traffic? Currently there is none just entries for port 53:
pi@inet-pi:~$ sudo iptables --list --line-number --numeric
Chain INPUT (policy ACCEPT)
num target prot opt source destination
Chain FORWARD (policy DROP)
num target prot opt source destination
1 DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0
2 DOCKER-ISOLATION-STAGE-1 all -- 0.0.0.0/0 0.0.0.0/0
3 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
4 DOCKER all -- 0.0.0.0/0 0.0.0.0/0
5 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
6 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
7 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
8 DOCKER all -- 0.0.0.0/0 0.0.0.0/0
9 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
10 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
11 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
12 DOCKER all -- 0.0.0.0/0 0.0.0.0/0
13 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
14 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
15 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
16 DOCKER all -- 0.0.0.0/0 0.0.0.0/0
17 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
18 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Chain DOCKER (4 references)
num target prot opt source destination
1 ACCEPT tcp -- 0.0.0.0/0 172.18.0.2 tcp dpt:9100
2 ACCEPT tcp -- 0.0.0.0/0 172.21.0.2 tcp dpt:443
3 ACCEPT tcp -- 0.0.0.0/0 172.18.0.3 tcp dpt:9115
4 ACCEPT tcp -- 0.0.0.0/0 172.21.0.2 tcp dpt:80
5 ACCEPT udp -- 0.0.0.0/0 172.21.0.2 udp dpt:67
6 ACCEPT tcp -- 0.0.0.0/0 172.18.0.5 tcp dpt:9090
7 ACCEPT tcp -- 0.0.0.0/0 172.21.0.2 tcp dpt:53
8 ACCEPT tcp -- 0.0.0.0/0 172.18.0.6 tcp dpt:9798
9 ACCEPT udp -- 0.0.0.0/0 172.21.0.2 udp dpt:53
10 ACCEPT tcp -- 0.0.0.0/0 172.18.0.4 tcp dpt:3000
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
num target prot opt source destination
1 DOCKER-ISOLATION-STAGE-2 all -- 0.0.0.0/0 0.0.0.0/0
2 DOCKER-ISOLATION-STAGE-2 all -- 0.0.0.0/0 0.0.0.0/0
3 DOCKER-ISOLATION-STAGE-2 all -- 0.0.0.0/0 0.0.0.0/0
4 DOCKER-ISOLATION-STAGE-2 all -- 0.0.0.0/0 0.0.0.0/0
5 RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (4 references)
num target prot opt source destination
1 DROP all -- 0.0.0.0/0 0.0.0.0/0
2 DROP all -- 0.0.0.0/0 0.0.0.0/0
3 DROP all -- 0.0.0.0/0 0.0.0.0/0
4 DROP all -- 0.0.0.0/0 0.0.0.0/0
5 RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
num target prot opt source destination
1 RETURN all -- 0.0.0.0/0 0.0.0.0/0
The other odd thing is that if I manually lookup something via dig pointing to unbound, it works!
pi@inet-pi:~$ dig @127.0.0.1 -p5335 www.google.com
; <<>> DiG 9.16.27-Debian <<>> @127.0.0.1 -p5335 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50429
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;www.google.com. IN A
;; ANSWER SECTION:
www.google.com. 300 IN A 142.250.73.228
;; Query time: 39 msec
;; SERVER: 127.0.0.1#5335(127.0.0.1)
;; WHEN: Sat Sep 10 07:48:20 EDT 2022
;; MSG SIZE rcvd: 59
So I'm still confused but convinced it's something on the pi and not in my network.