Vlan routing - route print from router (I didn't configure any of this, router just had it configured, I've never touched it). Looks like just a default route for vlan2 (eth0 / WAN port on router), and then a route to br0 for any local subnet traffic.
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
24.99.166.1 * 255.255.255.255 UH 0 0 0 vlan2
192.168.1.0 * 255.255.255.0 U 0 0 0 br0
24.99.166.0 * 255.255.254.0 U 0 0 0 vlan2
127.0.0.0 * 255.0.0.0 U 0 0 0 lo
default c-24-99-166-1.h 0.0.0.0 UG 0 0 0 vlan2
I'll try to attach the pcap from earlier for all port 53 traffic, ran from TCPDump on the pi. I didn't specify protocol (I don't think) just port - here's the command I used. tcpdump udp port 53 -w ~/dnsdump.pcap
That should capture TCP and UDP right? The pcap shows he only DNS coming from the pi was destined to Quad9 at that time - none shows up for OpenDNS IP's.
Still working on getting tcpdump on the router proving a bit more tricky that I thought. But I mean I don't think the router is messing with this right? When I point clients directly to the router for DNS, they use Cloudflare as their upstream resolver, verified by the 1.1.1.1/help page. I'll get the pcap either way though, curious to see it. Really curious to see the traffic coming in from the port the pi is on. Traffic leaving the pi is showing DNS to whatever upstream provider I pick, it would be really weird to see that same traffic come in to the router with a different DNS destination.
I think there's something going on with the routing. Why there is a bridge and a single vlan and no configuration done by you is odd. May want to ask around the merlin forums as well.
Looks like there are 2 vlans. Vlan1 has all the built-in switch ports (collectively br0), vlan 2 is for the WAN port (eth0).
I wouldn't expect a routing issue to only cause issues with DNS...I'd think I would be seeing way more issues. I'm currently waiting on a grep to finish on the pi searching the entire damn thing for any reference to 208.67.222.222 or 208.67.220.220, the OpenDNS IPs. Between that and getting tcpdump on the router it's going to be a bit. Plus I need to eat lol. I'll post an update once I've got more info, or if I just wipe the whole thing and reconfigure. Either way thank you for all the help on this, it's nice to have a second set of eyes on it all.
*** [ DIAGNOSING ]: Ports in use
*:445 smbd (IPv4)
*:139 smbd (IPv4)
It would seem you are running samba services along with Pi-hole.
Your Debug Log doesn't show smbd to fight for port 53, but just to be sure they don't interfere: Could you disable them?
There are routers that do block or intercept DNS traffic, though I would surprised if a Merlin variant would do so by default.
To see if something (your router or gateway) is intercepting DNS traffic, try the following nslookups from a client in your network (a client, so not your Pi-hole, not your router):
nslookup flurry.com
nslookup flurry.com 80.241.218.68
The first one should go through your default DNS server, which should be Pi-hole, while the second is forced explicitly through a public filtering DNS server.
Both commands should return 0.0.0.0 as address.
EDIT:
I may log off soon and miss your answer, so just in case:
If the first nslookup would return an actual IP, that would show that local DNS traffic is intercepted and redirected. The second nslookup result would do the same for outbound DNS traffic.
lookup results below, so looks like no interception by the router. I think the weirdest think we've uncovered so far is the tcp dumps from the pi and firewall logs from the router. TCPDump from the pi shows DNS going to an external resolver that is NOT OpenDNS (Quad9 in the pcap linked above). However the firewall logs on the router show DNS coming from the pi destined for the OpenDNS servers - 208.67.222.222 and 208.67.220.220. It's getting swapped around somewhere.
I ran a grep on the pi for any mention of the OpenDNS IP's, but it seems to be hanging. Any good way to search the ENTIRE filesystem for a string of text? In this case 208.67.222.222 and 208.67.220.220. I'm sure I can google it and find an answer but figured I through it out here in case anyone had a quick and easy way
Thanks for the nslookup test! I'll hold on to that one for future troubleshooting. Working on getting a pcap from the router itself, probably won't be until later this evening though.
Once I get some pcaps from the router / the grep finishes on the pi I'll post an update, most likely late this evening.
Thanks for the input from everyone, I'm pleasantly surprised at how much feedback I've gotten.
Final update for anyone still keeping an eye on this. Everything seems to be working fine now, reimaged the pi and started from scratch. Must have been some config hiding somewhere when I was messing around with other DNS and upstream DNS settings. Clients are pointing to pihole, pihole to Cloudflare, and all Cloudflare test sites report I'm using them.
Thanks again to everyone for the troubleshooting help on this!
proving a bit more tricky that I thought. But I mean I don't think the router is messing with this right? When I point clients directly to the router for DNS, they use Cloudflare as their upstream resolver, verified by the 1.1.1.1/help page. I'll get the pcap either way though, curious to see it. Really curious to see the traffic coming in from the port the pi is on. Traffic leaving the pi is showing DNS to whatever upstream provider I pick, it would be really weird to see that same traffic come in to the router with a different DNS destination.