Why is my gateway listed as an upstream server?

The issue I am facing:
Pi-hole is listing my gateway as an upstream server. I cannot tell if this means that some DNS lookups are skipping the Pi-hole or not.

Details about my system:
Gateway: Ubiquiti Cloud Gateway Max

  • DNAT rule (to force DNS requests to Pi-hole):
    • Protocol: TCP/UDP
    • Destination: 192.168.1.3 (match opposite)
    • Destination Port: 53
    • Translated IP Address: 192.168.1.3
  • Internet Primary WAN1 DNS: Cloudflare Zero Trust IPv4 addresses
  • Default network DNS: 192.168.1.3
  • all devices with Internet access are on Default network

Pi-hole:

  • static IP address: 192.168.1.3
  • nslookup indicates the address used is 127.0.0.1#53
  • not using DHCP (gateway handles)
  • set DNS server to 127.0.0.1 using sudo nmtui

What I have changed since installing Pi-hole:

  • installed cloudflared to use DoH with my Cloudflare Zero Trust endpoint
  • Custom DNS Servers: 127.0.0.1#5053
  • Multiple adlists and some custom domains
  • Some local DNS records
  • Disabled NTP

I neglected to mention that I do have conditional forwarding enabled, with the intention of network devices being named.

Please upload a debug log and post just the token URL that is generated after the log is uploaded by running the following command from the Pi-hole host terminal:

pihole -d

or if you run your Pi-hole as a Docker container:

docker exec -it <pihole-container-name-or-id> pihole -d

where you substitute <pihole-container-name-or-id> as required.

https://tricorder.pi-hole.net/tbgy3QAK/

You configured your gateway as reverse server in Conditional Forwarding:

     revServers = [
       "true,192.168.1.0/24,192.168.1.2",
       "true,192.168.2.0/24,192.168.1.2",
       "true,192.168.3.0/24,192.168.1.2"

The gateway is used as upstream for local lookups.

What is the definition of "local lookups"? It seems an extremely high usage rate (gateway: medina.station):

That's indeed unusual.

It seems to be caused by a flood of local service discovery lookups being forwarded to your router:

*** [ DIAGNOSING ]: Pi-hole log
-rw-r----- 1 pihole pihole 40M Mar 26 13:20 /var/log/pihole/pihole.log
   -----head of pihole.log------
   Mar 26 00:00:23 dnsmasq[590]: forwarded lb._dns-sd._udp.0.1.168.192.in-addr.arpa to 192.168.1.2
   Mar 26 00:00:23 dnsmasq[590]: query[PTR] lb._dns-sd._udp.0.2.168.192.in-addr.arpa from 192.168.1.142
   Mar 26 00:00:23 dnsmasq[590]: forwarded lb._dns-sd._udp.0.2.168.192.in-addr.arpa to 192.168.1.2
   Mar 26 00:00:23 dnsmasq[590]: query[PTR] b._dns-sd._udp.0.2.168.192.in-addr.arpa from 192.168.1.142
   Mar 26 00:00:23 dnsmasq[590]: forwarded b._dns-sd._udp.0.2.168.192.in-addr.arpa to 192.168.1.2
   Mar 26 00:00:23 dnsmasq[590]: query[PTR] db._dns-sd._udp.0.2.168.192.in-addr.arpa from 192.168.1.142
   Mar 26 00:00:23 dnsmasq[590]: forwarded db._dns-sd._udp.0.2.168.192.in-addr.arpa to 192.168.1.2
   Mar 26 00:00:23 dnsmasq[590]: query[PTR] b._dns-sd._udp.0.1.168.192.in-addr.arpa from 192.168.1.142
   Mar 26 00:00:23 dnsmasq[590]: forwarded b._dns-sd._udp.0.1.168.192.in-addr.arpa to 192.168.1.2
   Mar 26 00:00:23 dnsmasq[590]: query[PTR] db._dns-sd._udp.0.1.168.192.in-addr.arpa from 192.168.1.142
   Mar 26 00:00:23 dnsmasq[590]: forwarded db._dns-sd._udp.0.1.168.192.in-addr.arpa to 192.168.1.2
   Mar 26 00:00:23 dnsmasq[590]: query[PTR] lb._dns-sd._udp.0.1.168.192.in-addr.arpa from 192.168.1.142
   Mar 26 00:00:23 dnsmasq[590]: forwarded lb._dns-sd._udp.0.1.168.192.in-addr.arpa to 192.168.1.2
   Mar 26 00:00:23 dnsmasq[590]: query[PTR] 142.1.168.192.in-addr.arpa from 192.168.1.142
   Mar 26 00:00:23 dnsmasq[590]: forwarded 142.1.168.192.in-addr.arpa to 192.168.1.2

It seems your router would never answer those, which may trigger your 192.168.1.142 client to endlessly repeat its requests.

Let's ask your router directly, e.g.

nslookup lb._dns-sd._udp.0.1.168.192.in-addr.arpa 192.168.1.2

Unrelated to your issue, you are incorrectly trying to use CNAMEs with a public target domain (click for details)
-rw-r----- 1 pihole pihole 54K Mar 25 14:44 /etc/pihole/pihole.toml
   [dns]
     (…)
     cnameRecords = [
       "time1.aliyun.com,time.cloudflare.com",
       "time2.aliyun.com,time.cloudflare.com",
       "time3.aliyun.com,time.cloudflare.com",
       "pihole-cname.calebstauffer.com,cloudflare.com"
     ] ### CHANGED, default = []

As the description for Local CNAME records explains, this won't work reliably:

The target of a CNAME must be a domain that the Pi-hole already has in its cache or is authoritative for.
(…)
As consequence, if you set a target that isn't already known, the reply to the client may be incomplete.
(…)
Additionally, you can't CNAME external domains (bing.com to google.com) successfully as this could result in invalid SSL certificate errors when the target server does not serve content for the requested domain.

On laptop:

% nslookup lb._dns-sd._udp.0.1.168.192.in-addr.arpa 192.168.1.2

;; connection timed out; no servers could be reached

On SSH session to rPi with Pi-hole:

$ nslookup lb._dns-sd._udp.0.1.168.192.in-addr.arpa 192.168.1.2
;; communications error to 192.168.1.2#53: timed out
;; communications error to 192.168.1.2#53: timed out
;; communications error to 192.168.1.2#53: timed out
;; no servers could be reached

Is there a proper way to force a specific NTP server? I'd prefer all devives to use Cloudflare rather than Alibaba, or other (shady) providers.

No need to run that from different machines, results should not differ since we directly query your router at 192.168.1.2.

So your router indeed never answers those requests.

Would that be the case for regular reverse lookups as well?

nslookup 192.168.1.3 192.168.1.2

And what about hostnames that you'd expect your router to know?

nslookup <known.host> 192.168.1.2

Substitute <known.host> as required.


The most reliable way would be to configure your router's firewall to redirect all NTP/port 123 traffic to a DNS server of your choice. Or to your Pi-hole, and set your Pi-hole's NTP ntp.sync.server to your preferred NTP server.
That would also spare you from having to create CNAME records for any NTP server domain your clients are using.

If you can't do that, you probably could take advantage of a side-effect of using Pi-hole's new NTP server in conjunction with its new cache optimiser.
As quoted, a CNAME "must be a domain that the Pi-hole already has in its cache or is authoritative for.

If you'd set ntp.sync.server to time.cloudflare.com, Pi-hole would sync time once an hour and thus populate the cache.
By itself, that wouldn't be enough, since the TTL for that domain is 300 seconds, so the cache entry would disappear after 5 minutes, leaving your CNAMEs as incompletely resolved.
But Pi-hole v6's cache optimiser would keep the record cached for a grace period of an hour, and serve the stale record to clients with a shortened TTL while simultaneously refreshing the domain in the background.

This should allow the time server CNAMEs to almost always work, maybe apart from after a reboot or restart of Pi-hole.

You'll find ntp.sync.server under Settings | All settings » Network Time Sync.
All settings is only available in Expert mode.

% nslookup 192.168.1.3 192.168.1.2
;; connection timed out; no servers could be reached

Um, I wouldn't expect my router to know any host names... I intend for the Pi-hole to hande it all, but here it is:

% nslookup google.com 192.168.1.2
;; connection timed out; no servers could be reached

% nslookup pi.hole 192.168.1.2
;; connection timed out; no servers could be reached

From your debug log, you have other problems with DNS connectivity:

*** [ DIAGNOSING ]: Operating system
[i] Distro: Debian
[i] Version: 12
[✗] dig IPv4 return code: 9
[✗] dig response: ;; communications error to 205.251.193.151#53: timed out
[i] Retrying via IPv6
[✗] dig IPv6 return code: 9
[✗] dig response: ;; UDP setup with 2600:9000:5301:9700::1#53(2600:9000:5301:9700::1) for versions.pi-hole.net failed: network unreachable.
[✗] Error: dig command failed - Unable to check OS
[✗] Failed to resolve doubleclick.com via a remote, public DNS server (8.8.8.8)

You need to modify your firewall settings to allow Pi-hole to use a DNS other than itself.

-rw-r--r-- 1 root root 93 Mar 25 14:46 /etc/resolv.conf
   search localdomain
   nameserver 127.0.0.1
   nameserver 192.168.1.3

This is not the best setup. If the Pi is using Pi-hole for DNS, and the Pi-hole software is not working, the Pi cannot reach the internet to attempt a fix. Commands like uploading a debug log, running a Pi-hole repair, updating the time on the Pi, Pi-hole updates, etc, all need DNS resolution.

I recommend using a third party external DNS server for this - Cloudflare (1.1.1.1) would work nicely. You likely don't need ad-blocking on the Pi-hole host device.

Another option is to add the list of clients on the network to the /etc/hosts file on the Pi, and you won't need conditional forwarding (which seems to be causing you a DNS loop). Pi-hole will use the hosts file to determine client names. My DHCP server (the router) has a reserved IP for every client normally on my network (that's 70 clients). I copy that client list to each of five Pi-holes, and they all report the same names for any network client.

Example:

...
192.168.0.119	Steve
192.168.0.120	Apple-TV-MBR
192.168.0.121	Apple-TV-FR
192.168.0.125	Ipad-Pro-Mia
192.168.0.126	iPhone-12-Mia
...

Conditional Forwarding allows Pi-hole to forward resolution to another DNS server that holds DNS records Pi-hole itself doesn't, usually for locally defined domains.

If your router doesn't know any names, you can disable Conditional Forwarding, which in turn would make your router disappear from Pi-hole's upstreams.

From your debug log, you have other problems with DNS connectivity:

So what's wrong with the config? What change should I make? Remember Pi-hole is set to use cloudflared (installed on the same rPi), so there really shouldn't be a loop, unless I don't understand something. I don't have an issue with fixing Pi-hole manually.

Are you saying that the requests to the gateway are only for the names of devices on the network? I would expect that to occur infrequently, not be the largest upstream request segment. I've disabled conditional forwarding. Will wait and see if that drops the requests to the gateway.

Your OS on the machine running Pi-hole is using only Pi-hole for DNS:

Change one of the nameserver entries to use…

Note that resolv.conf is typically controlled by your OS network management tool, i.e. direct changes to that file itself likely won't persist. For permanent changes, you'd need to consult your OS's networking tool.

In your specific case, it could even be PTR lookups only.
But Windows fires those dns-sd type PTRs quite frequently (on some days, as often as 1,000 times with my Win11). And since your router does never answer them, Windows clients may just never get lazy to repeat them.

Disabling Conditional Forwarding won't make them disappear, but CF isn't doing anything beneficial in your case anyway, so it's advisable to disable it.

The intention of having the Pi-hole use itself for DNS is so that it uses cloudflared and my DoH endpoint. Is this not possible?

but CF isn't doing anything beneficial in your case anyway

Having the names of the devices is beneficial. I will try to export the list of network devices though, see if that works.

It is, but if…

Certainly, but your router doesn't know how to answer them, as the nslookup results and yourself show:

And as DNS requests to your router simply time out, that may suggest it is not even running an internal DNS server.

What make and model is your router?

What make and model is your router?

Certainly, but your router doesn't know how to answer them, as the nslookup results and yourself show:

I misunderstood what was meant by "host names". My gateway does have the names of the devices. Why would my gateway fail to answer?

It is, but if…

Sure, but that's my only option to ensure Cloudflare DoH is used. If there's an issue, then I expect I'll have to handle manually. It'd be nice to have it fallback gracefully, but AFAIK providing a second DNS server means it may go there even if the first is available.

It's quite possible to run with your nameservers, as long as you know how to handle yourself in case of difficulties, which it would seem you do. :wink:

If you want to ensure that DNS requests leave via DoH exclusively, did you consider to move your DoH proxy to your router, provided your router supports it?

Ubiquiti equipment often runs ISC's DHCP server, which doesn't populate a co-located DNS server by default.

I don't know any Ubiquiti Cloud line devices, but on some devices of the edge router series, ISC can be replaced by dnsmasq, which is both a DNS and DHCP server. Installing dnsmasq would have it answering DNS requests for local hostnames, allowing Conditional Forwarding to succeed.

Depending on client behaviour, you'd still see a fair portion of requests being answered by your router again.

I did not! And turns out, it does:
https://george.tsiokos.com/posts/2024/09/doh-unifi-cloudflare-zero-trust/!

So... now I set the DNS server for the rPi to the gateway's IP address, using sudo nmtui, correct?

(There are so many places to enter DNS servers in the Unifi software... I think I got it right. Ugh.)

From what I could tell, the IP address for time.cloudflare.com has been the same for years, so I could add it to /etc/hosts, and prevent that issue.

When I first updated to Pi-hole v6 I had issues with NTP so I turned it off, but I'll try again when it's safe.