Odd local DNS resolution behavior only with new entries? New entries, explicitly new entries I add say a few minutes ago, resolve for reverse lookup, but not for forward lookup! Odd!
I added 4 new DNS entries to the custom.list file.
I have stop and restarted both DHCP and DNS, i.e. pihole -restartdns.
No errors or issues reported. I got to a Windows or another Linux system, query DNS for the 4 new entries, they resolve via ip address but NOT by name. Tried long and short name, nothing.
I try any other random valid DNS name, already registered, they resolve forward and reserve just fine, without issue.
I have built DNS servers, maintained them, so I am familiar with the basics of DNS resolution, but this has me puzzled. The pihole local DNS UI makes it just about impossible to have a forward or reverse domain mismatch that a typical BIND DNS server can have due to misconfiguration.
This only is applicable to local DNS entries. Thought that would be obvious but for the sake of completeness stating it explicitly.
Did a full reboot just for the sake of trying... no apparent benefit or impact. Running on a Pi 4, I think, 2GB RAM, if memory serves. No obvious issues from top results. Pi OS latest as of this date, fully update to date.
Please upload a debug log and post just the token URL that is generated after the log is uploaded by running the following command from the Pi-hole host terminal:
Well, I think I figured it out... for some reason the /etc/resolv.conf referenced the 'nameserver' entries in an order such that the local pihole server was listed 'last' after the DNS forwarders.
For example...
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 192.168.1.1
Where pihole server is 192.168.1.1
What is interesting is that this seems to break FTLDNS variant of dnsmasq? In theory as I understand it, when 8.8.8.8 and 8.8.4.4 fail to resolve a local DNS entry they should cascade to 192.168.1.1. But that seems to fail. Hence reverse lookup works, but forward lookup fails.
Once I changed the nameserver order in /etc/resolv.conf. The local forward lookup issues disappeared. Why, some worked, and others did not, and furthermore, only the most recent entries in the custom.list failed, not a clue.
Does running 'pihole restartdns' not refresh any DNS caching? It always seems to update DNS entries such that they appear in the pihole web page for DNS local correctly, and typically DNS queries resolve. Or should I have explicitly restarted pihole such that FTLDNS was forced to restart? Not sure that would have solved anything, since I did try a full reboot, and that failed to work.
That theory is wrong.
If given a set of DNS servers, a client may decide to use any of those for any given request at its own discretion.
For a specific request to a chosen DNS server, a client may only fallback to one of the others if that chosen one would fail to provide a reply, e.g. if there is no route to it, or if it would not respond in time due to heavy load.
In your above case, however, 8.8.4.4 would have replied with NXDOMAIN to a request for a local hostname, which is a valid reply.
Your debug log suggests that there is no DHCP server on the link that Pi-hole machine is connected to, so I can't check what DNS servers it distributes.
Run from a client in your network (i.e. not from the Pi-hole machine), please share the results of the following commands:
nslookup pi.hole
nslookup <plain. hostname>
nslookup <hostname.FQDN> 192.168.1.253
where you substitute names with actual names from your configuration.
DHCP server is on the same device, but does NOT update DNS by design. We maintain the DHCP reservations manually, i.e.in effect static assignment. DHCP server I checked, it is providing 192.168.1.253 only.
When I had a dedicated DNS server, I enabled forwarding for 8.8.8.8 and 8.8.4.4, so DNS queries only left our internal network if not resolved on 192.168.1.253. This seemed correct, and worked.
How can NXDOMAIN be a valid reply? And break any cascade to the next server? Is that just the RFC qualification or the design? OK, So when 'a' DNS response is successful the DNS query is complete, i.e. NXDOMAIN is considered a valid response, never cascades to the next DNS server, but it does cascade, because I can find www.google.com for example.
Does NXDOMAIN not get returned by 192.168.1.253? Or forwarding kicks the query past 192.168.1.253? Because, 192.168.1.253 is not authoritative?
What is odd (to me) is that the reverse lookup worked at all? Reverse lookup has no comparable NXDOMAIN result? So the reverse lookup cascades to each DNS server but the forward lookup does not? Again, you noted NXDOMAIN is a valid response at least specific to forward lookup.
That should have been pi.hole (unless you've redacted that as well), and it should have been resolved successfully.
That shows that the client you ran that command from has been using your Pi-hole at 192.168.1.253 for DNS, at least for that specific request.
In lieu of a DHCP server response in your debug log, if that request would have been sent to another server (by-passing Pi-hole), that would have been proof of that client being aware of other DNS servers.
Inversely, however, Pi-hole having been used for that one request does not preclude other DNS servers, so we have to rely on:
If I recall correctly, your debug log (which has expired by now) showed (at least) your Pi-hole host to lack public IPv6 connectivity, but nevertheless, you also want to check your router's IPv6 DNS configuration, to avoid that IPv6 clients may use an IPv6 DNS server address as advertised by your router to by-pass Pi-hole.
Particularly, note that your router may advertise its own link-local IPv6 as DNS server, allowing IPv6 clients to by-pass Pi-hole, even in absence of public IPv6 connectivity.
NXDOMAIN is a DNS resolver telling you 'I know that a record of the requested type does not exist in public DNS' rather than 'I don't know that, but someone else may, so go ask someone else'.
Since public DNS does not hold a record for the request, resending the same request to yet another DNS server would be pointless.
There is no cascade: As already mentioned, a DNS client would commonly only switch to another DNS server if the current one would be inaccessible or unresponsive.
Specifically, there is no cascade of 'if one server does not know the answer, go query the next'. Any public DNS resolver is expected to provide replies as available from public DNS.
That should also help you unravel your wrongly assumed cascading in:
There is no cascading: Any of the DNS resolvers from the list can answer that request for a public domain, so the nameserver order of appearance does not matter.
You even cannot expect that clients would prefer the first server from that list: Though they often tend to do so after a restart, some DNS clients may first probe all servers and then pick the fastest responding one; and over time, clients may demote or even discard non-responsive or inaccessible ones from their set of DNS servers, i.e. for a given request at a given time, a client may use any of the DNS servers it is aware of.
Specific behaviour, as well as its potential control, would depend on a client's specific OS and DNS client software configuration.
To repeat:
So when throwing your Pi-hole into the mix, you must make sure that Pi-hole is the only DNS server used by your clients, or they may by-pass Pi-hole, missing out both on its blocking as well as its local DNS knowledge (which is what happened to you before).
It should be obvious that if you would still provide multiple nameservers to clients, as in;
That would allow that respective client to by-pass Pi-hole.
Moving Pi-hole to the top of that list would at best just make that by-pass occur less often.
To work reliably, Pi-hole has to be the sole DNS server for your clients.
Probably unrelated to your observation:
You should be aware that by using a public domain (*.org), your Pi-hole would shadow public DNS records, i.e. it would prevent that a client would ever be able to receive the public DNS record if Pi-hole holds that requested DNS record.
This may have unwanted side effects where public DNS replies would be expected, e.g. in some scenarios when using Pi-hole remotely, or when trying to access *.org HTTPS sites that expect a certificate for a public IP.
And it would also mean that for records not shadowed by Pi-hole, a reply as defined in public DNS could be served.
In your case, as you have defined only A records for your local names, that may be relevant for IPv6 replies especially (but if I remember correctly, your debug log showed you did take measures against that).
Also, you have configured your Pi-hole to only hold A records for your *.org FQDN.
Note that while some software will automatically expand queries by your local domain (like nslookup does), not all clients will do so. For those that request just the plain hostname, your Pi-hole may yet return NXDOMAIN. You could consider to define plain hostnames in addition to your FQDNs (or instead of them, if your Pi-hole configuration is made aware of your local domain).
You haven't disclosed if you actually own that *.org domain or just use it for fancyness.
If you would not own it, in order to avoid the potential side-effects just mentioned above, then I'd recommend to switch to one of the domains intended for private use, such as .internal, .home.arpa or .lan.
Note that .local would not be a good choice here neither, as that is reserved for mDNS usage and should not be used with DNS.