Poor DNS Cache utilisation after v5 upgrade

Help
1

Hi,

I've used a Pi-Hole now for a few years and am a huge fan! Unfortunately after upgrading to version 5, I have found that my cache usage has dramatically dropped.

Previously I would see only 20-40% of all DNS queries being forwarded on from the Pi-Hole. Now between 50-70% of all queries are forwarded.

I've read through the DNS cache - Pi-hole documentation article but see there's no reason to increase my cache size (I expect that the cache is just expiring records before they're requested again?).

My Pi-Hole is only used at home (11 devices), so I realise it will never see as many repeat DNS queries as you might on a busy network, but regardless there has been a dramatic drop in efficiency.

I'm running Pi-Hole on a Rasberry Pi 2b with Rasbian Lite OS. I have around 770,00 domains on the blocklist (most from EnergizedProtection's "Basic" collated list).
My system has no problems and appears to otherwise be working properly.

This all concerns me as the DNS server the Pi-Hole forwards to (on my Linux-based router) is using DoT to downstream DNS servers which are potentially high latency. I really want caching to be as efficient as possible.

Thank you very much in advance!

EDIT: Moved out of Help Community category as I have a totally standard install
EDIT2: Changed Cache efficiency to Cache Utilisation at Bucking Horn's suggestion

2

What benefits do you expect to gain by using an encrypted DNS server? In exchange for high latency, what do you get in return?

3

Pi-hole 5.0 contains an updated version of dnsmasq.
Going over the changes and improvements (specifically the verbose change log) that dnsmasq 2.81 incorporates (caching for TCP answers, CNAMEs, SRV records, among others), I 'd have to assume that -if anything- both cache utilisation and efficiency have improved.

Without dismissing your observation, I also cannot verify its fitness as evidence without additional data.

Ideally, you'd measure caching efficiency by observing a set of well defined queries and predetermined answers. In absence of that, you could resort to adequate sized statistical samples instead, as extractable from the long term database.

So far, your observation seems to be relying on data shown in Pi-hole's dashboard, and that may not be sufficient to substantiate your claim of "Poor DNS Cache efficiency".

For once, Pi-hole reports its forwarding and cache rate only via its main dashboard panel, and that is based on a rolling 24 hours time frame.
Depending on (an implied) periodic behaviour of your clients and your observation's point in time, some major fluctuations can very well be expected.

On the other hand, it is ultimately your clients behaviour that determine the amount of queries that will pass through Pi-hole, and TTL as provided by upstream DNS servers also has an impact on how long an entry will stay cached. These parameters are difficult to control, but you'd have to excercise a good degree of scrutiny to establish a baseline that you could check against to support your statement.

The only thing I can imagine to negatively impact cache rate is Conditional Forwarding:
In Pi-hole 5.0, CF has been improved to include reverse lookups for IPv6 in addition to IPv4. If you are using CF and if a significant portion of your DNS traffic would be those hourly reverse lookups, this could potentially increase forward rates, as those requests may not have happened before.

However, I would count that as an overall improvement, as you'd be be able to associate hostnames to IPv6 addresses better than before.

4

Thanks for your replies.

You are correct that I based my measurements only on the main dashboard panel. You're also correct that it's not a fully scientific comparison unless the same queries were made over the same time period before and after version 5. However regardless of this, it's significant how much the number of forwarded queries have changed.

Please believe me when I say that I've been watching the dashboard at least twice daily (morning and night) for about 7 days now. I'm doing this because I had a problem with some network devices querying DNS as fast as they could over and over and I want to ensure it doesn't happen again. (If you have a moment please read the first post in that link - it totally relates to DNS but the problem appears to be with the device firmware)
I'm making comparison with observations over years and I reported a 20% range in both cases as it does vary. While this is not a precise scientific measure there is a statistically significant difference between the two ranges.

Could you please confirm that no TTL values (or algorithms) have changed in version 5?

On my network no devices (aside from Smartphone OS's which I don't control) have IPv6 addresses (EDIT removed double-negative mistake). However I do get a small amount of IPv6 AAAA queries (less than ~3%). I've never noticed a high number of PTR queries - but believe they're usually under 5%

Things have have changed in my network:

  1. Pi-Hole upgrade to version 5
  2. Ubiquiti Unifi Access Point firmware update (see link above) for which the work around I found is now causing much less DNS queries. However these devices never used to query all that much anyway.
  3. Just before upgrading to version 5, I changed my DHCP options so that clients only query the Pi-Hole after reading one of your FAQs (or it may have been in Reddit). Before that the "secondary" DNS server was my linux-based Router running Unbound. (It is the upstream DNS server for Pi-Hole but itself hasn't changed for a long time.)
5

The latency isn't noticeably higher by the way. I'm being a perfectionist!
On this topic suggestion for new dashboard graph: average Pi-Hole query reply time!

You should research this - there's a reason Firefox now has DNS over HTTPs enabled by default and Microsoft is in the process of adding the same support to Windows. Here's a good blog on it (which does point out that it doesn't solve the problem DNSSEC is trying to address: authenticity.).

  • Short answer: Privacy and Security improvements
  • Longer answer: Avoid (some) tracking, avoid unwanted DNS filtering imposed by others (ISP/Government), prevent nasty ISPs playing with results (like redirecting your browser when a DNS query fails) and avoid targeted DNS spoofing attacks.

FYI: If you read my last post, "DNS Security" is why my Linux-based firewall is the downstream DNS server for my Pi-Hole. It's vastly inferior at the things the Pi-Hole is good at, but it does offer the choice of various encrypted DNS options. I chose DoT as it's (generally) the least impact of the encrypted DNS options (although I read there's situations where DoH can be better in a web browser due to the way it can reuse sessions.

6

Like I said before, I do not dismiss your observation.
I continue to question its fitness as evidence to support your claim (even if you'd change efficiency to utilisation, which feels like a more appropriate phrase to me).

I have already pointed you to the improvements that Pi-hole 5.0 and dnsmasq have incorporated, especially to the fact that TCP replies, CNAMEs and SRV are now being cached.

I could theorize that these changes should lead to an increased cache utilisation, but in the end, how these measures would impact specifically the cache rate you observe in your network depends on your clients' DNS queries entirely.

You've mentioned two aditional factors that may distort your observation one way or the other, i.e. having distributed a secondary name server besides Pi-hole until recently, and dealing with a high load of (illegal) queries from your Unifi APs (btw, 28,000 queries a day won't
overwhelm Pi-hole - a Zero can take that amount of queries an hour without breaking a sweat. UI may be struggling in presenting that amount of data when analysing Query Logs, though).

If you are determined to track this down: I fear you are the only one with all the necessary data to do so. It would be best to establish a baseline or at least to analyse your query patterns. Only then would you be able to correlate your observations with individual improvements introduced.

As for the DoT/DoH side of things: These have been discussed on our forums in detail repeatedly, just search for it.

7

Thanks for your help Bucking_Horn

I asked about changes partly as I'm unclear how much FTL varies from dnsmasq.

That's true. I have no reason to believe my DNS clients (other than the access points previously mentioned) have changed in behaviour in any way.

Ok, so what kind of cache utilisation would you expect from an "average" home network (if there were such a thing)?
I raised this thread as I have been surprised to see the Pi-Hole consistently forward more than 40% of queries - it was a surprise to me as it had always seemed very efficient.

I should add that the blocklist has probably peaked at ~20% under average conditions (our smartphones/devices have their own DNS blocking but with fewer rules), so it's not like the blocklist was causing there to be less forwarded queries.

Thanks. That's likely true, but it was causing Firefox on my smartphone to experience noticeable performance hits, even when it wasn't the active tab (but was loaded)!

Look, I only replied to that since jfb specifically asked. I noticed he was motivated enough to ask a tangential question, but not to address my problem! :wink:

9

The problem with encrypted DNS is that it provides a false assurance of privacy and security. As quoted in the first article "DNS lookups are sent to servers that can spy on your website browsing history".

Mozilla apparently believes that if they forward your DoH traffic to Cloudflare, this will no longer be the case. But, Cloudflare still gets to see your entire DNS history, so they haven't addressed the problem, they have simply chosen a DNS provider that they like.

Privacy - You send your DNS traffic through an encrypted tunnel, all hidden from your ISP. Then, once you have the IP in hand, you immediately (and in clear text) request that your ISP connect you to that IP. With very little effort, your ISP can figure out where you are browsing. And, the third party upstream DNS server has your DNS history, to do with what they wish.

Security - There is assurance that the DNS traffic between you and the upstream provider will not be altered by intermediaries, as it is contained in the encrypted tunnel. But, you have to trust the upstream DNS provider to provide you the correct information in the first place. You have no control over what they filter, re-direct, etc.

There is an alternative approach that provides equal security, better privacy, and puts you completely in control of your own DNS resolution (no filtering, no censoring, etc.). Run unbound (or BIND, or others) as a local recursive resolver.

Privacy - since the ISP sees all your IP requests anyway, there is no loss of privacy by letting them also see your DNS queries in progress (they are going to see the endpoint anyway). With your own recursive resolver running, you completely eliminate the third party DNS service, and communicate directly with the authoritative nameservers. Your privacy is increased with the upstream DNS service taken out of the loop.

Additionally, through the qname minimisation process, only the minimum amount of information is sent to each level of nameserver; enough to get you to the next level.

Security - With DNSSEC authentication (enabled by default in unbound), the replies from the authoritative nameservers are authenticated with an encryption algorithm, providing assurance that the reply has not been tampered with.

By running your own recursive resolver (not operating as a forwarder), you avoid DNS filtering imposed by others, nasty ISPs playing with results, and targeted DNS spoofing attacks. The only DNS queries going to your local resolver are from your own network via your Pi-hole

2 Likes
10

There have been no changes to TTL. For a TTL received from an upstream resolver, Pi-hole respects the TTL and passes this to the client and retains the query in cache for the duration of the TTL.

For locally answered queries (blocklist, hosts files, etc.), the TTL is unchanged from previous versions at 2 seconds. You can verify this in your own install:

cat /etc/dnsmasq.d/01-pihole.conf | grep ttl

As you noted, there is no way to predict the cache utilization for any user, as it is highly dependent on browsing habits and clients. One chatty client requesting the same domain at short intervals will be primarily served from cache.

As a one household point of comparision, here are the 24 hour cache stats from four Pi-holes on my home network, each serving a different subset of clients:

Most of the house including all the IOT devices - 30%

My computer only - 7%

My wife's computer only - 31%

A NAS and some IOS devices and a PC - 15%.

11

Thanks jfb.

Thanks for the cache stats - that's a helpful reference.

I'm currently getting 34% cache, 5% block for a combined network with the usual devices a tech person has at home. :wink:

I had expected this to be easy to solve as I'd seen such a dramatic change. Oh well, perhaps I just had a client query some domain repeatedly for a long period of time.

It may well have been the Ubiquiti unifi devices in the past artificially increasing the cache utilisation.

12

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.