Join us beta-testing Pi-hole FTL v5.9, Web v5.6 and Core v5.4

As always, please read through the changelog before updating to the beta versions.

Also ensure that you have either backed up your configuration or are willing to start from scratch in case things go wrong. Please use the Beta Category on our Discourse Forum to discuss the beta and/or report any findings. With warnings out of the way...

Highlights

  • Update embedded dnsmasq DNS server to version 2.86

  • Handle DHCPREBIND requests in the DHCPv6 server code

  • Fix bug which caused dnsmasq to lose track of processes forked to handle TCP DNS connections under heavy load

    The code checked that at least one free process table slot was available before listening on TCP sockets, but didn't take into account that more than one TCP connection could arrive, so that check was not sufficient to ensure that there would be slots for all new processes. It compounded this error by silently failing to store the process when it did run out of slots. Even when this bug is triggered, all the right things happen, and answers are still returned. Only under very exceptional circumstances, does the bug manifest itself: see https://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2021q2/014976.html

  • Major rewrite of the DNS server and domain handling code

    This should be largely transparent, but it drastically improves performance and reduces memory foot-print when configuring large numbers domains of the form local=/adserver.com/ or local=/adserver.com/# Lookup times now grow as log-to-base-2 of the number of domains, rather than greater than linearly, as before. The change makes multiple addresses associated with a domain work address=/example.com/1.2.3.4 address=/example.com/5.6.7.8

    It also handles multiple upstream servers for a domain better; using the same try/retry algorithms as non domain-specific servers. This also applies to DNSSEC-generated queries. Finally, some of the oldest and gnarliest code in dnsmasq has had a significant clean-up. It's far from perfect, but it is better.

  • Revise resource handling for number of concurrent DNS queries

    This used to have a global limit, but that has a problem when using different servers for different upstream domains. Queries which are routed by domain to an upstream server which is not responding will build up and trigger the limit, which breaks DNS service for all other domains which could be handled by other servers. The change is to make the limit per server-group, where a server group is the set of servers configured for a particular domain. In the common case, where only default servers are declared, there is no effective change.

  • Improve efficiency of DNSSEC

    The sharing point for DNSSEC RR data used to be when it entered the cache, having been validated. After that queries requiring the KEY or DS records would share the cached values. There is a common case in dual-stack hosts that queries for A and AAAA records for the same domain are made simultaneously. If required keys were not in the cache, this would result in two requests being sent upstream for the same key data (and all the subsequent chain-of-trust queries.) Now we combine these requests and elide the duplicates, resulting in fewer queries upstream and better performance. To keep a better handle on what's going on, the extra logging mode has been modified to associate queries and answers for DNSSEC queries in the same way as ordinary queries. The requesting address and port have been removed from DNSSEC logging lines, since this is no longer strictly defined.

  • Allow wildcards in dnsmasq config domain patterns

    Domain patterns in --address, --server and --local have, for many years, matched complete labels only, so --server=/google.com/1.2.3.4 will apply to google.com and www.google.com but NOT supergoogle.com. dnsmasq now introduces an optional * at the LHS of the domain string which changes this behaviour so as to include substring matches within labels. So, --server=/*google.com/1.2.3.4 applies to google.com and www.google.com AND supergoogle.com

  • FTL also imported the requested feature to Support Cisco Umbrella/OpenDNS Device ID Remote IP FTL#1096

  • Interface-dependent handling of pi.hole and hostname This makes FTL automatically reply with the appropriate IP address to both pi.hole and the machines hostname. Before this change, FTL always used a hard-coded address set during the weekly gravity updates (pihole -g). The new method is interface-aware and may reply with different addresses on different interfaces (e.g. Ethernet, WiFi or Wireguard network). The address FTL replies with can be overwritten using the REPLY_ADDR4/6 settings in /etc/pihole/pihole-FTL.conf.

  • Show automatically generated DNSSEC queries After each restart of pihole-FTL, there will be a high number of DNSSEC-related queries (DNSKEY and DS) as the DNSSEC chain of trust needs to build for all domains queried in your network. The number of queries will quickly drop afterwards when the chain has been primed with all the top-level domains you are typically visiting. You should then only rarely see a DS query when visiting an altogether new webpage. To ensure this new information is valuable for you, we will sketch how DNSSEC validation works in another upcoming blog post. If you dont want to see these queries, you can easily set SHOW_DNSSEC=false in /etc/pihole/pihole-FTL.conf to suppress analyzing them altogether (they will still be happening in the background).

  • Update embedded SQLite engine to version 3.36

    1. Improvement to the EXPLAIN QUERY PLAN output to make it easier to understand.
    2. Byte-order marks at the start of a token are skipped as if they were whitespace.
    3. An error is raised on any attempt to access the rowid of a VIEW or subquery. Formerly, the rowid of a VIEW would be indeterminate and often would be NULL.
    4. The memdb VFS now allows the same in-memory database to be shared among multiple database connections in the same process as long as the database name begins with /.
    5. Back out the EXISTS-to-IN optimization (item 8b in the SQLite 3.35.0 change log) as it was found to slow down queries more often than speed them up.
    6. Improve the constant-propagation optimization so that it works on non-join queries.
    7. The REGEXP extension is now included in CLI builds (use pihole-FTL sqlite in your terminal to access the embedded SQLite engine).
  • Enable .recover option for embedded SQLite engine Exemplary use to repair a corrupted long-term database: pihole-FTL /etc/pihole/pihole-FTL.db ".recover" | pihole-FTL ~/pihole-FTL_recovered.db On success, the old database can be replaced by the repaired one.

  • Other changes and bugfixes

    • Improve reliability under heavy TCP query load
    • Log when listening on the wildcard address. This will help debugging edge-case setups.
    • Fix crash when bind-address is used.
    • Fix for incorrect (but harmless) FATAL: Trying to access query ID -1 errors messages in pihole-FTL.log
    • Show hostnames also for clients specified by MAC address
    • Improve warning messages for defect hwclocks
    • Delay startup of FTL earlier to avoid database importing issues with incorrectly configured fake-clocks (if applicable)
    • Add a new theme that can automatically switch to dark mode if used on the device
    • Enable PHP8 support for Debian/Ubuntu
    • Be explicit in IPv6 RA values
    • Dont overwrite existing logrotate script

OK, enough reading.. how do I switch to the beta?

pihole checkout ftl release/v5.9 
pihole checkout core release/v5.4 
pihole checkout web release/v5.6

And, again, please use the "Beta" Category on our Discourse Forum to discuss the beta/report any findings. Well be there to give help and update the beta quickly in case you find any errors.

5 Likes

Thanks for all your help during beta testing. Everything has proven to be very stable over the last four weeks and we hope to have caught all the minor glitches as well.

What we're waiting for is the official release of dnsmasq v2.86 because we feel we should not release before dnsmasq is released, too. However, we are prepared for the next Pi-hole release at any time.

5 Likes

The first dnsmasq release candidate has been officially tagged. The absence of further bug reports (we reported one recently incl. the fix for it) is another promising sign that the release is coming closer.

1 Like

Thanks for your continued help in testing the Pi-hole beta and the dnsmasq release candidate.

Only recently, we were able to discover an issue with incorrectly handled replies when dnsmasq wasn't able to cache them. This finally lead to a series of three patches sent upstream to the dnsmasq project so everyone, also outside of Pi-hole, will be able to benefit from this involved testing. Working on the patches, Simon Kelley, the maintainer of dnsmasq found yet another long-standing bug with CNAME chains ending in a PTR record not being handled correctly. This lead to the second release candidate of the new dnsmasq being tagged.
Our great community is really something we can be proud of. And it pleases me to be part of it.

Further changes to the beta since we originally wrote the release notes you've seen above:

  • Fix dnsmasq server option interpretation (fix sent and accepted upstream)
  • Allow users to configure how FTL reacts to queries when the gravity database is not available (REPLY_WHEN_BUSY option)
  • Rate-limiting events are shown in the Pi-hole diagnosis system so you get notified on the web dashboard if any of your clients is rate-limited
  • Also display DNSSEC status of cache replies and internally generated DNSSEC-related queries
7 Likes