Pi-hole FTL v5.8, Web v5.5 and Core v5.3 released

Originally published at: https://pi-hole.net/2021/04/14/pi-hole-ftl-v5-8-web-v5-5-and-core-v5-3-released/

As always, please read through the changelog before updating with pihole -up.

Highlights

More details for your adlists

The web dashboard does now provide health-status and statistics about downloaded and processed adlists. You can see when they were last downloaded, when they were last changed and it they work at all or contain invalid domains. This aims at simplifying using additional lists.

Add Pi-hole darker theme

This PR adds a third default theme, an even `darker` variant of the dark one. It also fixes an issues with the existing `dark` theme where links in box headers (like the Query Log’s “show all”) were completely invisible.

Comparison between the existing dark (on the left) and the new darker (on the right):

 

Automated IP blocking mode

Until now, FTL’s IP and IP-NODATA-AAAA blocking modes sourced the IP to deliver on a blocked domain from the setupVars.conf values IPV4_ADDRESS and IPV6_ADDRESS. This is, however, quite a limitation, especially if the device running Pi-hole has more than one interface or of the address is changing. To address this, FTL v5.8 implements an automated IP blocking mode. Instead of reading the addresses from setupVars.conf, we dynamically determine the address of the interface a query arrived on. We then use this IP address in the blocked reply. This does not only reduce maintenance (IPV4_ADDRESS and IPV6_ADDRESS can now be removed from setupVars.conf) but also localizes blocked queries. The automated detection can be overwritten using REPLY_ADDR4 and REPLY_ADDR6 in pihole-FTL.conf.

Update embedded dnsmasq to v2.85

New features, bug fixes and several other tweaks are contained in this release. Check out the change log below for more information.

FTL CHANGELOG

  • Automate IP blocking mode #965
  • Simplify handling of .lua and .db files #1086
    • pihole-FTL something.lua
      automatically launches the embedded LUA engine
    • pihole-FTL something.db
      behaves the same way as sqlite3 something.db
    • Similar things like
      pihole-FTL something.db "SELECT * FROM abc;"
      are possible as well
  • Add more regex warnings to message table #1092
  • Circle CI: skip uploading build artifacts on forks #1093 (thanks @bershanskiy)
  • Update SQLite to 3.35.4 #1083 #1089 #1097
  • Various enhancements and a few memory-leak fixes #1084
  • Resize shared memory only when locking #1072
    This is not really a functional change, however, it makes the code more read- and understandable in some places.
  • Escape DHCP options if necessary #1070
  • Ensure FTL can be compiled from static tarballs #1091
  • Use properly-sized buffer for format_time() #1088 (thanks @bershanskiy)
  • Fix pihole-FTL test not terminating properly (noticed in a docker environment) #1067
  • Fix incorrect “FATAL: Trying to access upstream ID -1” warning in the logs #1061
  • DNS server improvements:
    • Fix problem with DNS retries in earlier versions.
      The new logic in 2.83/2.84 which merges distinct requests for the same domain causes problems with clients which do retries as distinct requests (differing IDs and/or source ports.) The retries just get piggy-backed on the first, failed, request.
      The logic is now changed so that distinct requests for repeated queries still get merged into a single ID/source port, but they now always trigger a re-try upstream.
      To avoid excessive spamming of the upstream server, similar queries are collected for up to two seconds when the query has already been sent upstream. All clients will get replies when the answer comes back.
    • Avoid treating a dhcp-host which has an IPv6 address as eligible for use with DHCPv4 on the grounds that it has no address, and vice-versa.
    • Add dynamic-host option
      A and AAAA records which take their network part from the network of a local interface. Useful for routers with dynamically prefixes.
    • Teach bogus-nxdomain and ignore-address to take an IPv4 subnet.
    • Use random source ports where possible if source addresses/interfaces in use. CVE-2021-3448 applies.
      It’s possible to specify the source address or interface to be used when contacting upstream name servers: server=8.8.8.8@1.2.3.4 or server=8.8.8.8@1.2.3.4#66 or server=8.8.8.8@eth0, and all of these have, until now, used a single socket, bound to a fixed port. This was originally done to allow an error (non-existent interface, or non-local address) to be detected at start-up. This means that any upstream servers specified in such a way don’t use random source ports, and are more susceptible to cache-poisoning attacks.
      We now use random ports where possible, even when the source is specified, so server=8.8.8.8@1.2.3.4 or server=8.8.8.8@eth0 will use random source ports. server=8.8.8.8@1.2.3.4#66 or any use of query-port will use the explicitly configured port, and should only be done with understanding of the security implications. Note that this change changes non-existing interface, or non-local source address errors from fatal to run-time. The error will be logged and communication with the server not possible.
    • Change the method of allocation of random source ports for DNS. Previously, without min-port or max-port configured, dnsmasq would default to the compiled in defaults for those, which are 1024 and 65535. Now, when neither are configured, it defaults instead to the kernel’s ephemeral port range, which is typically 32768 to 60999 on Linux systems. This change eliminates the possibility that dnsmasq may be using a registered port > 1024 when a long-running daemon starts up and wishes to claim it. This change does likely slightly reduce the number of random ports and therefore the protection from reply spoofing. The older behavior can be restored using the min-port and max-port config switches should that be a concern.
    • Scale the size of the DNS random-port pool based on the value of the dns-forward-max configuration.
    • TFTP tweak: Check sender of all received packets, as specified in RFC 1350 para 4.

 

Web CHANGELOG

  • Add details to adlist table #1673 (@DL6ER)
  • Don’t count new status types as blocked queries in long-term data #1743 (@yubiuser)
  • Add hint for update command & documentation link #1749 (@j15e)
  • Add Pi-hole darker theme #1731 (@DL6ER)
  • Trim CNAME target input field value data #1759 (@Yrlish)

Core CHANGELOG

14 Likes

The settings page still relies on those values; without them it just displays unknown:

pihole -d will also still check if the interface's addresses match the values in setupVars.conf.

Thanks for checking. Something we will need to keep in mind for the work on the development-v6 branch

Greetings-
I just updated and I don't see the new adlist feature. Am I looking in the wrong place or do I need to activate something?

click on the icon before the url to expand

1 Like

Greetings-
Thank you!

6 posts were split to a new topic: PHP-8 packages not found

Updating my docker container to the latest release didn't work for me, Pi-hole wouldn't resolve any DNS requests and had this in the logs:

[✗] DNS service is NOT listening

Rolling back to the 5.7 docker image worked fine. The last two releases have both had non-working docker images on release requiring me to roll back.

Anyone else using docker? Did it work for you?

I've updated to the latest docker image without issue. Perhaps your specific configuration has some interesting settings?

Brought up the docker container on a two different PCs (one running Debian and the other Ubuntu) with no pre-existing configuration at all and a very simple docker-compose.yml file and get the same error:

pihole | [✗] DNS service is NOT listening
pihole | kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

docker-compose.yml

version: "3"

# More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
services:
  pihole:
    restart: always
    container_name: pihole
    image: pihole/pihole:latest
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "67:67/udp"
      - "80:80/tcp"
      - "443:443/tcp"
    environment:
      TZ: 'Europe/London'
      WEBPASSWORD: 'xxxxxxxxxx'
    # Volumes store your data between container upgrades
    volumes:
      - './etc-pihole/:/etc/pihole/'
      - './etc-dnsmasq.d/:/etc/dnsmasq.d/'
    # Recommended but not required (DHCP needs NET_ADMIN)
    #   https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
    cap_add:
      - NET_ADMIN
    restart: unless-stopped

Do the exact same thing using 5.7 and it starts up fine:

pihole | [✓] DNS service is listening
pihole | [✓] UDP (IPv4)
pihole | [✓] TCP (IPv4)
pihole | [✓] UDP (IPv6)
pihole | [✓] TCP (IPv6)

I stand corrected -- my docker logs pihole shows the same failure message that yours does.. but in my case, pihole is actually responding to DNS queries. So the logs indicate failure, but pihole is listening for DNS requests.

$ docker exec -ti pihole bash
root@pihole:/# grep :0035 /proc/net/tcp
   2: 00000000:0035 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 435453 1 0000000024bafaa7 100 0 0 10 5

(port 53 is 0x35)

OK I think I've gotten to the bottom of it. It's a race condition. Here's what startup looks like:

s6 cont-init.d/20-start.sh
-> /start.sh
    -> test_configs
      -> sudo -u ${DNSMASQ_USER:-root} pihole-FTL test || exit 1

According to the FTL src, "pihole-FTL test" performs a full startup, and then immediately exits. This is to confirm that the configs are valid. Then the s6 20-start.sh continues:

-> if gravity is *NOT* skipped on boot ($SKIPGRAVITYONBOOT), then:
   -> gravity.sh
      -> pihole status
         -> looks for something listening on port 53, if it's not found (or if it's not pihole):
            -> echo -e "  ${CROSS} DNS service is NOT listening"

Perhaps in pihole <= 5.7, pihole-FTL test would fork and remain running (and listening on :53) long enough for gravity.sh to complete, so it got a clean pihole status at the end. but now, the FTL test process is gone by the time gravity's status check looks for a listener on port 53. Hence the error output.

Then 20-start.sh continues:

-> # Kill dnsmasq because s6 won't like it if it's running when s6 services start
   kill -9 $(pgrep pihole-FTL) || true # TODO: REVISIT THIS SO AS TO NOT kill -9
   -> prints "kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]"

This prints the kill usage because pgrep didn't find a pihole-FTL process. More evidence that this startup script expected pihole-FTL test to still be running by the time we got here.

Thanks for digging into this.. please join us in the discussion on Github:

If you have deleted or commented out the IP addresses in the setupvars, pihole -g returns the following error message:
"No IP addresses found! Please run 'pihole -r' to reconfigure".

Managed to get pi-hole running. It was failing to resolve DNS requests and was hanging on startup at the 'DNS service is NOT listening' message but by doing the following I have managed to get it running, although it still reports that message in the logs it does actually run. Steps taken:

  • Stop v5.7 and download v5.8
  • Completely delete all my existing configuration
  • Start pi-hole container
  • Login to the UI and re-enter all my settings

EDIT: Actually ignore that. It worked briefly until a reboot and then it's stopped working again and hangs on startup. Back to v5.7 for me.

@Zelo Please check out

After a bit of digging around I've got v5.8 running in Docker. It still shows the error message in the logs on startup, but it does actually run and resolve DNS requests. There's a bit of detail required here, but it's something that could easily affect other people so it's worth going over.

My network has a gateway that hands out IP addresses, DNS settings etc via DHCP (pretty standard) so this is the DNS server for my network and it forwards on requests to Pi-hole.

I have a small low power PC that runs Pi-hole in a docker container. This got it's address from a static reservation in the DHCP server. So at this point you have the scenario where the Pi-hole system is using the DHCP assigned DNS servers which are in turn itself.

During startup of the docker container it is trying to do some queries to the internet (looking for newer versions etc) which cannot work because there is no DNS server running because Pi-hole is not running. This in turn stops the docker container from starting up (or at the very least stall for a very long time).

I resolved this by setting a static IP and DNS etc on the server running the docker container so there is a way for DNS requests to work on the startup of the docker container.

BUT - this setup worked fine prior to the release of v5.8, so I think the comment about the startup changing and previously 'pihole-ftl test' being left running is true and this was allowing the startup to run previously.

It can be argued that I should have used a static IP on the server before, but controlling IPs through a single place on the gateway with reservations is much simpler to manage and it did work previously.

Either the docker container should avoid doing queries to the internet on startup before it is able to serve DNS requests or this situation should be documented so people can avoid it.

I did the upgrade, and am not using docker, but several hours after the upgrade my setup went to "DNS service is NOT listening". I was able to get this "fixed" temporarily by changing my DNS settings and saving, but I have noticed lots of database failure messages in the FTL log, for example here:

s ON network_addresses.network_id = network.id WHERE network_addresses.ip = ? AND interface != 'N/A' AND interface IS NOT NULL;] data (11)
[2021-04-16 10:46:00.746 823/T827] SQLite3 message: database corruption at line 67162 of [5d4c65779d] (11)
[2021-04-16 10:46:00.746 823/T827] SQLite3 message: statement aborts at 3: [SELECT MAX(ID) FROM queries] database disk image is malformed (11)
[2021-04-16 10:46:00.746 823/T827] Encountered step error in get_max_query_ID(): database disk image is malformed
[2021-04-16 10:46:00.747 823/T827] Error while trying to close database: database is locked
[2021-04-16 10:46:00.747 823/T827] Encountered error while trying to store queries in long-term database: database is locked
[2021-04-16 10:46:00.767 823/T827] ERROR: SQL query "DELETE FROM network_addresses WHERE lastSeen < 1587033960;" failed: database is locked
[2021-04-16 10:46:00.768 823/T827] Error while trying to close database: database is locked
[2021-04-16 10:46:00.768 823/T827] Error while trying to close database: database is locked
[2021-04-16 10:46:37.966 823M] SQLite3 message: database corruption at line 81887 of [5d4c65779d] (11)
[2021-04-16 10:46:37.966 823M] SQLite3 message: statement aborts at 25: [SELECT hwaddr FROM network WHERE id = (SELECT network_id FROM network_addresses WHERE ip = ? GROUP BY ip HAVING max(lastSeen));] database disk image is malformed (11)
[2021-04-16 10:46:37.967 823M] SQLite3 message: database corruption at line 81887 of [5d4c65779d] (11)
[2021-04-16 10:46:37.967 823M] SQLite3 message: statement aborts at 9: [SELECT name FROM network_addresses WHERE name IS NOT NULL AND ip = ?;] database disk image is malformed (11)
[2021-04-16 10:46:37.968 823M] SQLite3 message: database corruption at line 81887 of [5d4c65779d] (11)
[2021-04-16 10:46:37.968 823M] SQLite3 message: statement aborts at 20: [SELECT name FROM network_addresses WHERE name IS NOT NULL AND network_id = (SELECT network_id FROM network_addresses WHERE ip = ?) ORDER BY lastSeen DESC LIMIT 1] database disk image i (11)
[2021-04-16 10:46:37.970 823M] SQLite3 message: database corruption at line 81887 of [5d4c65779d] (11)
[2021-04-16 10:46:37.970 823M] SQLite3 message: statement aborts at 10: [SELECT interface FROM network JOIN network_addresses ON network_addresses.network_id = network.id WHERE network_addresses.ip = ? AND interface != 'N/A' AND interface IS NOT NULL;] data (11)
[2021-04-16 10:47:00.846 823/T827] SQLite3 message: database corruption at line 67162 of [5d4c65779d] (11)
[2021-04-16 10:47:00.846 823/T827] SQLite3 message: statement aborts at 3: [SELECT MAX(ID) FROM queries] database disk image is malformed (11)
[2021-04-16 10:47:00.846 823/T827] Encountered step error in get_max_query_ID(): database disk image is malformed
[2021-04-16 10:47:00.846 823/T827] Error while trying to close database: database is locked
[2021-04-16 10:47:00.846 823/T827] Encountered error while trying to store queries in long-term database: database is locked
[2021-04-16 10:47:00.867 823/T827] ERROR: SQL query "DELETE FROM network_addresses WHERE lastSeen < 1587034020;" failed: database is locked
[2021-04-16 10:47:00.868 823/T827] Error while trying to close database: database is locked
[2021-04-16 10:47:00.868 823/T827] Error while trying to close database: database is locked

Please add it to pihole/pihole:v5.8 won't start · Issue #834 · pi-hole/docker-pi-hole · GitHub