pihole-FTL + unbound with Cache DB Module Options (Redis)

This topic is only relevant for users, running the compiled version of unbound. All files, required can be found on GitHub.

The unbound documentation has a section “Cache DB Module Options”.
This section covers the use of a Redis backend.

This setup requires unbound to be compiled with the options “with-libhiredis” and “enable-cachedb”, which requires the installation of libhiredis-dev (sudo apt-get install libhiredis-dev).

The script on GitHub (/home/pi/compile_unbound.sh) has all the necessary commands to install and configure the redis-server and compile unbound with the required options. Here is the reference to the Redis configuration changes, the script makes (sed commands). These changes are also recommended in the unbound documentation (the Redis server must be configured to limit the cache size, preferably with some kind of least-recently-used eviction policy).

As usual, the Raspbian version of Redis isn’t the latest (cuurent stable is 5.0.5), but since the server is for internal use only, I can live with that.

Finally, unbound needs to be configured to actually use the backend. This is achieved by the configuration file /etc/unbound/unbound.conf.d/redis.conf (see github).

After compiling unbound and making the necessary configuration changes, you can verify it all works, by looking at the Redis data, using a web interface. To add this web interface to your lighttpd configuration, simply run the script /home/pi/install_phpRedisAdmin.sh, restart lighttpd and browse to http://<pihole_IP_address>/phpRedisAdmin/

Result:

  • I can verify the Redis backend is actually receiving entries, by looking at the web interface. Although the data isn’t really readable, you can almost always identify the domain.
  • The unbound log doesn’t appear to indicate where the data (IP address) is coming from (cache, Redis backend or lookup). Any ideas here?

Screenshot of phpRedisAdmin

Please discuss pro / con of this setup, if interested.

1 Like

The documentation will explain.

This backend communicates with the specified Redis server over a TCP connection to store and retrieve cache data. It can be used as a persistent and/or shared cache backend. It should be noted that Unbound never removes data stored in the Redis server, even if some data have expired in terms of DNS TTL or the Redis server has cached too much data; if necessary the Redis server must be configured to limit the cache size, preferably with some kind of least-recently-used eviction policy. This backend uses synchronous communication with the Redis server based on the assumption that the communication is stable and sufficiently fast. The thread waiting for a response from the Redis server cannot handle other DNS queries. Although the backend has the ability to reconnect to the server when the connection is closed unexpectedly and there is a configurable timeout in case the server is overly slow or hangs up, these cases are assumed to be very rare. If connection close or timeout happens too often, Unbound will be effectively unusable with this backend. It's the administrator's responsibility to make the assumption hold.

So the database gets resolved domains and they stays there forever ignoring TTL. First cache consulted and then database and if not found or not in time, resolved upstream.

TRUE, with respect of the following (configured /changed with sed commands in /home/pi/compile_unbound.sh, result in /etc/redis/redis.conf) settings:

  • cache size (maxmemory 50M)
  • least-recently-used eviction policy (maxmemory-policy allkeys-lru)

as recommend in the unbound documentation (the Redis server must be configured to limit the cache size, preferably with some kind of least-recently-used eviction policy)

If that order is correct that would mean that incorrect results are highly likely, as dns updates might never make it into the cache. (EDIT: did a quick test with a 15min TTL on a temporary external subdomain, it was correctly updated an denoted as “OK (forwarded) SECURE” in pi-hole, so everything should be ok).

Been doing some benchmarking over the past day, using 31k domains with the folowing command: time dig -f domains.list +noall +answer > /dev/null
Using redis it takes roughly 28minutes and a few seconds to resolve them. Without redis this takes about 33,5 minutes.
While using singular queries it’s way to fast either way to leave the noise margin, so in the big picture there appears to be a benefit.

To eliminate the additional writes on your SD card (database save to SD), run the following:

sudo sed -i '/save 900 1/s/^/#/g' /etc/redis/redis.conf
sudo sed -i '/save 300 10/s/^/#/g' /etc/redis/redis.conf
sudo sed -i '/save 60 10000/s/^/#/g' /etc/redis/redis.conf
sudo service redis stop
sudo rm /var/lib/redis/dump.rdb
sudo service redis start

The Redis cache will of course be empty after a Redis or system restart…

To clear the cache without restarting Redis:

redis-cli FLUSHALL

How do i execute this script(compile_unbound.sh) using SSH connection?
It says password is empty. I can’t simply execute this command. Pls help me on how to execute it…

instructions on Raspbian buster September 2019 (I’m using WinSCP and putty to get things done - see installation manual, reference below):

  • you need to be able to access the internet, check with dig google.com, you need a valid result before you continue
  • unbound cannot be already on your system, remove it when it is (sudo apt-get remove unbound && sudo apt-get autoremove), or start from scratch (fresh install)
  • copy the script in /home/pi (using WinSCP or anything equivalent), ensure to copy the RAW content from github
  • make it executable (sudo chmod +x /home/pi/compile_unbound.sh)
  • execute with sudo (sudo /home/pi/compile_unbound.sh)

the script will run for several minutes, installs Redis and Unbound
you need to copy the configuration files from GitHub in /etc/unbound (and everything below) into /etc/unbound (this folder will exist after you compiled unbound)

  • restart unbound (sudo service unbound restart)
  • check if unbound runs (sudo service unbound status), press q to exit the status screen

need more help on how to make a stable, secure and managable pihole, read my installation manual here (does NOT contain any instructions for unbound)

1 Like

I am getting struck at the copying files to etc/unbound step. What to do? It’s not working properly

Assumption: you have managed to run /home/pi/compile_unbound.sh and both Unbound and Redis are installed on your system, using the default settings that come with the packages.

The GitHub repository has a directory structure, you need to duplicate this, and copy the files on your system.
If unbound compiled successfully, /etc/unbound already exists. You need to get the RAW content of the file /etc/unbound/unbound.conf and copy (use WinSCP) the content on your system (same location). This file tells unbound to use all of the configuration files in /etc/unbound/unbound.conf.d, a folder you need to create yourself on your system.

Using the same method, you used to copy the file /etc/unbound/unbound.conf, you now need to get the RAW content of all (4) files in /etc/unbound/unbound.conf.d and copy the content on your system.

WARNING: the content of /etc/unbound/unbound.conf and /etc/unbound/unbound.conf.d/unbound.conf is different, don’t copy the same content twice!!!

The file /etc/unbound/unbound.conf.d/unbound.conf contains IP addresses. You need to change them to work on your system. If you don’t know how to do that, simply:

  • remove the lines interface: 127.10.10.2@5552 and interface: fdaa:bbcc:ddee:2::5552@5552
  • uncomment the line #port: 5552 (remove the # sign) and change the number into 5353 (resulting line: port: 5353).
  • If you don’t have IPv6 on your system, change the line do-ip6: yes into do-ip6: no

The changed settings are the same as the settings, explained in the pihole documentation for unbound, this will allow you to follow the rest of that guide, to complete the pihole configuration.

After you’ve completed copying and changing the fhe configuration files, run unbound-checkconf to verify you haven’t made any mistakes.

If everything is OK, run sudo service unbound restart, this will start unbound with the new config, and check if unbound is really running (sudo service unbound status)

now complete the setup, following the instuctions from the pihole guide, starting at Start your local recursive server and test that it's operational (skip sudo service unbound start, we’ve already done that.

Suc6

1 Like

What is not clear to me: Why do you want to do this?

From unbound.conf:

When Unbound cannot find an answer to a query in its built-in in-memory cache, it consults the specified backend. If it finds a valid answer in the back- end, Unbound uses it to respond to the query without performing iterative DNS resolution. If Unbound cannot even find an answer in the backend, it resolves the query as usual, and stores the answer in the backend.

My interpretation of this: DNS records get stored in this new external database and do never expire (unless done manually).

What would be the drawback of achieving the same thing in Pi-hole (dnsmasq) or unbound without the Redis detour? This could easily be done by artificially forcing the TTL value to say weeks or months (?)
Yes, min-cache-ttl is limited to one hour in pihole-FTL (for good reasons!) but this limit can obviously be modified when you are fine with compiling locally anyways:

According to tests, performed by @GieltjE (not sure what method was used to perform the test), it makes the system faster.
If, NOT yet sure about this, once a key has been created in the Redis database, there would be no more outgoing DNS queries, thus limiting the number of DNS queries for any given record to one (1). This would limit any analysis of DNS request, it would show you visited a site, but not how often. A comment of @GieltjE on a dutch forum indicates his test showed outgoing DNS queries DO happen, despite the entry in the Redis database. If that is correct, it would indeed make the Redis approach an unnecessary detour, as you indicated.
No answer found yet, searching…, why would the unbound developpers create such a mechanism, if all it does is add unnecessary overhead. It probably is an advantage under certain conditions, which need yet to be discovered, the only reason, mentioned by @GieltjE would be increased performance (if the test is valid - no offence).

Maybe you can dig out a pull requests or some meaningful commit messages on unbound's side revealing their motivation?

This should already happen with the cache inside Pi-hole. Admittedly, if the TTL is too small, the domain could be re-queried if you visit the page once an hour or once a day or so, however, here you could artificially increase the min-cache-ttl value to achieve the very same thing, can’t you?

The only “drawback” I see is that Pi-hole’s cache does not survive restarts, however, even I – as the main FTL developer – do not restart my production Pi-hole more than once to twice a week. Typical users run FTL for months without interruption.
Querying a page twice a week does not tell my ISP (or whomever you try to hide from) much. Using unbound already spreads the knowledge across name servers. My feeling is that you should at least query a domain once a week to ensure that you don’t loose connectivity to this page.

@jpgpi250 I think a few things got lost in translation or I wasn’t clear.

With redis the correct TTL is honored so switchting to redis doesn’t bring any downsides compared to (just) pi-hole’s FTL (in fact the behaviour is exactly the same!). This was verified using the pi-hole’s interface and an external DNS server and a temporary subdomain with a short TTL.

On the tweakers.net forum I also mentioned the testing that was performed, which basically boils down to the pi-holes benchmarking page with unlimited domains which was filtered for unique items.
That resulted in a 31k long list which gave a 5 minutes advantage for redis (28min vs 33,5min without redis).
This might not be conclusive testing because this is a sequential test, but it does show a good gap between using redis or the built in FTL cache.
Also noted on the tweakers forum is the fact that singular queries without an extra induced load are too fast for benchmarking. But that even our two person household generate enough different entries (31k) to be relevant during consecutive queries.
Another side note presented is that on sd-cards it is best to disable the redis disk cache.

pihole-FTL is restarted, and thus clears it’s cache, at least once a week, ref your reply here (building the tree to the (weekly)), without user intervention. I have no idea if pihole-FTL is automatically restarted when adding a whitlist, blacklist or regex entries, using the web interface, you might want to clarify this.
I sometimes restart pihole-FTL, but never restart unbound, unless I run into serious problems or change the configuration, hence the idea to ‘favour’ the unbound caching methods over the pihole-FTL cache.
I’ts sunday 12h23, so pihole -g ran last night. I cleared all caches yesterday morning, trying to get some meaningfull figures. Currently I have 658 entries in the pihole-FTL cache and 784 entries in the Redis database, meaning unbound ‘knows’ more than pihole-FTL, which is the goal…
I am aware these figures don’t mean much yet, the system hasn’t been up long enough to draw conclusions.

It’s NOT my intention to argue or offence participants here, just want to know the pro/con/… of the solution

Blocked domains gets a TTL of two seconds wich could explain the lower figure in cache.

As far as I know, blocked domains requests never make it to unbound, they are handled and responded to directly by pihole-FTL. The Redis count would never increase if the domain is blocked.

That is correct but it leaves that Pihole cache obey TTL while Redis does not. Any timed out TTL are gone.

Remember that there is in Unbound and Knot a option to provide one time, timed out domains and at the same time resolve it upstream to have fresh information.

The only advantage of using Redis I see, if a big organisation wants it’s intranet fast and easy in the database or a ISP providing a caching resolver.
Having it’s own time frame to refresh information.

I’m sorry, but I still don’t see the need to add more moving gears (Redis). Even with

Simply move my idea from overwriting the minimum TTL (even though I still consider this a bad idea, but leave that aside for the moment) from Pi-hole into unbound if this is a point with less movement?

What speaks against artificially increasing the minimum TTL in unbound for your use case?

cache-min-ttl: <seconds>
Time to live minimum for RRsets and messages in the cache. Default is 0. If the minimum kicks in, the data is cached for longer than the domain owner intended, and thus less queries are made to look up the data. Zero makes sure the data in the cache is as the domain owner intended, higher values, especially more than an hour or so, can lead to trouble as the data in the cache does not match up with the actual data any more.


I understand this and I’m not trying to argue against you, I’m just not yet convinced that we can win anything with Redis being part of the game compared to using existing resources (unbound's own cache). In fact, I see many advantages of handling this in the DNS resolver’s own cache rather than in Redis, the only point against this, I’ve noticed so far is the survival of a restart (which doesn’t seem to happen often with unbound).

Gieltje (forum Tweakers.net) writes despite the entries in Redis, Unbound still send upstream as soon the TTL is expired.

This was not expected and it seems that Unbound keep track of the TTL.

Maybe unexpected, but by keeping track of the ttl in redis a lot of potentially bad cache is automatically discarded.