@jpgpi250 I think a few things got lost in translation or I wasn't clear.
With redis the correct TTL is honored so switchting to redis doesn't bring any downsides compared to (just) pi-hole's FTL (in fact the behaviour is exactly the same!). This was verified using the pi-hole's interface and an external DNS server and a temporary subdomain with a short TTL.
On the tweakers.net forum I also mentioned the testing that was performed, which basically boils down to the pi-holes benchmarking page with unlimited domains which was filtered for unique items.
That resulted in a 31k long list which gave a 5 minutes advantage for redis (28min vs 33,5min without redis).
This might not be conclusive testing because this is a sequential test, but it does show a good gap between using redis or the built in FTL cache.
Also noted on the tweakers forum is the fact that singular queries without an extra induced load are too fast for benchmarking. But that even our two person household generate enough different entries (31k) to be relevant during consecutive queries.
Another side note presented is that on sd-cards it is best to disable the redis disk cache.
pihole-FTL is restarted, and thus clears it's cache, at least once a week, ref your reply here (building the tree to the (weekly)), without user intervention. I have no idea if pihole-FTL is automatically restarted when adding a whitlist, blacklist or regex entries, using the web interface, you might want to clarify this.
I sometimes restart pihole-FTL, but never restart unbound, unless I run into serious problems or change the configuration, hence the idea to 'favour' the unbound caching methods over the pihole-FTL cache.
I'ts sunday 12h23, so pihole -g ran last night. I cleared all caches yesterday morning, trying to get some meaningfull figures. Currently I have 658 entries in the pihole-FTL cache and 784 entries in the Redis database, meaning unbound 'knows' more than pihole-FTL, which is the goal...
I am aware these figures don't mean much yet, the system hasn't been up long enough to draw conclusions.
It's NOT my intention to argue or offence participants here, just want to know the pro/con/… of the solution
As far as I know, blocked domains requests never make it to unbound, they are handled and responded to directly by pihole-FTL. The Redis count would never increase if the domain is blocked.
I'm sorry, but I still don't see the need to add more moving gears (Redis). Even with
Simply move my idea from overwriting the minimum TTL (even though I still consider this a bad idea, but leave that aside for the moment) from Pi-hole into unbound if this is a point with less movement?
What speaks against artificially increasing the minimum TTL in unbound for your use case?
cache-min-ttl: <seconds>
Time to live minimum for RRsets and messages in the cache. Default is 0. If the minimum kicks in, the data is cached for longer than the domain owner intended, and thus less queries are made to look up the data. Zero makes sure the data in the cache is as the domain owner intended, higher values, especially more than an hour or so, can lead to trouble as the data in the cache does not match up with the actual data any more.
I understand this and I'm not trying to argue against you, I'm just not yet convinced that we can win anything with Redis being part of the game compared to using existing resources (unbound's own cache). In fact, I see many advantages of handling this in the DNS resolver's own cache rather than in Redis, the only point against this, I've noticed so far is the survival of a restart (which doesn't seem to happen often with unbound).
Found a discussion here which might explain some things.
@GieltjE: On tweakers.net, you stated (translation) as soon as de Redis cache provides an expired record, a new query is performed. This appears to be correct, but the request is (see below) performed after the response is provided to the client, this to update the expired entry (see below - prefetch=yes).
I, and probably others, added prefetch=yes to my unbound config, as the result of a discussion (@DL6ER - Everyone of the team agreed to remove the lines) , where it was decided to remove cache-min-ttl and cache-max-ttl from the recommended unbound config, so the behavior @GieltjE has seen is probably because of the prefetch=yes setting.
The expired record will be served to the client only, if the setting serve-expired: yes is configured, as pointed out by @anon55913113.
So I changed my unbound config, ensuring the following settings are already there or added, and there are no cache-xxx-ttl settings in the config.:
Some questions remain, regarding the use of Redis, as pointed out by @DL6ER (I still don’t see the need to add more moving gears)
Restarted unbound, emptied the Redis cache (redis-cli FLUSHALL), so it's to soon to tell what the result will be, however, it remains fascinating and worth discussing...
Don't forget to decrease the verbosity or restart unbound, after you're done, or your unbound log will explode.
some entries found in the unbound log:
Oct 21 12:29:39 unbound[15130:0] debug: redis_lookup of BB4FC6A97D927FBBAC80E218 B91B9A126683A2B795378885CEF766EC85E81FDE
Oct 21 12:29:39 unbound[15130:0] debug: redis_lookup: no data cached
Oct 21 12:29:39 unbound[15130:0] debug: redis_store DC9B09CE18ABA23DB4FC71D86033A74F67B73D9AA106BECA6192C0512AE16079 (73 bytes)
Oct 21 12:29:39 unbound[15130:0] debug: redis_store set completed
Oct 21 12:34:23 unbound[15130:0] debug: redis_lookup of 8D825D767F185F94C89F48EC5C5A87BC3751CA17DEDE6940ABF62EEF2A5C209D
Oct 21 12:34:23 unbound[15130:0] debug: redis_lookup found 875 bytes
Oct 21 12:34:23 unbound[15130:0] info: redis ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 0
These are random entries (not related) from the log. Especially the last one proves Redis is really doing something, even on a system where unbound hasn't been up very long (see my last entry, regarding prefetch=yes)
Oh yes, but you see there also the reason for why it still wasn't changed - I put it on some paper and then forgot. Problem is that I'm traveling as well now. If you want to submit a docs PR I'd be happy approving and merging it.
Is this a also starting from an empty Redis database?
I'm repeating myself here, but I still don't see what Redis effectively brings into play with serve-expired: yes besides a possible restarting resistance.
Answering my own question here. Found this document on how to optimize unbound
On a raspberry pi 3B and 3B+, this would imply the correct setting on a 3B would be num-threads: 4
With the great help of @mibere (hank you for this), I've made some changes to my unbound config. This implies unbound needs to be compiled with an additional option: --with-libevent, which requires the installation of libevent-dev (sudo apt-get -y install libevent-dev).
I've already applied some of the proposed settings, not all (yet), and have updated my GitHub repository to reflect the changes.
Feel free to comment on / discuss the settings in these configuration files.
Not sure, but I don't expect Redis to have a great impact (if any), when these settings are not present. To verify this, you could increase the verbosity of unbound, and check the unbound log for entries containing redis_lookup found (sudo cat /etc/unbound/unbound.log | grep redis - use the correct location). It's a lot of logging (verbosity 5), so ensure you have sufficient disk space...
I'm still using my initial settings, evaluating this and keeping the proposed settings from @mibere in mind (see above).
wanted to share some experience with unbound and redis on my end. I was searching for a way to have a shared cache across multiple unbound instances. Out of the box there is no way to do this and also redis is not able to provide such a functionality. But fortunately there is an alternative: KeyDB. This tool is 100% Redis compatible and also provides active replication. With that, setting up KeyDB on 2 servers together with unbound, it is possible to have 2 DNS servers sharing same data and provide fallback mechanism in case, one of the servers crashes.
Additionally I was not very happy about the fact that unbound is not setting the expiry date on redis records although the ttl is known at that point of time where those are written to the database. Because of that I changed the unbound code slightly with the help in this thread:
My fix is not a final one and I am pretty sure it can be optimized, but with the fix it is not necessary any more to deal with eviction policies, as redis/keydb now automatically removes the expired records. For everybody interested in the code I have attached the changed files that need to be extracted to the subfolder "cachedb" of the unbound source.
The GitHub repository has been removed, since it contained outdated / misleading information. It's impossible to provide the settings you need to use, since a lot of them are hardware dependent. Read this document to find the proper settings for your system.
