Many domains (including above) are failing to load entirely with a connection timeout issue. This behavior is being observed for multiple domains that are heavily traffic and popular.
Something seems to be rapidly triggering database rebuilds and that will cause blocks to be ignored while the database is being modified:
[2021-02-04 05:21:03.698 3754M] Received: Real-time signal 0 (34 -> 0)
[2021-02-04 05:21:04.850 3754/T3758] Compiled 6 whitelist and 6 blacklist regex filters for 2 clients in 3.5 msec
[2021-02-04 05:21:05.188 3754M] Received: Real-time signal 0 (34 -> 0)
[2021-02-04 05:21:05.292 3754/T3758] Compiled 5 whitelist and 6 blacklist regex filters for 2 clients in 3.5 msec
[2021-02-04 05:21:11.984 3754M] domain_in_list("download.redacted", 0x72d29f90, whitelist): Database is busy, assuming domain is NOT on list
[2021-02-04 05:21:11.984 3754M] domain_in_list("download.redacted", 0x72d29180, blacklist): Database is busy, assuming domain is NOT on list
[2021-02-04 05:21:11.984 3754M] domain_in_list("download.redacted", 0x72d2a8f0, gravity): Database is busy, assuming domain is NOT on list
[2021-02-04 05:21:11.985 3754M] domain_in_list("download.redacted", 0x72d29f90, whitelist): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:10.225 13302/T13306] Compiled 3 whitelist and 3 blacklist regex filters for 2 clients in 7.1 msec
[2021-02-04 09:03:10.966 13302M] Received: Real-time signal 0 (34 -> 0)
[2021-02-04 09:03:11.061 13302/T13306] Compiled 3 whitelist and 2 blacklist regex filters for 2 clients in 3.0 msec
[2021-02-04 09:03:12.855 13302M] domain_in_list("cloudsync-tw.synology.com", 0x72c2a0b0, whitelist): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.855 13302M] domain_in_list("cloudsync-tw.synology.com", 0x72c292a0, blacklist): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.855 13302M] domain_in_list("cloudsync-tw.synology.com", 0x72c2aa10, gravity): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.856 13302M] domain_in_list("cloudsync.cs.quickconnect.to", 0x72c2a0b0, whitelist): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.856 13302M] domain_in_list("cloudsync.cs.quickconnect.to", 0x72c292a0, blacklist): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.856 13302M] domain_in_list("cloudsync.cs.quickconnect.to", 0x72c2aa10, gravity): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.861 13302M] domain_in_list("cloudsync-tw.synology.com", 0x72c2a0b0, whitelist): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.861 13302M] domain_in_list("cloudsync-tw.synology.com", 0x72c292a0, blacklist): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.861 13302M] domain_in_list("cloudsync-tw.synology.com", 0x72c2aa10, gravity): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:13.182 13302M] Received: Real-time signal 0 (34 -> 0)
[2021-02-04 09:03:13.304 13302/T13306] Compiled 3 whitelist and 1 blacklist regex filters for 2 clients in 3.1 msec
[2021-02-04 09:03:14.255 13302M] Received: Real-time signal 0 (34 -> 0)
[2021-02-04 09:03:14.341 13302/T13306] Compiled 3 whitelist and 0 blacklist regex filters for 2 clients in 2.4 msec
Hey, nope, no scripts running or anything. I looked through some lower level traits. Looks like there were a mix of, both, buster and stretch references in apt sources. I think that messed some things up. Once I got that situated I tried a sudo apt full-upgrade and got pretty far before it reported the boot partition was not large enough for the new version and started rolling back.
TLDR; I started from a scratch Raspberry Pi OS Lite image and then followed this guide: Building A PiHole For Privacy and Security. Cranking on all cylinders and no performance issues.
This guide you referenced shows the author lost overview at some points in the process.
In addition to what @yubiuser wrote, also do not run system updates unattended in this way:
sudo apt-get update && sudo apt-get upgrade -y
This may cause all sorts of broken system issues. Sometimes, critical components get an update of their configuration files. -y mans "yes to all". This may instruct the services to either replace the config file (you modified locally) by the updated one or not updating the config files (means security improvements, etc. may be missing).
This is even more not needed at all, as the guide instructs you (a bit further up) to install unattended-upgrades. This thing is made exactly for this very purpose and knows how to handle such situations and what package updates it should rather skip (because they need user interaction).
The other apt-get calls may be problematic as well.
0 0 * * * sudo pihole restartdns
Is another very bad idea. Why should you restart your Pi-hole every night? This may take your Internet down for a short moment and it will (always!) destroy the entire DNS cache you have built up to this time. Also, it means a lot of work for the device. Pi-hole has been created to work reliably for years without interruption or restarts. There is absolutely no need to force restarts when this is not needed (e.g. due to configuration changes).
Yes, thank you for your input - I agree and prefer to update myself so I left out the crontab edits except the root list retrieval. I think the rest of the guide is pretty good though, no?
It is a close-by walkthrough, still there are some more strange bits in there. I just picked a few right now.
At the very least, uncomment this line:
#PermitRootLogin prohibit-password
Why? There is a misunderstanding here as well. Without this option, root is not permitted to log in at all. Setting this option actually weakens this by saying: Okay, root is permitted to log in. Just not via a password. It should generally be avoided. edit: This is outdated. The option defaults to yes by now.
Install Fail2Ban.
sudo apt install fail2ban
Fail2ban will block attackers IP if they fail to login after 5 failures for 10 minutes
What is missing is that fail2ban only works over IPv4. It does not work for IPv6.
Install log2ram.
If something does wrong and you are using log2ram, the logs will not survive a reboot making debugging really hard. I never use nor recommend it (because I don't think it'd needed). The guide should make this optional or, at least, say what the limitations are. In my support experience, it makes things a lot harder sometimes. If everything works well, and you have enough RAM memory, that is nothing against using it.
Also, looking again at the crontab recommendations, I realize that this add sudo to the commands even though these are already root's cron jobs. So sudo is superfluous.
The rest of the guide seems okay, but I'll admit that I haven't looked at everything in detail (mainly because it is so long). If you leave off the cron jobs and consider that fail2ban shouldn't be necessary at all inside your home network, this guide should work well.
Without the line it defaults to yes and root can login with a password. Uncommenting is safer, especially if you haven’t created the ssh keypair, which I haven’t.
Specifies whether root can log in using ssh(1). The argument must be “yes”, “without-password”, “forced-commands-only”, or "no”. The default is “yes”.
If this option is set to “without-password”, password authentication is disabled for root.
If this option is set to “forced-commands-only”, root login with public key authentication will be allowed, but only if the command option has been specified (which may be useful for taking remote backups even if root login is normally not allowed). All other authentication methods are disabled for root.
If this option is set to “no”, root is not allowed to log in.
Thus without-password allows root login only with public key authentication. This is often used with shell scripts and automated tasks.
Regarding fail2ban. I don’t have native ipv6 where I’m at so I disabled it completely in all areas. My ISP only does 6-to-4 which is deprecated and even unsupported by some servers, so no worries on that front.
ssh: This is interesting, it the default for this option changed, apparently. You live and learn, thanks. I'll slightly edit my posts above so readers won't get the wrong impression in the future.
fail2ban: It won't matter any way. It shouldn't be possible for anyone to penetrate your router's firewall. If they do, and they manage to conquer the router, they could still use (internal!) IPv6 from the router to the Pi-hole to circumvent IPv4. This is what I meant and I see I haven't been clear about this. Anyway, if they conquered your router than you are in big trouble.
i wrote that article. I've removed the crontab recommendations for apt, and there is a warning for pihole -up.
Note: The PiHole team does not recommend updating PiHole via cron jobs ( pihole -up ). Be aware that your server will update PiHole every Sunday via cron, and stay up-to-date on patch notes. If there is a major change, and you don’t want to update, sudo crontabe -e and comment out the line to update PiHole (place a # before the line.)
I've also updated the Fail2Ban install with the following:
Fail2ban will block attackers IP if they fail to login after 5 failures for 10 minutes. Note: Fail2Ban installed from the repo will only provide security on IPv4. If you want Fail2Ban to support IPv6, please look at this guide.
I've also updated the Log2ram section with this paragraph:
Log2ram is created for the Raspberry Pi. Since the Raspberry Pi uses a micro SD card for storage, constantly writing logs creates a lot of IOPS which can degrade the SD card. Log2ram creates a virtual /var/log/ directory in memory and synchronizes them back to the physical disk periodically. This reduces IOPS on the micro SD Card (if you’re logging DNS queries.) One complication is that logs stored in memory that do not get written to disk (because of a reboot for example) can make debugging an issue harder to track down. This is suggested for a PiHole because of how much logging the server is going to do, but be aware of the possible issues.
I appreciate all the criticism because it's only made the article better. unattended-upgrades is installed and configured, so including apt commands in cron was a bad idea, and I've removed them. I still think Fail2ban and a firewall is worthwhile, but I a) don't have IPv6 on my LAN and b) can only provide advice relevant to my habits and experience.
So I have a question. Is downloading root hints and configuring unbound to use them in the config, with a cron job to periodically fetch them preferred over having unbound use the dns-root-data package it manages itself?
I can see that the root hints I download are newer:
diff /usr/share/dns/root.hints /var/lib/unbound/root.hints
< ; last update: March 13, 2019
< ; related version of root zone: 2019031302
---
> ; last update: January 11, 2021
> ; related version of root zone: 2021011101
You don't have to update them often. To the best of my knowledge, the last change was adding an AAAA record for D but that was already back in 2016 (maybe even before). Even when one of them wouldn't be correct, many other root servers would still be available. I typically set up a cron job to run once a year.
I use the root servers themselves to fetch the information. You can select an arbitrary one, I typically select one close to the desired location of the unbound instance.