Hardware: Raspberry Pi 3 Model B+ (1.4GHz Cortex-A53 with 1GB RAM)
Using this as base image/kernel/os: Release v1.0.18 · homebridge/homebridge-raspbian-image · GitHub
I'm expecting to be able to quickly resolve domains such as bestbuy.com and newegg.com.
Many domains (including above) are failing to load entirely with a connection timeout issue. This behavior is being observed for multiple domains that are heavily traffic and popular.
Subdomains for both domains mentioned by you frequently appear on blocklists, and are currently found on Steven Black's hosts as well.
If you still want to access those subdomains, you must allow them via an entry in Pi-hole's Whitelist.
In general, you can find out if and in which lists a domain occurs via Tools | Query Lists.
In addition, it may be worth taking a further look at
Hi there, thanks for the reply. Yes, I have checked with the query tool and there were no entries for Newegg and only the two following for Best Buy:
analytics.bestbuy.com
smetrics.bestbuy.com
bestbuy.demdex.net
ehg-bestbuy.hitbox.com
Zillow.com is also cause for concern regarding this issue
Something seems to be rapidly triggering database rebuilds and that will cause blocks to be ignored while the database is being modified:
[2021-02-04 05:21:03.698 3754M] Received: Real-time signal 0 (34 -> 0)
[2021-02-04 05:21:04.850 3754/T3758] Compiled 6 whitelist and 6 blacklist regex filters for 2 clients in 3.5 msec
[2021-02-04 05:21:05.188 3754M] Received: Real-time signal 0 (34 -> 0)
[2021-02-04 05:21:05.292 3754/T3758] Compiled 5 whitelist and 6 blacklist regex filters for 2 clients in 3.5 msec
[2021-02-04 05:21:11.984 3754M] domain_in_list("download.redacted", 0x72d29f90, whitelist): Database is busy, assuming domain is NOT on list
[2021-02-04 05:21:11.984 3754M] domain_in_list("download.redacted", 0x72d29180, blacklist): Database is busy, assuming domain is NOT on list
[2021-02-04 05:21:11.984 3754M] domain_in_list("download.redacted", 0x72d2a8f0, gravity): Database is busy, assuming domain is NOT on list
[2021-02-04 05:21:11.985 3754M] domain_in_list("download.redacted", 0x72d29f90, whitelist): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:10.225 13302/T13306] Compiled 3 whitelist and 3 blacklist regex filters for 2 clients in 7.1 msec
[2021-02-04 09:03:10.966 13302M] Received: Real-time signal 0 (34 -> 0)
[2021-02-04 09:03:11.061 13302/T13306] Compiled 3 whitelist and 2 blacklist regex filters for 2 clients in 3.0 msec
[2021-02-04 09:03:12.855 13302M] domain_in_list("cloudsync-tw.synology.com", 0x72c2a0b0, whitelist): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.855 13302M] domain_in_list("cloudsync-tw.synology.com", 0x72c292a0, blacklist): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.855 13302M] domain_in_list("cloudsync-tw.synology.com", 0x72c2aa10, gravity): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.856 13302M] domain_in_list("cloudsync.cs.quickconnect.to", 0x72c2a0b0, whitelist): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.856 13302M] domain_in_list("cloudsync.cs.quickconnect.to", 0x72c292a0, blacklist): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.856 13302M] domain_in_list("cloudsync.cs.quickconnect.to", 0x72c2aa10, gravity): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.861 13302M] domain_in_list("cloudsync-tw.synology.com", 0x72c2a0b0, whitelist): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.861 13302M] domain_in_list("cloudsync-tw.synology.com", 0x72c292a0, blacklist): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:12.861 13302M] domain_in_list("cloudsync-tw.synology.com", 0x72c2aa10, gravity): Database is busy, assuming domain is NOT on list
[2021-02-04 09:03:13.182 13302M] Received: Real-time signal 0 (34 -> 0)
[2021-02-04 09:03:13.304 13302/T13306] Compiled 3 whitelist and 1 blacklist regex filters for 2 clients in 3.1 msec
[2021-02-04 09:03:14.255 13302M] Received: Real-time signal 0 (34 -> 0)
[2021-02-04 09:03:14.341 13302/T13306] Compiled 3 whitelist and 0 blacklist regex filters for 2 clients in 2.4 msec
Hmmm. Nice find. Any commands I can issue to remedy this?
Are you running any third party scripts to update domains (whitelist, blacklist, regex) or adlists?
Hey, nope, no scripts running or anything. I looked through some lower level traits. Looks like there were a mix of, both, buster and stretch references in apt sources. I think that messed some things up. Once I got that situated I tried a sudo apt full-upgrade and got pretty far before it reported the boot partition was not large enough for the new version and started rolling back.
TLDR; I started from a scratch Raspberry Pi OS Lite image and then followed this guide: Building A PiHole For Privacy and Security. Cranking on all cylinders and no performance issues.
0 1 * * * sudo apt-get update && sudo apt-get upgrade -y
0 4 * * * sudo apt-get autoremove
40 4 * * * sudo apt-get autoclean
0 0 * * * sudo pihole restartdns
30 2 * * SUN sudo pihole -up
05 01 15 */3 * wget -O /var/lib/unbound/root.hints https://www.internic.net/domain/named.root
Don't do that. Do not automatically update Pi-hole, read the release notes and update manually.
This guide you referenced shows the author lost overview at some points in the process.
In addition to what @yubiuser wrote, also do not run system updates unattended in this way:
sudo apt-get update && sudo apt-get upgrade -y
This may cause all sorts of broken system issues. Sometimes, critical components get an update of their configuration files. -y mans "yes to all". This may instruct the services to either replace the config file (you modified locally) by the updated one or not updating the config files (means security improvements, etc. may be missing).
This is even more not needed at all, as the guide instructs you (a bit further up) to install unattended-upgrades. This thing is made exactly for this very purpose and knows how to handle such situations and what package updates it should rather skip (because they need user interaction).
The other apt-get calls may be problematic as well.
0 0 * * * sudo pihole restartdns
Is another very bad idea. Why should you restart your Pi-hole every night? This may take your Internet down for a short moment and it will (always!) destroy the entire DNS cache you have built up to this time. Also, it means a lot of work for the device. Pi-hole has been created to work reliably for years without interruption or restarts. There is absolutely no need to force restarts when this is not needed (e.g. due to configuration changes).
Yes, thank you for your input - I agree and prefer to update myself so I left out the crontab edits except the root list retrieval. I think the rest of the guide is pretty good though, no?
It is a close-by walkthrough, still there are some more strange bits in there. I just picked a few right now.
At the very least, uncomment this line:
#PermitRootLogin prohibit-password
Why? There is a misunderstanding here as well. Without this option, root is not permitted to log in at all. Setting this option actually weakens this by saying: Okay, root is permitted to log in. Just not via a password. It should generally be avoided.
edit: This is outdated. The option defaults to yes by now.
Install Fail2Ban.
sudo apt install fail2banFail2ban will block attackers IP if they fail to login after 5 failures for 10 minutes
What is missing is that fail2ban only works over IPv4. It does not work for IPv6.
Install log2ram.
If something does wrong and you are using log2ram, the logs will not survive a reboot making debugging really hard. I never use nor recommend it (because I don't think it'd needed). The guide should make this optional or, at least, say what the limitations are. In my support experience, it makes things a lot harder sometimes. If everything works well, and you have enough RAM memory, that is nothing against using it.
Also, looking again at the crontab recommendations, I realize that this add sudo to the commands even though these are already root's cron jobs. So sudo is superfluous.
The rest of the guide seems okay, but I'll admit that I haven't looked at everything in detail (mainly because it is so long). If you leave off the cron jobs and consider that fail2ban shouldn't be necessary at all inside your home network, this guide should work well.
I disagree. That means only a ssh keyfile can be used to log in, which is pretty safe.
Without this line, rootisn't allowed to log in at all though. Which is even safer.
edit: See below, this was incorrect.
Without the line it defaults to yes and root can login with a password. Uncommenting is safer, especially if you haven’t created the ssh keypair, which I haven’t.
From the manpage:
PermitRootLogin
Specifies whether root can log in using ssh(1). The argument must be “yes”, “without-password”, “forced-commands-only”, or "no”. The default is “yes”.
If this option is set to “without-password”, password authentication is disabled for root.
If this option is set to “forced-commands-only”, root login with public key authentication will be allowed, but only if the
commandoption has been specified (which may be useful for taking remote backups even if root login is normally not allowed). All other authentication methods are disabled for root.If this option is set to “no”, root is not allowed to log in.
Thus
without-passwordallows root login only with public key authentication. This is often used with shell scripts and automated tasks.
Regarding fail2ban. I don’t have native ipv6 where I’m at so I disabled it completely in all areas. My ISP only does 6-to-4 which is deprecated and even unsupported by some servers, so no worries on that front.
ssh: This is interesting, it the default for this option changed, apparently. You live and learn, thanks. I'll slightly edit my posts above so readers won't get the wrong impression in the future.
fail2ban: It won't matter any way. It shouldn't be possible for anyone to penetrate your router's firewall. If they do, and they manage to conquer the router, they could still use (internal!) IPv6 from the router to the Pi-hole to circumvent IPv4. This is what I meant and I see I haven't been clear about this. Anyway, if they conquered your router than you are in big trouble.
Hello,
i wrote that article. I've removed the crontab recommendations for apt, and there is a warning for pihole -up.
Note: The PiHole team does not recommend updating PiHole via cron jobs (
pihole -up). Be aware that your server will update PiHole every Sunday via cron, and stay up-to-date on patch notes. If there is a major change, and you don’t want to update,sudo crontabe -eand comment out the line to update PiHole (place a # before the line.)
I've also updated the Fail2Ban install with the following:
Fail2ban will block attackers IP if they fail to login after 5 failures for 10 minutes. Note: Fail2Ban installed from the repo will only provide security on IPv4. If you want Fail2Ban to support IPv6, please look at this guide.
I've also updated the Log2ram section with this paragraph:
Log2ram is created for the Raspberry Pi. Since the Raspberry Pi uses a micro SD card for storage, constantly writing logs creates a lot of IOPS which can degrade the SD card. Log2ram creates a virtual /var/log/ directory in memory and synchronizes them back to the physical disk periodically. This reduces IOPS on the micro SD Card (if you’re logging DNS queries.) One complication is that logs stored in memory that do not get written to disk (because of a reboot for example) can make debugging an issue harder to track down. This is suggested for a PiHole because of how much logging the server is going to do, but be aware of the possible issues.
I appreciate all the criticism because it's only made the article better. unattended-upgrades is installed and configured, so including apt commands in cron was a bad idea, and I've removed them. I still think Fail2ban and a firewall is worthwhile, but I a) don't have IPv6 on my LAN and b) can only provide advice relevant to my habits and experience.
So I have a question. Is downloading root hints and configuring unbound to use them in the config, with a cron job to periodically fetch them preferred over having unbound use the dns-root-data package it manages itself?
I can see that the root hints I download are newer:
diff /usr/share/dns/root.hints /var/lib/unbound/root.hints
< ; last update: March 13, 2019
< ; related version of root zone: 2019031302
---
> ; last update: January 11, 2021
> ; related version of root zone: 2021011101
If you want the latest root hints, install them yourself. OS packages lag quite a bit.
You don't have to update them often. To the best of my knowledge, the last change was adding an AAAA record for D but that was already back in 2016 (maybe even before). Even when one of them wouldn't be correct, many other root servers would still be available. I typically set up a cron job to run once a year.
I use the root servers themselves to fetch the information. You can select an arbitrary one, I typically select one close to the desired location of the unbound instance.
Something like
dig @199.7.83.42 . ns > root.hints
does the job for me at home.
It is just a bit more advanced that the arbitrary user may need so we left the guide more generic on this.
That makes sense to me. Thanks for the informed reply. I noticed in my diff the only difference was the date info, not the data.
This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.
