It would be cool if there was a feature where when FTL was doing gravity, adding or removing items from whitelist, blacklist, regex, etc. to have it make IP tables rules to forward DNS requests to the upstream servers until all tasks are done. Now I know I can setup primary and secondary DNS in my router, but it would be nice if we could at least get some resolution while it is doing intensive tasks. I don't know if the new api and version 5.0 could be built to handle something like this.
I have about 4 million domains being blocked, 50 regex rules, 20 items in the whitelist and few items on the blacklist. The web interface and DNS freezes when I have to update any of these for whatever reason.
This is one of the reasons Pi-Hole does the cron gravity update at a random time early Sunday morning (between 3 and 5 am), when the system is likely to be idle.
How often are you updating gravity? And when you do, how long does it typically take?
You can't set up a secondary DNS in the router, only a second one and which one is used is then up to the client so you can end up with a lot of queries bypassing the pi-hole.
My solution was to add another pi-hole and set both as DNS servers so if one is down or busy the other happily answers queries. When the other comes back up it takes a while but eventually it looks like the load automatically (by client rules) gets divided again.
Look at your memory and CPU use, you might get by with a Zero W but might be happier with a Pi 3.
Well the YouTube ads script runs every 15 minutes, and I have 70% memory usage with about 3 million domains now as OpenDns has me covered. This is all on a RPI 3B+. It takes about 1 minute and 15 seconds on average. Reboots every so often for stabilization. This is an internal-only Server and has approximately 210 Mbps of downstream bandwidth available to it. Would going the cloud route work better with a resolver on the cloud while internal is down? I could make a script or binary to update dynamic up address in iptables.
My Config:
1 GCP Pi-hole Ubuntu 16.04 LTS 512MB
1 Raspberry Pi 3
1 Raspberry Pi 3b+
Both pi’s are at separate physical locations and WAN
With the stats you mentioned I'd do another Pi 3b+ located along with your existing Pi-hole one and leave it at that. Worst case find the pi-hole update code and make sure they don't pick the same random moment. Maybe just lie to one Pi about the local time zone?
Trying to deal with a cloud based setup sure sounds like a lot more work than just another Pi, I wouldn't do it.
A script running where?
Did you notice that OpenDNS now offers filtered IP v6 servers, switched to them a few days back and they are doing a good job.
The script is from another thread under faqs and pertains to YouTube ad blocking. I’m pretty sure it defaults to 15 minutes. It compiles a block list from pihole.log and external sources. Can change that in cron to once a day at 1 am... Reboots not that often say maybe once every two weeks, etc to apply updates to raspbian and stuff.
Doing this every 15 minutes is putting a lot of load on your Pi and also on the servers on the other end.
This might not be the best time. The logs are rotated nightly at midnight, so the pihole.log only contains data back to midnight. Doing the script at 1 am gets one hour of data, while people are typically sleeping. Better to do this cron at 11:45 pm or so, to get a full day's data.
I agree with @Stan-qaz that your best solution is a second Pi-Hole running in parallel with the one you have. Set them up the same and list Pi1 as DNS1, Pi2 as DNS2. You improve reliability and have the opportunity to fiddle with a Pi-Hole without stopping your network DNS traffic.
Sounds like a great answer! I can definitely get another RPI and have that act as a backup. I could then modify my script from August that syncs settings to both of them from the master so that I don't have to go back and forth. I also could have one ssh into the other and execute a script that copies over whitelist, blacklist, regex, adlist sources, etc every hour or so (not large files) I can also slim down on the number of domains I have to further reduce how long it takes to do gravity.
Run Cron job at 11:45 PM to block YouTube ads
This is my current network setup:
Admin network: (192.168.1.0/24)
Wireless AP: 192.168.1.2
Printer: 192.168.1.3
Pi-hole A: 192.168.1.4 (current)
Pi-hole B: 192.168.1.5 (planned)
DNS Server: Google (Bypass network)
Wireless network: (192.168.2.235-254)
DNS Servers: Pi-holes
I would avoid that. It's just one more variable that will cause you problems in future troubleshooting of some problem or another (you'll forget you made the change).
When Pi-Hole sets up the cron script for gravity update on each individual install, it randomly picks a time on Sunday morning between 3 and 5 am. It is unlikely that your two installs will have the same time, but you can check them each and change the time as necessary.
sudo grep -v '#\|^$' /etc/cron.d/pihole will show you the un-commented lines. Look for the gravity line.
Here are two examples from two Pi-Holes I have running in parallel:
While adding or removing domains from whitelist, blacklist, regex does trigger a "soft" FTL restart (time pihole restartdns reload-lists), resolution should still work.
Updating the gravity.db also doesn't cause FTL to stop anymore as a second database is build and swapped in total at the end of the run.
