I have just completed deploying redundant pi-hole DNS/DHCP servers in our environment and they are working quite well.
So that both servers will be able to respond to DNS requests for all clients with DHCP leases, I have scripts on each that regenerates /etc/pihole/custom.list from the other servers /etc/pihole/dhcp.leases. But this is being accomplished using cron which is a bit clunky and incurs some latency.
I was wondering if there was a "hook" in pihole I could take advantage of when the dhcp leases change rather than poling with cron every 5 minutes.
Came up with a satisfactory solution using systemd and inotifywait to monitor the /etc/pihole/dhcp.leases file. When it is modified (but not zero length), I munge it into "hosts" format using awk/sort. I compare the resulting file to the last one generated - simple renewals, which are the bulk of changes to the dhcp.leases file don't change resulting custom.list file. But if it has, I copy it to the other servers /etc/pihole/custom.list and reload piehole. The second server, of course, has a reciprocating service.
Updates are pretty nearly instantaneous. As soon as a client has a lease, it can be resolved on either server. Code follows:
# /lib/systemd/system/SyncDHCP.service
[Unit]
Description=Updates DHCP DNS between two pihole servers
[Service]
ExecStart=/usr/bin/SyncDHCP.sh
[Install]
WantedBy=multi-user.target
#! /usr/bin/bash
# /usr/bin/SyncDHCP.sh
Target=dns1 # Points at the server recieving custom.list
Src=/etc/pihole/dhcp.leases
Dst=$(mktemp)
Tmp=$(mktemp)
Msg(){
printf "%s - %s\n" $(date +%u%m%d%H%M%S) "$"
}
inotifywait -m $Src -e modify |
while read -r f a
do
size=$(cksum $Src | awk '{print $2}')
if [ "$size" -ne 0 ]
then
awk '$4!=""{printf "%s %s\n",$3,$4}' $Src | sort -k3 > $Tmp
if ! cmp -s $Dst $Tmp
then
diff $Dst $Tmp | awk '$1~/<|>/{
if ($1~/</)printf "released %s %s\n",$2,$3;
if ($1~/>/)printf "assigned %s %s\n",$2,$3;
}'
scp $Tmp root@$Target:/etc/pihole/custom.list > /dev/null &&
ssh root@$Target "pihole restartdns reload" &&
cp $Tmp $Dst &&
Msg "Updated custom.list on $Target"
fi
fi
done
2 Likes
Made some improvements to this service to allow stopping/restarting cleanly. Be sure to modify the "Target" section with your hostnames. And keep in mind, you need to install inotifywait and set up public key authentication for the root accounts between the two systems.
/lib/systemd/system/SyncDHCP.service
[Unit]
Description=Updates DHCP DNS between two pihole servers
After=network.target
[Service]
ExecStart=/usr/bin/SyncDHCP.sh Start
ExecStop=/usr/bin/SyncDHCP.sh Stop
ExecRestart=/usr/bin/SyncDHCP.sh Restart
[Install]
WantedBy=multi-user.target
/usr/bin/SyncDHCP.sh
#! /usr/bin/bash
Flag="/tmp/SyncDHCP.running"
Src="/etc/pihole/dhcp.leases"
function Msg {
printf "%s - %s\n" $(date +%u%m%d%H%M%S) "$*"
}
function Stop {
Msg "Stopping SyncDHCP service..."
rm -f $Flag
proc=$(ps -aux | awk '/inotifywait.*dhcp\.leases/{print $2}')
[ -n "$proc" ] && kill -HUP $proc
sleep 3 # Wait for filesystem cleanup
}
function Start {
Msg "Starting SyncDHCP service..."
Dst=$(mktemp)
Tmp=$(mktemp)
# Modify these target names to conform to your deployment
if [ $(hostname) == "dns1" ]
then
Target=dns2
else
Target=dns1
fi
# Create the "run flag file"
touch $Flag
while [ -e $Flag ]
do
inotifywait -e modify $Src | \
while read -r f a
do
size=$(cksum $Src | awk '{print $2}')
if [ "$size" -ne 0 ]
then
awk '$4!="*"{printf "%s %s\n",$3,$4}' $Src | sort -k3 > $Tmp
if ! cmp -s $Dst $Tmp
then
diff $Dst $Tmp | awk '$1~/<|>/{
if ($1~/</)printf "released %s %s\n",$2,$3;
if ($1~/>/)printf "assigned %s %s\n",$2,$3;
}'
scp $Tmp root@$Target:/etc/pihole/custom.list > /dev/null && \
ssh root@$Target "chmod 744 /etc/pihole/custom.list;pihole restartdns reload" && \
cp $Tmp $Dst && \
Msg "Updated custom.list on $Target"
fi
fi
done
done
rm -f $Dst
rm -f $Tmp
}
function Restart {
Stop
Start
}
case "$1" in
Start)
Start
;;
Stop)
Stop
;;
Restart)
Restart
;;
Status)
systemctl status SyncDHCP.service
;;
*)
echo "Usage: $0 {Start|Stop|Restart|Status}"
;;
esac
2 Likes
This could be tackled by DNS means, by simply configuring your two Pi-hole's DHCP and DNS options.
Configure your DHCP servers for a different Pi-hole domain name and a distinctive Range of IP addresses to hand out via Settings | DHCP, e.g. lan1 and lan2.
Then head to the bottom of Settings | DNS and enable Pi-hole's Conditional Forwarding and point requests for lan1 to your second Pi-hole and vice versa.
1 Like
Hmm, multiple local domain names is not desirable in our case. Are you certain that is required?
1 Like
Pi-hole will forward the DNS request based on that domain name.
If you'd use the same local domain on both Pi-holes, you'd have configured a partial DNS loop for unknown hostnames.
You may be able to short-circuit that loop by defining a client-specific blocking rule on just one(!) Pi-hole for local hostname requests from the respective other.
Try adding the following rule as RegEx filters:
^[^\.]+(\.local\.domain)?$
where local.domain is your local domain name.
In addition, you'd have to create a Group (e.g. 'prevent local loops'), then add your respective other Pi-hole as its sole Client and attach it to your 'prevent local loops' group.
1 Like
Yes, I thought I had tried this and wound up with DNS loops.
Our environment is a little involved: 6 local subnets, DHCP forwarding to the piholes is handle by our Cisco router, each subnet requires separate (but single) domain name AND search domains list for each lease. To top it off, we have a parent company to which I must forward requests for the corporate domains.
Our current method is working great and it is fast!
Even if get conditional forwarding working, aren't I going to have an extra forward for (potentially) half the systems on our local subnets?
Do you use the "Local DNS" functionality in the Pi-hole admin console? Or is everything in your environment a static DHCP lease that you manage this way? I tried out your script and it seems the script logic nukes the existing custom.list file, which is where local DNS records are stored (which are different compared to static DHCP leases).
I.e.,
awk '$4!="*"{printf "%s %s\n",$3,$4}' $Src | sort -k3 > $Tmp
Generates the $Tmp file from $Src, $Src being dhcp.leases. The diff is nice to show logging, but the $Tmp file goes on as is to overwrite the contents of custom.list, which is fine as long as you don't use the local DNS functionality (particularly relevant for static IPs that don't use DHCP at all).
I guess you could create static leases for hosts configured for a static IP; they'll never retrieve the lease but it doesn't matter as long as the lease and static IP are the same. That or tweak the script logic to not trample existing data, but the former seems easier. Just curious what you did in your environment to not trample existing, manually-configured DNS records.
We do not use the "Local DNS"; in our environment, custom.list is used only for the purpose of storing "the other" servers active leases. We DO have some static DHCP leases but we handle them with a .conf file under /etc/dnsmasq.d. Any purely static IPs we just put in /etc/hosts.
Thanks for the reply.
We DO have some static DHCP leases but we handle them with a .conf file under /etc/dnsmasq.d.
I assume you have to manually sync this .conf file on both Pi-holes; is that right? I take it something like gravity-sync isn't expecting your custom .conf file and won't keep it synced between both hosts as it would if it were looking at /etc/dnsmasq.d/04-pihole-static-dhcp.conf, which is updated via the webpage (and you're not using that).
Any purely static IPs we just put in /etc/hosts.
Same question here: /etc/hosts must be manually replicated on both Pi-holes, correct?
Bottom line is that your automation focuses solely on syncing DHCP leases. Static DHCP leases and static DNS records must be manually configured in /etc/dnsmasq.d/xx-custom.conf and /etc/hosts respectively. Let me know if I misunderstood something and thanks again.
Really wish at least Pi-hole would allow multi-subnet DHCP in the webpage at a minimum. Ideally, native, first-party sync capability for failover would be lovely. But I understand it's a consumer-type product. It's just that it's even gained enterprise adoption in some cases, which is awesome and crazy at the same time.
Out of curiosity, why not use the webpage for this in combination with gravity-sync? gravity-sync would handle replication of custom.list and 04-pihole-static-dhcp.conf.
Unrelated note:
It seems you have to make sure each Pi-hole does not have an overlapping pool, since you never transfer the dhcp.leases contents into the dhcp.leases file on the other Pi-hole; you're only moving it into custom.list. So in this scenario, and assuming two Pi-holes have the same pool they're serving from:
- New client comes online
- Pi-hole 2 responds and hands out a lease (192.168.1.2), also updating custom.list on Pi-hole 1
- A second, new client comes online
- Pi-hole 1 responds and, unaware that 192.168.1.2 has been handed out by Pi-hole 2, hands out a lease (192.168.1.2), also updating custom.list on Pi-hole 2
Now each custom.list on contains a row meant to address two separate hosts that are sharing a conflicting IP. I'm curious why you didn't try and synchronize dhcp.leases between the two servers.
1 Like
Not familiar with gravity-sync. SSH/SCP is native, works fine, and is useful for myriad other purposes. Once you have something that works, you move on to the next problem.
Correct: non-overlapping pools. Nothing new there. SOP since the 1990s (yes, I have been doing this that long, longer, actually).
2 Likes
I dealt with the overwriting of the custom.list file by saving a master copy called custom.list.permanent and then concatenating it to $Tmp just before the secure copy command to the other host.
Perm="/etc/pihole/custom.list.permanent"
cat $Perm >> $Tmp
Of course this means that I now must add and edit local DNS records directly in custom.list.permanent instead of through the pi-hole Web UI.
That's one way. And I'm glad someone is finding this useful!
But, since pihole utilizes custom.list and /etc/hosts equally effectively, one could "awk and cat" to /etc/hosts on the alternate server. This would free the custom.list to be used exclusively (as intended) by the web interface. I didn't do this in my implementation because I am "old school" and add all my statics directly to /etc/hosts - I had no use for custom.list. So it became the target for the dynamic address changes.
So far, my piholes are working great (and I have bigger fish to fry elsewhere). But if I need to come back to this, I will investigate appending the dynamics directly to a static block in /etc/hosts on the alternate server. Because it would be nice if that part of the web interface worked as intended.
1 Like