Running 2 Pi-Holes (DNS and DHCP) in sync with DHCP failover

Help
1

Folks,

I have successfully set up my 2 piholes for DNS (staying in sync) as well as DHCP on primary with failover to the secondary. There isn't much sense trying to run Pihole truely HA given how DNS and DHCP work...

Since I had to stitch things together, I figured I would share how I did it.

Credit to the folks who I gathered scripts, etc from is given at the top of the scripts

First I modified a script I found on reddit to keep all the Pihole files AND DHCP files in sync from primary to secondary. I run it as root, so I did not need the sudo in the original script.

#!/bin/bash -x
#  README
:'
Version 2.2
-----------------------------
zeit0dn1 modified to add sync'ing DHCP files.  Another script monitors and enables/disables DHCP on the secondary pihole.

Credit to https://www.reddit.com/r/pihole/comments/9hi5ls/dual_pihole_sync_20/
-----------------------------
Credit to redditor /u/jvinch76  https://www.reddit.com/user/jvinch76 for creating the basis for this modification.
-----------------------------
Original Source https://www.reddit.com/r/pihole/comments/9gw6hx/sync_two_piholes_bash_script/
Previous Pastebin https://pastebin.com/KFzg7Uhi
-----------------------------
Reddit link https://www.reddit.com/r/pihole/comments/9hi5ls/dual_pihole_sync_20/
-----------------------------
Improvements:  check for existence of files before rsync and skip if not present, allow for remote command to be run without password by adding ssh keys to remote host no no longer require hard coding password in this script, HAPASS removed
-----------------------------

I had been thinking of a script like his to keep my primary and secondary pihole in sync, but could not find the motivation to create it.
/u/jvinch76 did the heavy lifting and I made changes I hope you find useful.

I modified the code to increase the frequency of the sync to every 5 minutes and reduce the file writes by using rsync to compare the files and only transfer changes.
Furthermore, gravity will be updated and services restarted only if files are modified and a sync occurs.

I am unsure of the performance cost, but it is likely there is a trade-off with rsync being more cpu heavy, but this script reduces the disk write to minimal amounts if no sync is necessary.

Why run dual piholes?
If you are not, you really, really should be.  If the primary pihole is being updated, undergoing maintenance, running a backup, or simply failed you will not have a backup pihole available.
This will happen on your network.  Your only other option during an outage (usually unexpected) is to configure your DHCP server to forward to a non-pihole, public DNS, thusly defeating why you have pihole installed in the first place.
Furthermore, DNS is high availability by design and the secondary\tertiary DNS always receives some portion of the DNS traffic and if configured with a public DNS IP, your devices will be bypassing the safety of pihole blocking.
If you are running a single pihole and have that pihole listed as the only DNS entry in your DHCP setting, all devices on your network will immediately be unable to resolve DNS if that pihole goes offline.
I recommend running a PI3 as your primary and a PI3/PI2/ZeroW as your secondary.  PI2/ZeroW is more than sufficient as a secondary and emergency failover.

What about using my pihole for DHCP?
I still prefer to use my router for DHCP, if you need help refer to /u/jvinch76 post https://www.reddit.com/r/pihole/comments/9gw6hx/sync_two_piholes_bash_script/
or other docs about using pihole for DHCP with this script.

/u/LandlordTiberius

'

# INSTALLATION STEPS ON PRIMARY PIHOLE
: '
1. Login to pihole
2. type "SUDO NANO ~/piholesync.rsync.sh" to create file
3. cut and paste all information in this code snippet
4. edit PIHOLE2 and HAUSER to match your SECONDARY pihole settings
5. save and exit
6. type "chmod +x ~/piholesync.rsync.sh" to make file executable

# CREATE SSH file transfer permissions
7. type "ssh-keygen"
8. type "ssh-copy-id root@192.168.1.3" <- type the same HAUSER and IP as PIHOLE2, this IP is specific to your network, 192.168.1.3 is an example only
9. type "yes" - YOU MUST TYPE "yes", not "y"
10. type the password of your secondary pihole

# ENABLE REMOTE COMMANDS USING SSH Keys ON Remote pihole
11  type "cd ~/.ssh"
12. type "eval `ssh-agent`" <- this step may not be needed, depending upon what is running on your primary pihole
13. type "ssh-add id_rsa.pub"
14. type "scp id_rsa.pub root@192.168.1.3:~/.ssh/"
15. login to secondary pihole (PIHOLE2) by typing "ssh root@192.168.1.3"
16. type "cd ~/.ssh"
17. type "cat id_rsa.pub >> authorized_keys"
18. type "exit"
# see https://www.dotkam.com/2009/03/10/run-commands-remotely-via-ssh-with-no-password/ for further information on running ssh commands remotely without a password.

# INSTALL CRON Job
19. type "crontab -e"
20. scroll to the bottom of the editor, and on a new blank line,
21. type "*/5 * * * * /bin/bash /root/piholesync.rsync.sh" <- this will run rsync every 5 minutes, edit per your preferences\tolerence, see https://crontab.guru/every-5-minutes for help
22. save and exit

# DONE
'


#VARS
FILES=(black.list blacklist.txt regex.list whitelist.txt mydomain.lan.list dhcp.leases) #list of files you want to sync
PIHOLEDIR=/etc/pihole #working dir of pihole
PIHOLE2=192.168.1.99 #IP of 2nd PiHole
HAUSER=root #user of second pihole

#LOOP FOR FILE TRANSFER
RESTART=0 # flag determine if service restart is needed
for FILE in ${FILES[@]}
do
  if [[ -f $PIHOLEDIR/$FILE ]]; then
  RSYNC_COMMAND=$(rsync -ai $PIHOLEDIR/$FILE $HAUSER@$PIHOLE2:$PIHOLEDIR)
    if [[ -n "${RSYNC_COMMAND}" ]]; then
      # rsync copied changes
      RESTART=1 # restart flagged
     # else
       # no changes
     fi
  # else
    # file does not exist, skipping
  fi
done


FILE="adlists.list"
RSYNC_COMMAND=$(rsync -ai $PIHOLEDIR/$FILE $HAUSER@$PIHOLE2:$PIHOLEDIR)
if [[ -n "${RSYNC_COMMAND}" ]]; then
  # rsync copied changes, update GRAVITY
  ssh $HAUSER@$PIHOLE2 "pihole -g"
# else
  # no changes
fi


#DHCP Files

FILE="/etc/dnsmasq.d/04-pihole-static-dhcp.conf"
RSYNC_COMMAND=$(rsync -ai $FILE $HAUSER@$PIHOLE2:$FILE)
if [[ -n "${RSYNC_COMMAND}" ]]; then
  # rsync copied changes, update GRAVITY
  ssh $HAUSER@$PIHOLE2 "pihole -g"
# else
  # no changes
fi

FILE="/etc/dnsmasq.d/02-pihole-dhcp.conf"
RSYNC_COMMAND=$(rsync -ai $FILE $HAUSER@$PIHOLE2:$FILE)
if [[ -n "${RSYNC_COMMAND}" ]]; then
  # rsync copied changes, update GRAVITY
  ssh $HAUSER@$PIHOLE2 "pihole -g"
# else
  # no changes
fi

if [ $RESTART == "1" ]; then
  # INSTALL FILES AND RESTART pihole
  ssh $HAUSER@$PIHOLE2 "service pihole-FTL stop"
  ssh $HAUSER@$PIHOLE2 "pkill pihole-FTL"
  ssh $HAUSER@$PIHOLE2 "service pihole-FTL start"
fi
            

#END OF SCRIPT

Then, I modified this script:

#!/bin/bash
#modified script from: https://discourse.pi-hole.net/t/good-solution-to-automatically-revert-to-normal-if-pi-hole-dies/10059/4
#credit to the script that was shared.
#zeit0dn1 simplified/modified to use dhcp config files that are kept in sync from primary DHCP pihole using rsync

#primary Pihole
target=192.168.3.1

#ping command used to determine if host is up
count=$( ping -c 3 -w 3 $target | grep icmp* | wc -l )

#lock file dir
dir=/tmp/
 
#command to check for FTL errors
probe=$(curl -s http://${target}/admin/ | grep offline)
check=$(echo $probe)
 
 
#config for starting DHCP server
dhcpStartRange="192.168.1.50"
dhcpStopRange="192.168.1.100"
dhcpRouter="192.168.1.1"
dhcpLeaseTime="1"
dhcpDomain="mydomain.com"


#check for reply, if we do not get any ping responses, we are down
if [ $count -eq 0 ]
then
	#do dhcp
    echo "PiHole-PROD DHCP server is not responding!"
	#check for lockfile
    if [ -e ${dir}dhcp.on ]
    then
		echo "Secondary DHCP server already enabled. No changes or notifications performed."
		exit 0
	else
		echo "Generating lock file"
		#generate lockfile to prevent double notifications
		touch ${dir}dhcp.on
		echo "Done."

		#Enable DHCP A B C D E. A=Range start, B=Range end, C=Gateway D=Lease Time E=Domain
		#        pihole -a enabledhcp "192.168.1.200" "192.168.1.251" "192.168.1.1" "1h" "local"
		#FLTDNS crashes if hour string is specified under D
		#pihole -a enabledhcp 192.168.0.111 192.168.0.219 192.168.1.1 1 timeghost.com
		#the full path is need or crontab cannot find the command
		/usr/local/bin/pihole -a enabledhcp $dhcpStartRange $dhcpStopRange $dhcpRouter $dhcpLeaseTime $dhcpDomain

 
		#restart dns and activate DHCP server
		echo "Restarting DNS."
		/usr/local/bin/pihole restartdns

 	fi
	exit 0
#check for FTLDNS error via web
elif [ $count -eq 3 ] && [[ $check == *"offline"* ]]
then
        echo "Pihole-PROD host is up, but with FTLDNS error!"
		#check for lockfile
        if [ -e ${dir}dhcp.on ]
        then
        	echo "Secondary DHCP server already enabled. No changes or notifications performed."
        	exit 0
		else
        	echo "Generating lock file"
			#generate lockfile to prevent double notifications
        	touch ${dir}dhcp.on

      	
			#Enable DHCP A B C D E. A=Range start, B=Range end, C=Gateway D=Lease Time E=Domain
			#        pihole -a enabledhcp "192.168.1.200" "192.168.1.251" "192.168.1.1" "1h" "local"
			#FLTDNS crashes if hour string is specified under D
			/usr/local/bin/pihole -a enabledhcp $dhcpStartRange $dhcpStopRange $dhcpRouter $dhcpLeaseTime $dhcpDomain
 
		#restart dns and activate DHCP server
		echo "Restarting DNS."
		/usr/local/bin/pihole restartdns

		exit 0
	fi
else # PiHole-PROD is UP!  we got our ping responses and no FTL error
	if [ -e ${dir}dhcp.on ]
	then
		echo "PiHole-PROD is UP. Shutdown Secondary DHCP"
		#restart pihole / disable DNS seerver
		echo "Restarting DNS."
		/usr/local/bin/pihole -a disabledhcp
		/usr/local/bin/pihole restartdns
		
		#remove our lock file
		rm -rf ${dir}dhcp.on
     	
		exit 0
	else #nothing to do, everything is fine
		echo "PiHole-PROD is UP, nothing to do."
	fi

fi


#END OF SCRIPT

For testing, just run this script from command line.

So now I run from crontab every minute and pipe it to syslog using logger

* * * * * /pathToScript/piholedhcpcheck.sh | /usr/bin/logger -t piholedhcpcheck

**NOTE: for DHCP to provide both DNS servers (commas and spaces are important)
add:
dhcp-option=6, 192.168.1.1, 192.168.1.99
to /etc/dnsmasq.d/02-pihole-dhcp.conf
on both piholes and restart with: pihole restartdns

It works very well for me. You could go up to a minute without DHCP, but that isn't a huge deal...

I also keep the dhcp.lease file in sync so the secondary knows what the current leases are.

Feedback is always welcome. I hope this helps someone.

It sure would be nice if this was part of the pihole toolkit as running a single DNS server sucks when you have to reboot or throw an error cause your messing around.

Cheers!

1 Like
2

Thanks for sharing this! I've been planning on picking up another pi to set up as a backup pihole, and I've been looking for something to give me an idea of how to keep the all the lists in sync.

1 Like
3

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.