Pi-hole Gemini (Two-way Pi-Hole lists sync) Readme - 03-12-2019
Based on https://www.reddit.com/user/LandlordTiberius Dual pihole sync 2.0 script (https://www.reddit.com/r/pihole/comments/9hi5ls/dual_pihole_sync_20/) which was based on Sync two PiHoles bash script (https://www.reddit.com/r/pihole/comments/9gw6hx/sync_two_piholes_bash_script/) by https://www.reddit.com/user/jvinch76
While I personally started with LandlordTiberius' script, I made changes for my personal setup, and thought that my version of the script could be useful to others who are running 2 Pi-holes that aren't using the DHCP and just want to keep their white lists, black lists, block lists and gravity synced between 2 Pi-holes that also didn't want the sync to be on a timer (cron) or using a file monitoring utility like inotify.
The pihole-gemini script can be found here: https://pastebin.com/mc2vs1Ya
Requirements:
- 2 systems running as Pi-holes
- SSH access enabled on both systems
- A user on each system with sudo permission and ssh access (the username MUST be the same on both for this script)
- rsync should be installed on both systems
Purpose:
The main purpose of this script is to keep the lists of 2 Pi-holes in sync. While there are other scripts out there that do a great job of keeping the black and white lists synchronized between two Pi-holes, I wasn't happy with how they were being triggered. Basically, I didn't want to wait for a cron to run to push an update, and frankly, didn't want cron jobs firing off when I felt that I didn't need them to, but I also didn't want to run a service like inotify (as lean as it is) to monitor the files for changes. After lots of digging around Pi-hole's files, I decided the best place to trigger my script from was the end of Pi-hole's gravity.sh script. This afforded me the maximum amount of integration with Pi-hole as the script is triggered whenever the gravity.sh script is run. The gravity.sh script is triggered whenever something is added or removed from a white or black list, when adding new block lists, or removing, enabling or disabling existing block lists. So pihole-gemini will run whenever gravity is updated, including when Pi-hole's gravity is updated from the command line using the "pihole -g" command.
With the number of changes and additional logic I've added to my version of the script, I guess this is really more of a 'fork' than an update. As such, I've decided to name it "Gemini" (the interstellar twins) since I couldn't find any references to interstellar clones, I figured interstellar twins was the closet celestial body to imply, well, two of something very similar.
Features:
- A good amount of logging information has been added to be able to go back and look at recent jobs to make sure everything ran as expected.
- The script is designed for you to set both ip addresses in one script and simply use that one script on both Pi-holes without having to make custom edits on each Pi-hole to define the 'other' Pi-hole's ip address.
- Ability to define custom ports. The script was written to allow for the configuration of custom SSH ports. This allows you to use a custom port (instead of the default 22) for SSH connections. The port can be defined for each connection, so both Pi-holes could use different ports for SSH. This is in case a secondary Pi-hole is running in a VM using a different non-standard port than the primary Pi-hole.
Benefits of running sync from the gravity.sh script:
- The script runs automatically when it needs to. It does not use cron or file monitoring to be triggered.
- When using the Update Gravity page in the web interface, or when adding or removing block lists from the Settings page, the results of the sync job(s) are displayed along with the blocklist information.
- When updating gravity, a "local" gravity update will also trigger a "remote" gravity update, making this a 2-way sync on gravity updates. This happens from both the web interface and from using the pihole -g command at the prompt.
Logging information:
- The script creates a new log file every day, and every job for the day is appended to the file.
- The default directory I'm using for logs is currently /tmp. You can change this in the script's user-defined variables section, and you *SHOULD* change it if you wish to preserve the log files. Log files in the /tmp directory are automatically deleted on system restarts. If you do change the log directory, be sure to set the LOGKEEPDAYS variable to ensure that old log files are cleaned up at the interval you desire.
NOT FOR DHCP Configurations:
The current version of the script does NOT sync DHCP files. If you are using Pi-hole for DHCP configurations, I would recommend finding another script that DOES use cron (for scheduled checks). This is because, for redundancy, a primary DHCP server should be tested at regular intervals by the backup server, so the backup can take over if the primary is down. I would also run something like inotify on the primary DHCP server to keep things like the lease files in sync. This puts DHCP redundancy well outside the scope of pihole-gemini, as the script is only designed for keeping white lists, black lists, block lists and gravity in sync. Because of the way this script is designed to be triggered and used, it is simply not capable of keeping DHCP stuff synchronized in any meaningful manner that would be usable for providing DHCP redundancy.
Other notes:
- While the script does keep the lists synchronized on two Pi-holes, be aware that all other aspects of the Pi-Holes are completely independent, including the statistics displayed on the respective Pi-hole admin pages, and the disable functions. So if you need to disable blocking, you will need to have both Pi-hole admin pages open, and manually disable blocking on EACH ONE for the time period you want it disabled for. I am still trying to find where the disable functions from the web interface are located, and if there's a way to "piggyback" a script to the function in the way the pihole-gemini script piggybacks the gravity.sh script.
- ALL gravity updates triggered by pihole-gemini are triggered using the --skip-download option to prevent the block lists from redownloading. The script should never have to trigger a full gravity update including down-loading the block lists, since gravity updates are what trigger the script.
Credits:
This version of jvinch76's sync two piholes bash script (pihole-gemini) was written by
https://www.reddit.com/user/GeorgeT93
This script originally started life as
https://www.reddit.com/r/pihole/comments/9gw6hx/sync_two_piholes_bash_script/
by https://www.reddit.com/user/jvinch76
The modifications I've made were actually based on the updated version at
https://www.reddit.com/r/pihole/comments/9hi5ls/dual_pihole_sync_20/
by https://www.reddit.com/user/LandlordTiberius
Setting up and configuring the pihole-gemini script. The script can be found at https://pastebin.com/mc2vs1Ya
These steps need to be taken on both piholes.
1.) Log in to Pi-hole as the user that will be used for running the sync process. Make sure this user has both ssh and sudo access to the local Pi-hole system.
2.) Change to the /usr/local/bin directory.
$ cd /usr/local/bin
3.) Create the script file and open it for editing.
$ sudo nano pihole-gemini
4.) Paste the script into the pihole-gemini script file.
5.) Change the values in the USER-DEFINED VARIABLES section to match your setup.
6.) Save the script (ctrl+o, then ) and exit the editor (ctrl+x).
7.) Make the script executable.
$ sudo chmod +x pihole-gemini
8.) Create an ssh key to allow remote connections without supplying a password for the user that you're using to sync files between Pi-holes (the connections will use the key generated here instead.)
$ ssh-keygen
Answer the prompts (leaving them blank will use the default values) to generate the ssh key.
If you get a permission denied error, you may need to manually create the .ssh folder in the home folder of the user that will be used to sync the files, then make sure the correct user owns the folder. The example below uses the 'pi' user. If you did not get a permission denied error, you can skip to step 9.
$ cd ~
$ sudo mkdir .ssh
$ sudo chown user:group .ssh
So for the user pi, the command would be:
$ sudo chown pi:pi .ssh
Now you should retry the ssh-keygen command (start step 8 over.)
9.) Check that the ssh service is running.
$ eval ssh-agent
If you get a response like "Agent pid 1234", then the service is running. Note that the numbers 1234 are for demonstrative purposes, and the actual number displayed on your system will be different.
10.) Add the ssh key to the local Pi-hole. Once you are sure the ssh service is running, then add the key, being sure to use the filename you created when you ran ssh-keygen. If you left it blank, it will be the default filename (id_rsa), which is what I'm using in this example.
$ ssh-add id_rsa
If you set a passphrase during ssh-keygen, you will be prompted for the passphrase in order to add the key.
11.) Send the key to your 'other' pi-hole system. You should use the 'other' pi-hole's username @ the other pi-hole's ip address in the command
$ ssh-copy-id other-pi-username@other-pi-ip-address
12.) After configuring both pi-hole's ssh keys, test the ssh login from the command line.
If you are NOT using a custom port for ssh, use
$ ssh username@other-pihole-ip
If you ARE using a custom port # for ssh, substitute your port # for the ## in the example below.
$ ssh -p ## username@other-pihole-ip
On your first login, it may prompt you for the passphrase you set in ssh-keygen (if you set one.) Enter the passphrase, and immediately after logging in, issue the "exit" command to disconnect from the 'other' pihole, and try the ssh command again. You should not be prompted for the passphrase after entering it the first time.
Once you've confirmed the ability to log in without having to supply a password or passphrase (ensuring the ssh key is working), you can issue the "exit" command to close the ssh session and return to the local prompt, however, you could perform step 13 remotely to get the 'other' pi-hole integrated before closing the connection. If you wish to do this, perform step 13 before issuing the "exit" command, and then perform step 13 again (this time locally) to finish full pi-hole integration.
13.) Finally, we need to integrate the script into Pi-hole. We will do this by editing Pi-hole's gravity.sh script, but first, we'll back it up.
$ sudo cp /opt/pihole/gravity.sh /opt/pihole/gravity.sh.bak
Then we'll edit the gravity.sh file
$ sudo nano /opt/pihole/gravity.sh
Press the or key on your keyboard and hold it down until you get to the bottom of the file.
The very last line should read:
"${PIHOLE_COMMAND}" status
We will be adding a new command directly ABOVE that line, so that "${PIHOLE_COMMAND}" status remains the last line of the file. The line we need to add is:
su -c '/usr/local/bin/pihole-gemini' - pi
Note that the "pi" at the end of the line should be replaced with the username of your sync user account. Once we're done editing it, we can save (ctrl+o, then ) and exit the editor (ctrl+x).
Once you've finished step 13, you're done. You can invoke the script directly by calling pihole-gemini at the command line, and should do so, to manually test the script to ensure everything is working as expected. From now on, it will run automatically whenever you update gravity, add or remove items from the white or black list, or add or remove items from the block list (including enabling or disabling block lists).
Important note: When upgrading to a new version of Pi-hole, you may have to repeat step 13 if the gravity.sh file gets updated in order to re-enable the pihole-gemini sync.
Link to script: https://pastebin.com/mc2vs1Ya
(Edited to fix some formatting errors.)
