High availability (HA) for Pi-hole (running two Pi-hole's)

Hi everyone...

Happy to colaborate on this. It´s something I´ve trying to setup latelly since my primary raspi died and my wife can´t use internet until i fix it. :slight_smile:

I think the DNS is not the bigger single point of failure but the DHCP (for thouse that use it in convination with dns filtering

We fully understand second DNS is not a backup but it will become an active for some clients, so both raspberries should have it active and responding to queries.. We will not have convined reports... Gravity list and updates is also not really important and very easy to have same settings on both.

With 5.0 all the whitelist/blocklist are already on the gravity.db, so should be easy to copy it using the already developed scripts and ensuring we run the corresponding pihole command on the destination pi and not on the source one (yes, it happedned to me)

So, we need to add some files of the /etc/dnsmasq.d folder for the DHCP server, specially the 99-second-DNS.conf and 04-pihole-static-dhcp.conf.

What I´m unsure is if we can activate the DHCP on the backup raspi once we detect primary is down...

Thanks in advance
Miguel

Below is my first attempt to use lsyncd to synchronize some files between 2 Pi-hole servers.
Thanks to @fireheadman for inspiration :wink:

Notes:

  • I'm using one-way sync, primary --> secondary server
  • Separate 'rsync' account is used on the second/backup server with passwordless login and ability to sudo (that is not covered as already described in multiple sources on Internet)
  • reload after successfull sync is yet to be implemented, 'pihole restartdns reload-lists' is recommended
  • latest version (2.2.3) lsyncd is used, that was installed from sources
  • with lsyncd installed from sources the necessary system scripts will be missing, there're a few workarounds found:
    • install [older] lsyncd using apt, then remove it, but not purge; then install from sources or [re-]run sudo make install
    • download the scripts from external source and install them manually
  • Inclusion with 'filter' directive is available since lsyncd version 2.2.3
  • lsyncd help and examples available here

Primary Pi-hole configuration:

sudo mkdir /etc/lsyncd
sudo mkdir /var/log/lsyncd
sudo touch /var/log/lsyncd/lsyncd.status
sudo touch /var/log/lsyncd/lsyncd.log

sudo nano /etc/lsyncd/lsyncd.conf.lua

settings {
  logfile = "/var/log/lsyncd/lsyncd.log",
  statusFile = "/var/log/lsyncd/lsyncd.status",
  statusInterval = 20
}
 
sync {
  default.rsyncssh,
  delete = false,
  source = "/etc/pihole/",
  host = "rsync@192.168.1.6",
  targetdir = "/etc/pihole/",
  delay = 20,
  filter = {
		'+ gravity.db',
		'+ custom.list',
		'- **'
	},
  rsync = {
    archive = true,
    whole_file = true,
    rsync_path = "sudo rsync",
    _extra = { "--omit-dir-times" }
  }
}

sync {
  default.rsyncssh,
  delete = false,
  source = "/etc/dnsmasq.d/",
  host = "rsync@192.168.1.6",
  targetdir = "/etc/dnsmasq.d/",
  filter = { '- 01-pihole.conf'  },
  delay = 20,
  rsync = {
    archive = true,
    whole_file = true,
    rsync_path = "sudo rsync", 
    _extra = { "--omit-dir-times" }
  }
}

Additional directories could be added to sync as needed. For example, I recently added a folder containing stubby configuration file.
Secondary/backup Pi-hole - make sure rsync is installed.

ToDo:
Execute 'pihole restartdns reload-lists' over ssh on the remote server after successful sync.
This example could be used.

I would propose the option to use rqlite or dqlite as a database.
This would probably be the most robust solution for distributed Pi-hole, allowing for many instances with little to no extra config.

1 Like

Do you have any tutorial on how to set this up in combination with PiHole? Am interested!

Unfortunately no, neither is an sqlite drop-in replacement, one is a library which would have to be integrated into pi-hole, the other uses sqlite as a storage engine (with some caveats) but only allows access via an HTTP API. It isn't a trivial integration to be honest, but I wanted to put it out as a possible option.

The simplest option would probably be to use mySQL or pgSQL and convert commands using sqlite3 to equivalent commands for those DBs, but this would only work for the gravity.db as pihole-FTL should have an embedded sqlite one.

1 Like

Ok thanks. It also appears the lsyncd solution is now broken even, as the PiHole 5.0 releases doesn't use those files it syncs but uses the database option instead. But I guess there's a way to revert that or so.

It still works, however the initially posted configuration needs adjustment, please see my configuration example above.

Thanks, got this working now! Am also using the lsync postscript, seems to work well enough for the purpose.

With Pi-hole version 5, shouldn't the pihole-FTL.db be backed up as well?

EDIT:
Having read @DL6ER's comment above, trying to understand why pihole-FTL.db is not required. When gravity.db is transferred to the 2nd Pi, does pihole-FTL.db get auto-generated / auto-computed?

This is the long term database of queries for the device on which it is running. That database will continue running as-is on both instance of Pi-hole, recording the queries per the privacy setting on each device.

This seems rather straightforward and clear to me

2 Likes

I mean it’s not hard, you have a router setup as the dns on the network, set it up so that it forwards all dns queries to pi 1 by default, but if pi 1 isn’t active then it sends it to pi 2

Should be fairly straightforward thing to setup on your home router.

Have your preferred pi as dns server1 lets say 192.168.1.30

Then the other one as dns server2 at 192.168.1.31

Most home routers allow you to have two dns servers on the network, by default let’s say quad9 is 9.9.9.9 or 149.112.112.112.

Then just use your preferred script to get them to update one another

Thanks, this worked for me!

I used the following script to handle some - maybe unnecessary - stuff:

#!/bin/bash
#
#

/usr/bin/rsync "$@"
result=$?
(
	if [ $result -eq 0 ]; then
		ssh root@10.0.0.116 "sed -e 's|=10.0.0.2/23|=10.0.0.116/23|;s|IPV6_ADDRESS=xxx|IPV6_ADDRESS=xxx|' -i /etc/pihole/setupVars.conf"
		ssh root@10.0.0.116 "sed -e 's|10.0.0.2|10.0.0.116|;s|2003:e6:xxx|2003:fd:xxx|' -i /etc/pihole/local.list"
		ssh root@10.0.0.116 "pihole restartdns reload ; \
				systemctl restart lighttpd.service ; \
				systemctl restart pihole-FTL.service"
	fi
) >/dev/null 2>/dev/null </dev/null

exit $result

and this is how my lsyncd config looks like:

settings {
	logfile = "/var/log/lsyncd/lsyncd.log",
	statusFile = "/var/log/lsyncd/lsyncd.status",
	statusInterval = 30
}
 
sync {
	default.rsyncssh,
	delete = true,
	source = "/etc/pihole/",
	host = "root@10.0.0.116",
	targetdir = "/etc/pihole/",
	delay = 20,
	rsync = {
		binary = "/root/rsync-with-pihole-restart.sh",
		archive = true,
		whole_file = true,
		_extra = { "--omit-dir-times" }
		},
	filter = {
        filter = {
                '- pihole-FTL.db',
                '- pihole-FTL.db-journal',
                '- localversions',
                '- localbranches'
                }
}

sync {
	default.rsyncssh,
	delete = true,
	source = "/etc/dnsmasq.d/",
	host = "root@10.0.0.116",
	targetdir = "/etc/dnsmasq.d/",
	delay = 20,
	rsync = {
		binary = "/root/rsync-with-pihole-restart.sh",
		archive = true,
		whole_file = true,
		_extra = { "--omit-dir-times" }
		}
}

Cheers,
Bjoern

1 Like

I too would like the ability to cluster 2 or more Pi-hole instances that doesn't require custom scripts that could inevitably break without me noticing. A clustering solution provided via the GUI would be such a great addition to the already great features of Pi-hole.

In addition to the syncing of adlists, whitelists etc the syncronisation of stats would be great also. I would likely plan to use a load balancer infront of the pi-holes so I personally wouldn't have the need for a cluster IP solution created/used by Pi-hole.

3 Likes

Just wanted to share my solution here with you incase sb is interested: GitHub - shaderecker/ansible-pihole: Bootstrap a Raspberry Pi with Ansible and install Docker + Pi-hole

  • Complete deployment automated in Ansible
  • Basic bootstrapping, configuration and updates
  • Pi-hole as docker container
  • High availability failover cluster with keepalived (IPv4 & IPv6)
  • Synchronisation of settings between Pi-hole instances with rsync:
    • gravity.db (Adlists, Domains, Clients, Groups, Group Assignments of all aforementioned items)
    • custom.list (Local DNS Records)
    • 05-pihole-custom-cname.conf (Local CNAME Records)

Maybe better to create your own thread here to discuss.
Maybe if you agree, have a mod split this topic ?

I noticed you do a one on one rsync of the gravity dbase file:

Wouldn't that potentially also rsync the dbase file that might still be in transition performing some UPDATE query or something else ?
Isnt it safer to dump the dbase instead on the "primary" and copy the dump over to the "secondaries" eg:

sqlite3 /etc/pihole/gravity.db ".backup main gravity.db.dump"

And restore/import on the secondary hosts if checksum check (md5 or whatever) has changed with:

sudo -u pihole sqlite3 /etc/pihole/gravity.db ".restore main gravity.db.dump"

Would need to stop pihole-FTL to prevent tables being locked while importing.

EDIT: wrong dbase
EDIT2: looks like I'm able to split these posts to a new thread if you like ?

1 Like

Please use Issues · shaderecker/ansible-pihole (github.com) for discussion.

2 Likes

Thank you for your suggestions, I moved the topic to GitHub Discussions:
save to directly rsync dbase? · Discussion #2 · shaderecker/ansible-pihole · GitHub

2 Likes

Yeah stupid of me.
Github is better spot for this.