Good solution to automatically revert to "normal" if Pi Hole dies?

General Customizing Pi-hole
, ,
1

Hello!

I really like the idea having Pi Hole be the DNS resolver for my network because it "automatically" covers the plethora of devices around the house all at once.

HOWEVER, I just know Murphy's Law will strike eventually: the Pi Hole will die when I'm away on a business trip and no one in my household will be able to fix the Pi Hole device, or login into the main router to set the DNS setting back to 8.8.8.8 to restore internet visibility until I get home to sort things out.

Hence, I ask the community: is there a good solution to automatically revert to "normal" DNS resolution if Pi Hole dies?

For example:

  • Can a router with Tomato or some other popular 3rd party OSS be set to periodically check Pi Hole's "health", and "revert" if the Pi Hole gets sick?

  • I have two BeagleBone Black Rev C's (BBB) running various home automation things. The point is that I plan to have one BBB run Pi Hole, and I could setup the other BBB to act as a watchdog over Pi Hole's health on BBB 1. As such, any recos for implementing such a watchdog scheme?

Help!

2 Likes
2

It should certainly be possible with the gear you describe.

You could set up a script on one (or both, why not?) of your BBBs and have them test dns connectivity periodically, and if it goes down for longer than you consider acceptable change
the router settings away from the Pi-hole.

My suggestion would be to have the script connect to your router by ssh and change the router DNS settings back away from your Pi-hole that way. (If your router supports using ssh keys for authentication all the better).

I'd be wary of having the scripts too eager to change settings like that, so I would also add some sanity checking, such as ensuring connectivity in general is still in place, and that 8.8.8.8 is reliably reachable, and returning sensible numbers before changing back to it.

A less automated option, and also handy to have around as plan C if plan B above fails, but involving some participation by a household member: Have a saved copy of your router configuration but with 8.8.8.8 as the DNS saved on one (or more) of their computers, and show them how to upload that to the router. That way they don't need to worry about what what settings to change, or anything like that. Your instructions for them would be as simple as Click bookmark, type username & password, go to "configuration", click "restore setting", select this file, click yes.

3

I run a secondary instance of pi-hole (as a backup Pi-hole DNS in case the main one fails).

You could do the same thing with the BBBs run DNS1 on 1 and DNS2 on the second (I think they are capable of doing that from a HW perspective).

In my case DNS1 is also the DHCP server.

DNS2, monitors DNS1 for network presence and (possible) FTLDNS errors. In both cases or any (where DNS1 is up but FTLDSN has an error), DNS2 takes over from a DHCP (and DNS) perspective.

It's as @robgill said, a heartbeat script (executed every 1 minute) that checks the status.

If it's something you want to pursuit i can share relevant snippets of the script code that will help you out.

You could also do this :slight_smile:

4

I would be interested in this code. I run two Pi's in parallel (one for my router and the 25 clients on that), the other just for this computer), and I would like to be able to configure one to take over from other in event of failure.

5

Code is a little messy.
Change parameters accordingly:

#!/bin/bash
target=192.168.1.3
count=$( ping -c 3 -w 3 $target | grep icmp* | wc -l )
dir=/media/Network/
 
probe=$(curl -s http://${target}/admin/ | grep offline)
check=$(echo $probe)
 
#check for reply
if [ $count -eq 0 ]
then
#do dhcp
        echo "Main DHCP server is not responding!"
#check for lockfile
        if [ -e ${dir}dhcp.on ]
        then
        echo "DHCP server already enabled. No changes or notifications performed."
        exit 0
else
        echo "Generating lock file"
#generate lockfile to prevent double notifications
        touch ${dir}dhcp.on
        echo "Done."
#Enable DHCP A B C D E. A=Range start, B=Range end, C=Gateway D=Lease Time E=Domain
#        sudo pihole -a enabledhcp "192.168.1.200" "192.168.1.251" "192.168.1.1" "1h" "local"
#FLTDNS crashes if hour string is specified under D
        sudo pihole -a enabledhcp "192.168.1.200" "192.168.1.251" "192.168.1.1" "1" "local"
#remove possible existing configuration files
        sudo rm -rf /etc/dnsmasq.d/02-pihole-dhcp.conf
        sudo rm -rf /etc/dnsmasq.d/03-pihole-wildcard.conf
        sudo rm -rf /etc/dnsmasq.d/04-pihole-static-dhcp.conf
        fi
 
#check for 02-pihole-dhcp.conf
if [ -e /etc/dnsmasq.d/02-pihole-dhcp.conf ]
then
    echo "02-pihole-dhcp.conf present. No changes made."
else
    echo "02-pihole-dhcp.conf missing. Generating file."
 
#generate file and pass strings
sudo touch /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo grep -q -F 'dhcp-authoritative' /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo sh -c "echo 'dhcp-authoritative' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
sudo grep -q -F 'dhcp-range=192.168.1.200,192.168.1.251,1h' /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo sh -c "echo 'dhcp-range=192.168.1.200,192.168.1.251,1h' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
sudo grep -q -F 'dhcp-option=option:router,192.168.1.1' /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo sh -c "echo 'dhcp-option=option:router,192.168.1.1' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
sudo grep -q -F 'dhcp-leasefile=/etc/pihole/dhcp.leases' /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo sh -c "echo 'dhcp-leasefile=/etc/pihole/dhcp.leases' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
sudo grep -q -F 'domain=local' /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo sh -c "echo 'domain=local' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
sudo grep -q -F 'dhcp-option=6,192.168.1.5,192.168.1.3' /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo sh -c "echo 'dhcp-option=6,192.168.1.5,192.168.1.3' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
echo "Done."
fi
 
#check for 03-pihole-wildcard.conf
if [ -e /etc/dnsmasq.d/03-pihole-wildcard.conf ]
then
    echo "03-pihole-wildcard.conf present. No changes made."
else
    echo "03-pihole-wildcard.conf missing. Generating file."
 
#generate file (no strings to pass)
sudo touch /etc/dnsmasq.d/03-pihole-wildcard.conf
echo "Done."
fi
 
#check for 04-pihole-static-dhcp.conf
if [ -e /etc/dnsmasq.d/04-pihole-static-dhcp.conf ]
then
    echo "04-pihole-static-dhcp.conf present. No changes made."
else
    echo "04-pihole-static-dhcp.conf missing. Generating file."
 
#generate file
sudo touch /etc/dnsmasq.d/04-pihole-static-dhcp.conf
 
#pass strings in order for existing dhcp clients to keep IPs
sudo grep -q -F 'dhcp-host=00:00:00:00:00:00,192.168.1.2,StaticHost1' /etc/dnsmasq.d/04-pihole-static-dhcp.conf
sudo sh -c "echo 'dhcp-host=00:00:00:00:00:00,192.168.1.2,StaticHost1' >> /etc/dnsmasq.d/04-pihole-static-dhcp.conf"
echo "Done."
 
#restart dns and activate DHCP server
echo "Restarting DNS."
pihole restartdns
echo "Done."
 
#notify
echo "Notification sent."
fi

else
#check for FTLDNS error via web
 if [ $count -eq 3 ] && [[ $check == *"offline"* ]]
 
    then
        echo "Pi-hole is up with FTLDNS error!"
#check for lockfile
        if [ -e ${dir}dhcp.on ]
        then
        echo "DHCP server already enabled. No changes or notifications performed."
        exit 0
  else
        echo "Generating lock file"
#generate lockfile to prevent double notifications
        touch ${dir}dhcp.on
        echo "Done."
#Enable DHCP A B C D E. A=Range start, B=Range end, C=Gateway D=Lease Time E=Domain
#        sudo pihole -a enabledhcp "192.168.1.200" "192.168.1.251" "192.168.1.1" "1h" "local"
#FLTDNS crashes if hour string is specified under D
        sudo pihole -a enabledhcp "192.168.1.200" "192.168.1.251" "192.168.1.1" "1" "local"
#remove possible existing configuration files
        sudo rm -rf /etc/dnsmasq.d/02-pihole-dhcp.conf
        sudo rm -rf /etc/dnsmasq.d/03-pihole-wildcard.conf
        sudo rm -rf /etc/dnsmasq.d/04-pihole-static-dhcp.conf
        fi
 
#check for 02-pihole-dhcp.conf
 if [ -e /etc/dnsmasq.d/02-pihole-dhcp.conf ]
 then
    echo "02-pihole-dhcp.conf present. No changes made."
 else
    echo "02-pihole-dhcp.conf missing. Generating file."
 
#generate file and pass strings
    sudo touch /etc/dnsmasq.d/02-pihole-dhcp.conf
    sudo grep -q -F 'dhcp-authoritative' /etc/dnsmasq.d/02-pihole-dhcp.conf
    sudo sh -c "echo 'dhcp-authoritative' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
    sudo grep -q -F 'dhcp-range=192.168.1.200,192.168.1.251,1h' /etc/dnsmasq.d/02-pihole-dhcp.conf
    sudo sh -c "echo 'dhcp-range=192.168.1.200,192.168.1.251,1h' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
    sudo grep -q -F 'dhcp-option=option:router,192.168.1.1' /etc/dnsmasq.d/02-pihole-dhcp.conf
    sudo sh -c "echo 'dhcp-option=option:router,192.168.1.1' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
    sudo grep -q -F 'dhcp-leasefile=/etc/pihole/dhcp.leases' /etc/dnsmasq.d/02-pihole-dhcp.conf
    sudo sh -c "echo 'dhcp-leasefile=/etc/pihole/dhcp.leases' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
    sudo grep -q -F 'domain=local' /etc/dnsmasq.d/02-pihole-dhcp.conf
    sudo sh -c "echo 'domain=local' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
    sudo grep -q -F 'dhcp-option=6,192.168.1.5,192.168.1.3' /etc/dnsmasq.d/02-pihole-dhcp.conf
    sudo sh -c "echo 'dhcp-option=6,192.168.1.5,192.168.1.3' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
    echo "Done."
 fi
 
#check for 03-pihole-wildcard.conf
 if [ -e /etc/dnsmasq.d/03-pihole-wildcard.conf ]
then
    echo "03-pihole-wildcard.conf present. No changes made."
else
    echo "03-pihole-wildcard.conf missing. Generating file."
 
#generate file (no strings to pass)
sudo touch /etc/dnsmasq.d/03-pihole-wildcard.conf
echo "Done."
 fi
 
#check for 04-pihole-static-dhcp.conf
 if [ -e /etc/dnsmasq.d/04-pihole-static-dhcp.conf ]
then
    echo "04-pihole-static-dhcp.conf present. No changes made."
else
    echo "04-pihole-static-dhcp.conf missing. Generating file."
 
#generate file
sudo touch /etc/dnsmasq.d/04-pihole-static-dhcp.conf
 
#pass strings in order for existing dhcp clients to keep IPs
sudo grep -q -F 'dhcp-host=00:00:00:00:00:00,192.168.1.2,StaticHost1' /etc/dnsmasq.d/04-pihole-static-dhcp.conf
sudo sh -c "echo 'dhcp-host=00:00:00:00:00:00,192.168.1.2,StaticHost1' >> /etc/dnsmasq.d/04-pihole-static-dhcp.conf"
echo "Done."
  
#restart dns and activate DHCP server
echo "Restarting DNS."
pihole restartdns
echo "Done."
 
#notify
echo "Notification sent."
exit 0
 fi
else
    echo "DNS1 is Alive!"
 
#remove DHCP settings.
 
 if [ -e /etc/dnsmasq.d/02-pihole-dhcp.conf ]
 then
echo "Removing settings."
sudo rm -rf /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo rm -rf /etc/dnsmasq.d/03-pihole-wildcard.conf
sudo rm -rf /etc/dnsmasq.d/04-pihole-static-dhcp.conf
 
#restart pihole / disable DNS seerver
echo "Restarting DNS."
sudo pihole -a disabledhcp
sudo pihole restartdns
sudo rm -rf ${dir}dhcp.on
echo "Done."
 
#notify
echo "Notification sent."
else
    echo "Lock file already deleted. No action performed."
 fi
 fi
#remove DHCP settings.
fi

Run this via crontab every minute and it will take over in case of failure on target.

3 Likes
6

Thanks much.

7

Thanks robgill, RamSet, jfb!!!

Question regarding robgill's statement:

Putting aside whether one of my two BBBs or my router would randomly go berserk and mess up the network traffic or something:

The implication from robgill's note is that a lazy person (like Yours Truly) could simply setup and run identically configured Pi Hole instances on each of my BBBs, and neither program would conflict with each other. Plus, if an asteroid hit only one of the BBBs, the other BBB would be none the wiser and happily continue to run its Pi Hole instance. True? False?

If this crazy idea is true, which PiHole+BBB would do the bulk of the work during normal, parallel operation?

8

That's @jfb's question :slight_smile:

This statement is correct as long as you run both as DNS resolver (and not both as DHCP servers). It would be correct in the case of them being DHCP servers when you would have them set up with different settings.
However running 2 DHCP servers in the same LAN for the same class of IP's would be really bad practice (unless your network topology configuration would require it ¯\(ツ)/¯ ).

I think in your setup, one router and two BBBs with standalone Pi-hole installations (as DNS only) will work wonders :slight_smile:

One thing to keep in mind:
In your DHCP server (I'm assuming the router does that) you would have to specify BBB1's IP and BB2's IP as Primary and Secondary DNS settings (so that all the clients that get an IP from it, will get the correct DNS settings).

If that doesn't work (no option under DHCP server for DNS servers), you could set it up under your WAN connection's DNS (same BBB1's IP and BB2's IP as Primary and Secondary DNS) but that will only show queries in the query log on the Pi-hole Admin interface as originating from your router's IP and not the client's IP (since the client would request from the router and the router request from the pi-hole).

You could also manually specify DNS1 and DNS2 on your clients. That will make logging on the Pi-hole a lot easier.

That would be DNS1 but DNS2 is not there for backup/failsafe only.

Some queries will bleed through the secondary DNS.

That's why if you have DNS1=Pi-hole and DNS2=Google, some site (at random) will show ads sometimes and most times, it won't.

If you Have DNS1=BBB1 Pi-hole and DNS2=BBB2 Pi-hole, you're set :slight_smile:

and if DNS1=BBB1's Pi-hole is offline, DNS2=BBB2 Pi-hole will take the full load ('till DNS1 is back up).

1 Like
9

In my setup, with two Apple routers (one a wired extender to the other), the vast majority of my traffic goes to the primary PiHole. The primary Apple router is the DHCP server, the second is in bridge mode. DNS addresses are listed in the Apple router as pihole primary in the first slot, pihole secondary in the second slot. All the network devices get their DNS address from the router. In the last 24 hours, the primary PiHole received 31,600 queries, the secondary received 400 queries.

Interestingly, 300 of the queries to the secondary come from a Sense home energy monitor. This device seems to be DNS agile and will find any DNS server - it has settled on the secondary.

I like this setup. I can change things on the individual Pi's - updates, etc, and when one is offline for reboot or such, I get no sneak-through ads. I do occasionally sync the white lists through the teleporter.

Note - with this setup, the queries to the PiHoles are logged by individual IP addresses, not shown as coming from the router only. That's probably an Apple thing, but I like that behavior.

10

Update:
TL;DR - I couldn't get Pi-Hole running on my BBB, so I bought Pi Zero W's and Pi-Hole dropped right in

First, thanks to the tips on this forum, the Installing Pi-hole on a BeagleBoard to block ads - Applied Caffeine post, and some Reddit posts on this, I came really, really close to getting Pi-Hole running on my BBB. Specifically, the webpage GUI comes up, but FTL refuses to start. All I get is "Lost" "Connection" "To" "API" in the summary blocks at the top of the Dashboard. I suspect the dnsmasq or something is mis-configured, but who knows? The point is: after spending two evenings on this, I began wondering if I wasn't just trying to fit a square peg in a round hole. In my frustration, I decided to check how much new Raspberry Pi's cost these days, and before you know it a new Pi Zero W was on it's way to me via Amazon. I got it working with Pi-Hole the evening it arrived. Then, as per the guidance on this forum, I bought another Pi Zero W and quickly brought it up with its own instance of Pi-Hole.

(I haven't told my two BBB's about their new neighbors yet -- I'm sure I'll come up with some house-wide scheme to use all four computers at once for some common purpose.)

Anyway, with the two Pi's running in parallel now, it's interesting the variance of traffic/blocking "splits" between the first Pi-Hole I installed and the second one -- anything from 95/5 to 70/30. Tomorrow when the fam is out of the house I'll shutdown the original completely and confirm the second Pi seamlessly picks up the whole load.

Finally, thanks to everyone that contributed to this thread!!!

11

Edited to add, regarding:
In your DHCP server (I’m assuming the router does that) you would have to specify BBB1’s IP and BB2’s IP as Primary and Secondary DNS settings (so that all the clients that get an IP from it, will get the correct DNS settings).

For future reference, this is exactly what I'm doing: I've setup static IPs for each Pi running Pi-Hole, and on my router's DHCP server config page I set Primary DNS to Pi-1's static IP address and the secondary DNS to Pi-2's. It's been working like charm -- all the phones, DVRs, tablets, BBBs, personal and work laptops (even with the work VPN) are working fine.

Again, many thanks to everyone that contributed to this thread!!!

1 Like
12

Update: "fail over" works just fine!

I finally had a window of time where everyone was out of the house, so I shutdown Pi-1 and Pi-2 did just fine on its own. (I suppose this really says more about the Ubiquity router's ability to seamlessly fail over to the Secondary DNS spec.)

1 Like
13

Couple of things to I want to check, 192.168.1.3 is the main DHCP DNS server for you right?

In this block,
#pass strings in order for existing dhcp clients to keep IPs
sudo grep -q -F 'dhcp-host=00:00:00:00:00:00,192.168.1.2,StaticHost1' /etc/dnsmasq.d/04-pihole-static-dhcp.conf
sudo sh -c "echo 'dhcp-host=00:00:00:00:00:00,192.168.1.2,StaticHost1' >> /etc/dnsmasq.d/04-pihole-static-dhcp.conf"
echo "Done."

What exactly is the 192.168.1.2? Is that the backup dns dhcp server?

14

That is correct.

It is not. It’s a device that has a static ip asigned via the main dhcp server and upon lease expire, it will renew from the backup dhcp server (if still on), the same ip lease.

15

Thanks for the quick reply.
Any recommendation where to keep the script and how to schedule cron job?
Sorry if its the wrong place to ask this question.

16

You can keep it anywhere you want as long as you make it an executable and specify the right (full) path in your crontab.

Use http://crontab.guru for your crontab details.

Use crontab -e from underneath the root user to add the crontab entry.

Mine, checks every minute for any issues.

17

Works perfectly, thank you sir!
Commented out the lan reservation part as Google wifi can reserve any IP even if its not in its lan range but in the same subnet. Not sure if thats how other DHCP works so I use google wifi for all lan reservations even though it has range of just two ip address which are reserved for Pi1 and 2.

Edit: Is there a command to enable Enable IPv6 support (SLAAC + RA) as well on backup server?

18

just having a read up on having a failsafe Pi-hole within my network and could do with a little help please?
My pihole (pihole1) is acting as my DHCP server as, my ISP provided router does not allow manual setting of DNS servers.
I also have unbound set up on this pihole.

So, if I set up a 2nd pihole (Pihole2), I guess I would not need to have that serving DHCP leases, as pihole1 would have assigned an IP to devices which have a lease time? Correct?

Is there any means to 'dictate' that pihole1 is the primary DNS server, and use only pihole2 should pihole1 fail for some reason?
I'm thinking along the lines of a monitoring script similar to @RamSet has above, just more basic...along the lines of:

ping pihole1 every couple of minutes, 
if there is no response 
pihole enable on pihole2
else if pihole1 does respond
pihole disable on pihole2

If this 'automated' enabling and disabling of pihole2 is too messy / will not work, how does one handle which pihole network devices use?
Im also thinking about stats and data etc. How will I know which pihole has served the DNS request?
in the case of a blocked site I want to unblock for example?

also apologies for the hijack and long post!

19

That is the current/main logic behind it.

One thing to keep in mind though.

Pinging the host (only) does not guarantee an accurate state of Pi-hole.

The hot might be up and Pi-hole (FTLDNS) in error state...

That's why you need to add an extra check for

if {host} is up check for Pi-hole status

For the script above it would look like this:

if [ $count -eq 3 ] && [[ $check == *"offline"* ]]

Where $count is the number of pings and $check is the probe on the admin page.

Based on this condition, you can trigger the script to do certain stuff.

In my case, the script sends me a notification on each status change.

That's a cool thing to have but you need to keep in mind that if there are no "lock" files that register the notification state (only once per status change) it would send notifications, every time the scrip is executed.

That is correct, but it depends on how much time is allotted to the lease.

If you assign a 24h lease to the DHCP clients, and no new clients join during the outage, then you are set if pihole1 returns to a normal state in less than 24h. If the outage is > than your lease time, you might get a different IP assigned to the clients.

Passing the already assigned static IP reservations is a healthy way of handling the situation from a new device perspective. That way, the IPs don't get assigned by accident, by the new dhcp server, to the new device (since it had no record of an existing lease).

That has no relevance in the above setup/script.

No there is not.
It is better to run them in parallel.

Most OS' tend to listen to the "DNS1 as primary and DNS2 as fail-safe" however, it is a known thing that sometimes, queries go to DNS2 even if DNS1 is online.

If however DNS1 is down, all queries will go through DNS2.

20

Here is an alternate solution that doesn't require monitoring scripts - have both Pi-Holes serve DHCP on non-overlapping ranges. Clients can get an IP from either Pi-Hole. Set static IPs for the Pi-Hole at some low part of the IP range (<10). Pi1 DHCP range 50-100, Pi2 DHCP range 150-200, for example.

During the DHCP handshake, have each Pi-Hole provide the IP of the opposite Pi-Hole as a second DNS. Each client will now have the DNS for two Pi-Holes. There is no active load-balancing; clients will use either or both Pi-Holes as long as they are both up. If either Pi-Hole fails, the clients move seamlessly to the other.

1 Like