Good solution to automatically revert to "normal" if Pi Hole dies?

cron
bash
dhcp

#1

Hello!

I really like the idea having Pi Hole be the DNS resolver for my network because it “automatically” covers the plethora of devices around the house all at once.

HOWEVER, I just know Murphy’s Law will strike eventually: the Pi Hole will die when I’m away on a business trip and no one in my household will be able to fix the Pi Hole device, or login into the main router to set the DNS setting back to 8.8.8.8 to restore internet visibility until I get home to sort things out.

Hence, I ask the community: is there a good solution to automatically revert to “normal” DNS resolution if Pi Hole dies?

For example:

  • Can a router with Tomato or some other popular 3rd party OSS be set to periodically check Pi Hole’s “health”, and “revert” if the Pi Hole gets sick?

  • I have two BeagleBone Black Rev C’s (BBB) running various home automation things. The point is that I plan to have one BBB run Pi Hole, and I could setup the other BBB to act as a watchdog over Pi Hole’s health on BBB 1. As such, any recos for implementing such a watchdog scheme?

Help!


#2

It should certainly be possible with the gear you describe.

You could set up a script on one (or both, why not?) of your BBBs and have them test dns connectivity periodically, and if it goes down for longer than you consider acceptable change
the router settings away from the Pi-hole.

My suggestion would be to have the script connect to your router by ssh and change the router DNS settings back away from your Pi-hole that way. (If your router supports using ssh keys for authentication all the better).

I’d be wary of having the scripts too eager to change settings like that, so I would also add some sanity checking, such as ensuring connectivity in general is still in place, and that 8.8.8.8 is reliably reachable, and returning sensible numbers before changing back to it.

A less automated option, and also handy to have around as plan C if plan B above fails, but involving some participation by a household member: Have a saved copy of your router configuration but with 8.8.8.8 as the DNS saved on one (or more) of their computers, and show them how to upload that to the router. That way they don’t need to worry about what what settings to change, or anything like that. Your instructions for them would be as simple as Click bookmark, type username & password, go to “configuration”, click “restore setting”, select this file, click yes.


How to handle a pi-hole failure / crash?
#3

I run a secondary instance of pi-hole (as a backup Pi-hole DNS in case the main one fails).

You could do the same thing with the BBBs run DNS1 on 1 and DNS2 on the second (I think they are capable of doing that from a HW perspective).

In my case DNS1 is also the DHCP server.

DNS2, monitors DNS1 for network presence and (possible) FTLDNS errors. In both cases or any (where DNS1 is up but FTLDSN has an error), DNS2 takes over from a DHCP (and DNS) perspective.

It’s as @robgill said, a heartbeat script (executed every 1 minute) that checks the status.

If it’s something you want to pursuit i can share relevant snippets of the script code that will help you out.

You could also do this :slight_smile:


#4

I would be interested in this code. I run two Pi’s in parallel (one for my router and the 25 clients on that), the other just for this computer), and I would like to be able to configure one to take over from other in event of failure.


#5

Code is a little messy.
Change parameters accordingly:

#!/bin/bash
target=192.168.1.3
count=$( ping -c 3 -w 3 $target | grep icmp* | wc -l )
dir=/media/Network/
 
probe=$(curl -s http://${target}/admin/ | grep offline)
check=$(echo $probe)
 
#check for reply
if [ $count -eq 0 ]
then
#do dhcp
        echo "Main DHCP server is not responding!"
#check for lockfile
        if [ -e ${dir}dhcp.on ]
        then
        echo "DHCP server already enabled. No changes or notifications performed."
        exit 0
else
        echo "Generating lock file"
#generate lockfile to prevent double notifications
        touch ${dir}dhcp.on
        echo "Done."
#Enable DHCP A B C D E. A=Range start, B=Range end, C=Gateway D=Lease Time E=Domain
#        sudo pihole -a enabledhcp "192.168.1.200" "192.168.1.251" "192.168.1.1" "1h" "local"
#FLTDNS crashes if hour string is specified under D
        sudo pihole -a enabledhcp "192.168.1.200" "192.168.1.251" "192.168.1.1" "1" "local"
#remove possible existing configuration files
        sudo rm -rf /etc/dnsmasq.d/02-pihole-dhcp.conf
        sudo rm -rf /etc/dnsmasq.d/03-pihole-wildcard.conf
        sudo rm -rf /etc/dnsmasq.d/04-pihole-static-dhcp.conf
        fi
 
#check for 02-pihole-dhcp.conf
if [ -e /etc/dnsmasq.d/02-pihole-dhcp.conf ]
then
    echo "02-pihole-dhcp.conf present. No changes made."
else
    echo "02-pihole-dhcp.conf missing. Generating file."
 
#generate file and pass strings
sudo touch /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo grep -q -F 'dhcp-authoritative' /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo sh -c "echo 'dhcp-authoritative' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
sudo grep -q -F 'dhcp-range=192.168.1.200,192.168.1.251,1h' /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo sh -c "echo 'dhcp-range=192.168.1.200,192.168.1.251,1h' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
sudo grep -q -F 'dhcp-option=option:router,192.168.1.1' /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo sh -c "echo 'dhcp-option=option:router,192.168.1.1' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
sudo grep -q -F 'dhcp-leasefile=/etc/pihole/dhcp.leases' /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo sh -c "echo 'dhcp-leasefile=/etc/pihole/dhcp.leases' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
sudo grep -q -F 'domain=local' /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo sh -c "echo 'domain=local' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
sudo grep -q -F 'dhcp-option=6,192.168.1.5,192.168.1.3' /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo sh -c "echo 'dhcp-option=6,192.168.1.5,192.168.1.3' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
echo "Done."
fi
 
#check for 03-pihole-wildcard.conf
if [ -e /etc/dnsmasq.d/03-pihole-wildcard.conf ]
then
    echo "03-pihole-wildcard.conf present. No changes made."
else
    echo "03-pihole-wildcard.conf missing. Generating file."
 
#generate file (no strings to pass)
sudo touch /etc/dnsmasq.d/03-pihole-wildcard.conf
echo "Done."
fi
 
#check for 04-pihole-static-dhcp.conf
if [ -e /etc/dnsmasq.d/04-pihole-static-dhcp.conf ]
then
    echo "04-pihole-static-dhcp.conf present. No changes made."
else
    echo "04-pihole-static-dhcp.conf missing. Generating file."
 
#generate file
sudo touch /etc/dnsmasq.d/04-pihole-static-dhcp.conf
 
#pass strings in order for existing dhcp clients to keep IPs
sudo grep -q -F 'dhcp-host=00:00:00:00:00:00,192.168.1.2,StaticHost1' /etc/dnsmasq.d/04-pihole-static-dhcp.conf
sudo sh -c "echo 'dhcp-host=00:00:00:00:00:00,192.168.1.2,StaticHost1' >> /etc/dnsmasq.d/04-pihole-static-dhcp.conf"
echo "Done."
 
#restart dns and activate DHCP server
echo "Restarting DNS."
pihole restartdns
echo "Done."
 
#notify
echo "Notification sent."
fi

else
#check for FTLDNS error via web
 if [ $count -eq 3 ] && [[ $check == *"offline"* ]]
 
    then
        echo "Pi-hole is up with FTLDNS error!"
#check for lockfile
        if [ -e ${dir}dhcp.on ]
        then
        echo "DHCP server already enabled. No changes or notifications performed."
        exit 0
  else
        echo "Generating lock file"
#generate lockfile to prevent double notifications
        touch ${dir}dhcp.on
        echo "Done."
#Enable DHCP A B C D E. A=Range start, B=Range end, C=Gateway D=Lease Time E=Domain
#        sudo pihole -a enabledhcp "192.168.1.200" "192.168.1.251" "192.168.1.1" "1h" "local"
#FLTDNS crashes if hour string is specified under D
        sudo pihole -a enabledhcp "192.168.1.200" "192.168.1.251" "192.168.1.1" "1" "local"
#remove possible existing configuration files
        sudo rm -rf /etc/dnsmasq.d/02-pihole-dhcp.conf
        sudo rm -rf /etc/dnsmasq.d/03-pihole-wildcard.conf
        sudo rm -rf /etc/dnsmasq.d/04-pihole-static-dhcp.conf
        fi
 
#check for 02-pihole-dhcp.conf
 if [ -e /etc/dnsmasq.d/02-pihole-dhcp.conf ]
 then
    echo "02-pihole-dhcp.conf present. No changes made."
 else
    echo "02-pihole-dhcp.conf missing. Generating file."
 
#generate file and pass strings
    sudo touch /etc/dnsmasq.d/02-pihole-dhcp.conf
    sudo grep -q -F 'dhcp-authoritative' /etc/dnsmasq.d/02-pihole-dhcp.conf
    sudo sh -c "echo 'dhcp-authoritative' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
    sudo grep -q -F 'dhcp-range=192.168.1.200,192.168.1.251,1h' /etc/dnsmasq.d/02-pihole-dhcp.conf
    sudo sh -c "echo 'dhcp-range=192.168.1.200,192.168.1.251,1h' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
    sudo grep -q -F 'dhcp-option=option:router,192.168.1.1' /etc/dnsmasq.d/02-pihole-dhcp.conf
    sudo sh -c "echo 'dhcp-option=option:router,192.168.1.1' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
    sudo grep -q -F 'dhcp-leasefile=/etc/pihole/dhcp.leases' /etc/dnsmasq.d/02-pihole-dhcp.conf
    sudo sh -c "echo 'dhcp-leasefile=/etc/pihole/dhcp.leases' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
    sudo grep -q -F 'domain=local' /etc/dnsmasq.d/02-pihole-dhcp.conf
    sudo sh -c "echo 'domain=local' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
    sudo grep -q -F 'dhcp-option=6,192.168.1.5,192.168.1.3' /etc/dnsmasq.d/02-pihole-dhcp.conf
    sudo sh -c "echo 'dhcp-option=6,192.168.1.5,192.168.1.3' >> /etc/dnsmasq.d/02-pihole-dhcp.conf"
    echo "Done."
 fi
 
#check for 03-pihole-wildcard.conf
 if [ -e /etc/dnsmasq.d/03-pihole-wildcard.conf ]
then
    echo "03-pihole-wildcard.conf present. No changes made."
else
    echo "03-pihole-wildcard.conf missing. Generating file."
 
#generate file (no strings to pass)
sudo touch /etc/dnsmasq.d/03-pihole-wildcard.conf
echo "Done."
 fi
 
#check for 04-pihole-static-dhcp.conf
 if [ -e /etc/dnsmasq.d/04-pihole-static-dhcp.conf ]
then
    echo "04-pihole-static-dhcp.conf present. No changes made."
else
    echo "04-pihole-static-dhcp.conf missing. Generating file."
 
#generate file
sudo touch /etc/dnsmasq.d/04-pihole-static-dhcp.conf
 
#pass strings in order for existing dhcp clients to keep IPs
sudo grep -q -F 'dhcp-host=00:00:00:00:00:00,192.168.1.2,StaticHost1' /etc/dnsmasq.d/04-pihole-static-dhcp.conf
sudo sh -c "echo 'dhcp-host=00:00:00:00:00:00,192.168.1.2,StaticHost1' >> /etc/dnsmasq.d/04-pihole-static-dhcp.conf"
echo "Done."
  
#restart dns and activate DHCP server
echo "Restarting DNS."
pihole restartdns
echo "Done."
 
#notify
echo "Notification sent."
exit 0
 fi
else
    echo "DNS1 is Alive!"
 
#remove DHCP settings.
 
 if [ -e /etc/dnsmasq.d/02-pihole-dhcp.conf ]
 then
echo "Removing settings."
sudo rm -rf /etc/dnsmasq.d/02-pihole-dhcp.conf
sudo rm -rf /etc/dnsmasq.d/03-pihole-wildcard.conf
sudo rm -rf /etc/dnsmasq.d/04-pihole-static-dhcp.conf
 
#restart pihole / disable DNS seerver
echo "Restarting DNS."
sudo pihole -a disabledhcp
sudo pihole restartdns
sudo rm -rf ${dir}dhcp.on
echo "Done."
 
#notify
echo "Notification sent."
else
    echo "Lock file already deleted. No action performed."
 fi
 fi
#remove DHCP settings.
fi

Run this via crontab every minute and it will take over in case of failure on target.


#6

Thanks much.


#7

Thanks robgill, RamSet, jfb!!!

Question regarding robgill’s statement:

Putting aside whether one of my two BBBs or my router would randomly go berserk and mess up the network traffic or something:

The implication from robgill’s note is that a lazy person (like Yours Truly) could simply setup and run identically configured Pi Hole instances on each of my BBBs, and neither program would conflict with each other. Plus, if an asteroid hit only one of the BBBs, the other BBB would be none the wiser and happily continue to run its Pi Hole instance. True? False?

If this crazy idea is true, which PiHole+BBB would do the bulk of the work during normal, parallel operation?


#8

That’s @jfb’s question :slight_smile:

This statement is correct as long as you run both as DNS resolver (and not both as DHCP servers). It would be correct in the case of them being DHCP servers when you would have them set up with different settings.
However running 2 DHCP servers in the same LAN for the same class of IP’s would be really bad practice (unless your network topology configuration would require it ¯\(ツ)/¯ ).

I think in your setup, one router and two BBBs with standalone Pi-hole installations (as DNS only) will work wonders :slight_smile:

One thing to keep in mind:
In your DHCP server (I’m assuming the router does that) you would have to specify BBB1’s IP and BB2’s IP as Primary and Secondary DNS settings (so that all the clients that get an IP from it, will get the correct DNS settings).

If that doesn’t work (no option under DHCP server for DNS servers), you could set it up under your WAN connection’s DNS (same BBB1’s IP and BB2’s IP as Primary and Secondary DNS) but that will only show queries in the query log on the Pi-hole Admin interface as originating from your router’s IP and not the client’s IP (since the client would request from the router and the router request from the pi-hole).

You could also manually specify DNS1 and DNS2 on your clients. That will make logging on the Pi-hole a lot easier.

That would be DNS1 but DNS2 is not there for backup/failsafe only.

Some queries will bleed through the secondary DNS.

That’s why if you have DNS1=Pi-hole and DNS2=Google, some site (at random) will show ads sometimes and most times, it won’t.

If you Have DNS1=BBB1 Pi-hole and DNS2=BBB2 Pi-hole, you’re set :slight_smile:

and if DNS1=BBB1's Pi-hole is offline, DNS2=BBB2 Pi-hole will take the full load ('till DNS1 is back up).


#9

In my setup, with two Apple routers (one a wired extender to the other), the vast majority of my traffic goes to the primary PiHole. The primary Apple router is the DHCP server, the second is in bridge mode. DNS addresses are listed in the Apple router as pihole primary in the first slot, pihole secondary in the second slot. All the network devices get their DNS address from the router. In the last 24 hours, the primary PiHole received 31,600 queries, the secondary received 400 queries.

Interestingly, 300 of the queries to the secondary come from a Sense home energy monitor. This device seems to be DNS agile and will find any DNS server - it has settled on the secondary.

I like this setup. I can change things on the individual Pi’s - updates, etc, and when one is offline for reboot or such, I get no sneak-through ads. I do occasionally sync the white lists through the teleporter.

Note - with this setup, the queries to the PiHoles are logged by individual IP addresses, not shown as coming from the router only. That’s probably an Apple thing, but I like that behavior.


#10

Update:
TL;DR - I couldn’t get Pi-Hole running on my BBB, so I bought Pi Zero W’s and Pi-Hole dropped right in

First, thanks to the tips on this forum, the https://appliedcaffeine.org/pi-hole-on-beagleboard.html post, and some Reddit posts on this, I came really, really close to getting Pi-Hole running on my BBB. Specifically, the webpage GUI comes up, but FTL refuses to start. All I get is “Lost” “Connection” “To” “API” in the summary blocks at the top of the Dashboard. I suspect the dnsmasq or something is mis-configured, but who knows? The point is: after spending two evenings on this, I began wondering if I wasn’t just trying to fit a square peg in a round hole. In my frustration, I decided to check how much new Raspberry Pi’s cost these days, and before you know it a new Pi Zero W was on it’s way to me via Amazon. I got it working with Pi-Hole the evening it arrived. Then, as per the guidance on this forum, I bought another Pi Zero W and quickly brought it up with its own instance of Pi-Hole.

(I haven’t told my two BBB’s about their new neighbors yet – I’m sure I’ll come up with some house-wide scheme to use all four computers at once for some common purpose.)

Anyway, with the two Pi’s running in parallel now, it’s interesting the variance of traffic/blocking “splits” between the first Pi-Hole I installed and the second one – anything from 95/5 to 70/30. Tomorrow when the fam is out of the house I’ll shutdown the original completely and confirm the second Pi seamlessly picks up the whole load.

Finally, thanks to everyone that contributed to this thread!!!


#11

Edited to add, regarding:
In your DHCP server (I’m assuming the router does that) you would have to specify BBB1’s IP and BB2’s IP as Primary and Secondary DNS settings (so that all the clients that get an IP from it, will get the correct DNS settings).

For future reference, this is exactly what I’m doing: I’ve setup static IPs for each Pi running Pi-Hole, and on my router’s DHCP server config page I set Primary DNS to Pi-1’s static IP address and the secondary DNS to Pi-2’s. It’s been working like charm – all the phones, DVRs, tablets, BBBs, personal and work laptops (even with the work VPN) are working fine.

Again, many thanks to everyone that contributed to this thread!!!


#12

Update: “fail over” works just fine!

I finally had a window of time where everyone was out of the house, so I shutdown Pi-1 and Pi-2 did just fine on its own. (I suppose this really says more about the Ubiquity router’s ability to seamlessly fail over to the Secondary DNS spec.)