This script takes an input file (blocksites.txt) containing links to blocklists (1 per line), retrieves the domains for all of the blocklists, and writes them to the output file (myblocklist.txt). An optional separate domain file (domainlist.txt) can be included in the output.
I uploaded the output domain file (myblocklist.txt) to my NAS device. One entry to pi-hole's blocklist table loads everything (370K+ domains) I want included with pi-hole's core blocklists.
This method keeps pi-hole's blocklist table clean and I don't have to mess around with any of the internal files. Also, I don't plan to update these added blocklists as often as pi-hole refreshes it's blocklists.
#!/bin/bash
###################################################################
#
# build.sh - build blocking domain list
#
# Inputs:
# blocksites.txt - sites providing block lists
# one per line
# domainlist.txt - Domains to be added to myblocklist.txt
# This file is optional
# Outputs:
# myblocklist.txt - List of all domains from blocksite servers
# and domainlist.txt
#
###################################################################
#
BLOCKSITES="blocksites.txt"
DOMAINLIST="domainlist.txt"
MYBLOCKLIST="myblocklist.txt"
{
echo "# Last updated: $(date)"
while IFS='\n' read -r URL; do # read block sites
{
echo "# From: ${URL}"
curl ${URL}
echo # add \n
} # capture server output
done < "$BLOCKSITES" # from site list
if [ -f "$DOMAINLIST" ]
then
cat "$DOMAINLIST"
fi
} > "$MYBLOCKLIST"
ToDo:
Sort unique to eliminate duplicates (may have to clean up domain names).
Load domains into database for more control.
Just a not: the upcoming pihole v5.0 supports per client blocking and can individually assign blocklists to (groups of) clients. With your approach you loose this flexibility.
This solution raises some red flags. Blocklists are controlled externally which could cause problems if an unwanted domain was added to the group blocklist. A better solution might be to have group whitelists.
The core pi-hole blocklists seem to be fairly good, my additional domains would most likely apply to all users.
This is definitely a work in progress - thanks for the feedback.
You make a good point. One thing I like about pihole is it's small cpu/memory footprint. As more and more features are added more horsepower will be required to run it. Seems to me an external process to filter out unwanted domains, as opposed to white listing for the purpose of overriding what is not wanted in the first place, might make the whole process more efficient.
I have seen other posts that indicate some interest in this capability or something similar.
Anyway, I get to exercise some skills that I rarely get to use.
Quite the opposite. Pi-hole V5.0 has many more features and an entirely different architecture for the blocklists, yet it runs on much less memory and with lower average CPU requirements.
I don't think you will be able to make this more efficient than our ace developer already has.
