Multiple regex lists

regex
blacklisting

#1

I found a regex list on reddit, it’s in the first comment.

This list seems to change regularly, I also want to use my own list (converted wildcard list - pihole 3.x), so in order to combine these, without too much work, I wrote the following script.

The script uses a ram drive (reduce SD card writing) to process the lists, if You want to use a ram drive, you need to create it, before running the script. To create the ram drive (run this script only once):

#!/bin/bash

# Make sure only root can run our script
if [ "$(id -u)" != "0" ]; then
  echo "This script must be run as root" 1>&2
  exit 1
fi

# ramdrive
mkdir /home/pi/tmp
sudo sed -i '$ a tmpfs /home/pi/tmp tmpfs nodev,nosuid,size=64M 0 0' /etc/fstab

If you don’t want to use a ramdrive, simply change the paths…

In order to use my (converted wildcard) regex list in the script, I’ve put it in /var/www/html (the pihole webserver), so it can be downloaded by the script.

The script downloads both lists and combines the lists. It the copies the result to /etc/pihole and restarts pihole-FTL to process them

#!/bin/bash

# Make sure only root can run our script
if [ "$(id -u)" != "0" ]; then
  echo "This script must be run as root" 1>&2
  exit 1
fi

mount /home/pi/tmp
curl https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list -o /home/pi/tmp/regex1.txt
chmod 666 /home/pi/tmp/regex1.txt
echo >> /home/pi/tmp/regex1.txt
# this is my pihole's IP address, replace it with your own!!!
wget http://192.168.2.57/regex.list -O /home/pi/tmp/regex2.txt
cat /home/pi/tmp/regex1.txt /home/pi/tmp/regex2.txt > /home/pi/tmp/regex.txt
sort /home/pi/tmp/regex.txt | uniq -u > /home/pi/tmp/regex.list
sed -i '/^$/d' /home/pi/tmp/regex.list
sed -i '/^#/d' /home/pi/tmp/regex.list
mv /home/pi/tmp/regex.list /etc/pihole/regex.list
umount /home/pi/tmp
service pihole-FTL stop
service pihole-FTL start

notes:

  • to add a list, simply download it to the next available temp file (regex3.txt) and add it to the ‘cat’ line.
  • not all downloaded list end with a new line, so add an empty line to the list, using echo
  • automate the script by creating an new file in /etc/cron.d

I’m sure this script can be improved, works for me…
@DL6ER: Feature Request: Allow multiple rexex lists, download and process them with ‘pihole-g’


#2

This is not so simple as

is not necessarily sufficient to remove duplicates, e.g.

((^)|(\.))twitter.com
(^)|(\.)twitter.com

Both regex are perfectly valid any subject to the personal preference of the list’s creator.