Problem of adding custom regex.list

Hi,
I have my own private list to block users.There are 13000 addresses.i want to add my regex.list.

if I add one by one it will take too long.So ,i created a script.
My blacklist.txt file ;

> televizyon-izle.gen.tr
> hdjustintvizle.tv
> izlemachd.com
> sonmodamodelleri.com
> dizimagazin.net
> zeytinoyun.com
> asoyun.net
> magazincafe.com
> gameofglam.com
.
.
.
end 13000 line

My script ;
more pihole_regex_custom_list.sh
> #!/bin/bash
> input="blacklist.txt"
> while IFS= read -r line
> do
> echo "(^|\.)${line//./\.}$"
> done < "$input"

i run " bash pihole_regex_custom_list.sh> /etc/pihole/regex.list"

My regex.list file is;

$^|\.)televizyon-izle\.gen\.tr
$^|\.)hdjustintvizle\.tv
$^|\.)izlemachd\.com
$^|\.)sonmodamodelleri\.com
$^|\.)dizimagazin\.net
$^|\.)zeytinoyun\.com
$^|\.)asoyun\.net
$^|\.)magazincafe\.com
$^|\.)gameofglam\.com
.
.
.

1)there should be no dollar signs at the beginning of the line ?
there must be a dollar sign at the end of the line?
2)there sholud be a parentheses sign at the early of the line ?

I spent a lot of time on this but I couldn’t solve it.Please Help me
Thank you very much for your help

Why are you doing this? If the domains already exist in your blacklist, there is no advantage to duplicating them as regex entries. 13,000 blacklist entries is a lot easier for Pi-Hole to process than 13,000 regex filters.

Indeed it is not very efficient.

The correct way of writing this Wildcard Regex:

(^|\.)zeytinoyun\.com$

Normally I would write anchor only one side but Wildcard is an exception because it can catch sub-domains and the main domain. The anchors are ^ and $.

jfp and msatter thank you.

jfp,my list(blacklist.txt) has nothing to do with the /etc/pihole/black.list
i want to use regex.list.
for example,i want to block cnn.com
Blacklist.txt block exact cnn.com
regex.list block cnn.com,www.cnn.com,wwwwwwww.cnn.com
So, regex.list most effective than blacklist.list.
I think we agree on this

msatter you are true.
The correct way of writing this Wildcard Regex:

(^|\.)zeytinoyun\.com$

if I add one by one it will take too long.So ,i created a script.
My script ;
more pihole_regex_custom_list.sh
> #!/bin/bash
> input="blacklist.txt"
> while IFS= read -r line
> do
> echo "(^|.)${line//./.}$"
> done < "$input"

When i run my script,output is

$^|.)zeytinoyun.com

there’s only one thing I want to ask.
what’s wrong in my script ?

Thank you very much for your help

Regex will become effective when you can replace 20 or more lines by one Regex.

When you past code then please use the </> button above because Discourse removes parts of your code.

I read from your result that the last $ is put on the next line and that is because of the \n (new-line) in the blacklist.txt. If it is a list created by a DOS/Windows computer than expect a/also /r (return).

Remove the \n from the line and add a new one behind $. Creating is simple, $" becomes $\n"

For removing you can use sed and you have to find that out for yourself how that is done.

msatter Thank you very much for your answer.
i can added to regex list.But another problem right now.

if i added from pihole web interface, my regex.list working.
if i added from command line, my regex.list not working.

i think,no permission problem.
-rw-rw-r-- 1 pihole www-data 160496 Eki 20 09:48 regex.list

i added from web interface “abcd.com.tr” .and i can not accessed to abcd.com.No problem.

pihole.log

root@johny:/etc/pihole# more /var/log/pihole.log| grep abcd.com.tr
Oct 20 09:22:26 dnsmasq[22394]: query[A] abcd.com.tr from 192.168.x.y
Oct 20 09:22:26 dnsmasq[22394]: /etc/pihole/regex.list abcd.com.tr is 0.0.0.0
Oct 20 09:22:39 dnsmasq[22394]: query[A] www.abcd.com.tr from 192.168.x.y
Oct 20 09:22:39 dnsmasq[22394]: /etc/pihole/regex.list www.abcd.com.tr is 0.0.0.0
Oct 20 09:24:48 dnsmasq[22394]: query[A] www.abcd.com.tr from 192.168.x.y
Oct 20 09:24:48 dnsmasq[22394]: /etc/pihole/regex.list www.abcd.com.tr is 0.0.0.0

But i added from command line “defg.com.tr” and i restarted pihole services,i can accessed to defg.com.tr :frowning:

pihole.log
root@johny:/etc/pihole# more /var/log/pihole.log| grep defg.com.tr

Oct 20 10:04:35 dnsmasq[5891]: query[A] defg.com.tr from 192.168.x.y
Oct 20 10:04:35 dnsmasq[5891]: forwarded defg.com.tr to 212.c.d.e
Oct 20 10:04:35 dnsmasq[5891]: forwarded defg.com.tr to 212.g.h.f

i uninstalled pihole,and then i installed pihole.But i took same error.

When i runned pihole -d,i saw this;
*** [ DIAGNOSING ]: contents of /var/log/lighttpd

-rw-r--r-- 1 www-data www-data 414 Eki 20 09:41 /var/log/lighttpd/error.log
   2019-10-20 07:35:05: (server.c.1242) logfiles cycled UID = 0 PID = 14138
   2019-10-20 08:09:01: (mod_fastcgi.c.2702) FastCGI-stderr: PHP Warning:  file_put_contents(/etc/pihole/regex.list): failed to open stream: Permission denied in /var/www/html/admin/scripts/pi-hole/php/func.php on line 51
   2019-10-20 09:41:03: (server.c.1558) server stopped by UID = 0 PID = 1
   2019-10-20 09:41:04: (log.c.164) server started

But the log time is 08:09:01. The time is now 10:11:00. ?

I don’t know if this error could be related to I can’t add regex from command line.

Shortly ;

if i added from command line, my regex.list not working.

Can you help me,Thank you very much for your help

What command and parameters are you using to add from the prompt?

When I use - -help on pihole then the command for me would be:

sudo pihole --wild abcd.com.tr

Check first with sudo pihole --help if --wild is available to you.

Remember you are adding Wildcards domains and that they are stored in Regex is for a technical reason.

If you use direct writing to the list then you have be specific about the dot in (^| \.) , and made sure the \ is present.

msatter

thank you very much for your patience.i have only one purpose.
i have a my own list (9800 line). i want to add all of it to regex.list.

root@johny:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.6 LTS
Release:        16.04
Codename:       xenial

root@johny:~# more blacklist.txt
televizyon-izle.gen.tr
hdjustintvizle.tv
izlemachd.com
sonmodamodelleri.com
dizimagazin.net
zeytinoyun.com
asoyun.net
magazincafe.com
gameofglam.com
.
.
.
9800 line

When i add form original command,i am not took an error
root@johny:~# pihole --regex televizyon-izle.gen.tr
[i] Adding televizyon-izle.gen.tr from regex list…

[i] Pi-hole blocking is enabled
[i] Using cached Event Horizon list…
[i] 1158034 unique domains trapped in the Event Horizon
[i] Number of blacklisted domains: 0
[i] Number of regex filters: 1

[✓] Restarting DNS service
[✓] DNS service is running
[✓] Pi-hole blocking is Enabled

root@johny:/etc/pihole# more regex.list
televizyon-izle.gen.tr

Oct 20 15:08:59 dnsmasq[26969]: query[A] www.televizyon-izle.gen.tr from 192.168.a.b
 Oct 20 15:08:59 dnsmasq[26969]: /etc/pihole/regex.list www.televizyon-izle.gen.tr is 0.0.0.0

No problem.

But ,When i copy my blacklist, i took an error

cp blacklist.txt /etc/pihole/regex.list

root@johny:~# ls -l /etc/pihole/regex.list
-rw-rw-r-- 1 pihole www-data 160496 Eki 20 15:11 /etc/pihole/regex.list
root@johny:/etc/pihole# more regex.list
televizyon-izle.gen.tr
hdjustintvizle.tv
izlemachd.com
sonmodamodelleri.com
dizimagazin.net
zeytinoyun.com
asoyun.net
magazincafe.com
i restarted pihole services.And log;

root@johny:/etc/pihole# more /var/log/pihole-FTL.log
[2019-10-20 15:13:03.621 27493] Compiled 9874 Regex filters and 0 whitelisted domains in 60.4 msec **(0 errors)**

But ;
When i query magazinkafe.com,the log is

Oct 20 15:15:37 dnsmasq[27493]: query[A] www.magazinkafe.com from 192.168.a.b
Oct 20 15:15:37 dnsmasq[27493]: forwarded www.magazinkafe.com to 212.g.h.j
Oct 20 15:15:38 dnsmasq[27493]: query[A] www.magazinkafe.com from 192.168.a.b

when i added the whole list (cp command ),The query bypassed regex.

I can’t add my list on the web.


Because ,the addresses adding side by side :((

Sorry, I couldn’t find a file with the regex command

root@johny:~# pihole --regex blacklist.txt
  [i] Adding blacklist.txt to regex list...

  [i] Pi-hole blocking is enabled
  [i] Using cached Event Horizon list...
  [i] 1158034 unique domains trapped in the Event Horizon
  [i] Number of blacklisted domains: 0
  [i] Number of regex filters: 3

  [✓] Restarting DNS service
  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled

This command insert the word(blacklist.txt),not the 9800 line in the file(blacklist.txt) :frowning:

i am going crazy. :frowning:

how do I add the list in my regex list.?

Thank you very much for your help

It seems that you believe in magic. This is not the case with Pihole.

You first have to do the hard work and then append the outcome to regex.list…and you add wildcard format into a regex so you first have to convert the list to Wildcard. Pihole --wilcard takes only one domain at the time.

To open your eyes add a Wildcard in a empty regex.list file. Using the webinterface and type on the prompt cat regex.list and ptess enter, do you see now the format used? It’s also in the webinterface visible.

msatter thank you.

I copied my custom list from the windows computer, not linux computer.
the regex filter worked properly. :slight_smile:
Thank you very much for your patience.

Then your first posting almost contained the correct code to do that when I had the chance to test it.

I have adapted it so that it works correctly.

#!/bin/bash
input="blacklist.txt"
echo
while IFS= read -r line
do
echo "(^|\.)${line//./\\.}$"
done < "$input"

Usage in the /etc/pihole directory:

pihole_regex_custom_list.sh >> regex.list

>> is appending to a file