Problem of adding custom regex.list

secooonder · October 18, 2019, 5:27pm

Hi,
I have my own private list to block users.There are 13000 addresses.i want to add my regex.list.

if I add one by one it will take too long.So ,i created a script.
My blacklist.txt file ;

> televizyon-izle.gen.tr
> hdjustintvizle.tv
> izlemachd.com
> sonmodamodelleri.com
> dizimagazin.net
> zeytinoyun.com
> asoyun.net
> magazincafe.com
> gameofglam.com
.
.
.
end 13000 line

My script ;
more pihole_regex_custom_list.sh
> #!/bin/bash
> input="blacklist.txt"
> while IFS= read -r line
> do
> echo "(^|\.)${line//./\.}$"
> done < "$input"

i run " bash pihole_regex_custom_list.sh> /etc/pihole/regex.list"

My regex.list file is;

$^|\.)televizyon-izle\.gen\.tr
$^|\.)hdjustintvizle\.tv
$^|\.)izlemachd\.com
$^|\.)sonmodamodelleri\.com
$^|\.)dizimagazin\.net
$^|\.)zeytinoyun\.com
$^|\.)asoyun\.net
$^|\.)magazincafe\.com
$^|\.)gameofglam\.com
.
.
.

1)there should be no dollar signs at the beginning of the line ?
there must be a dollar sign at the end of the line?
2)there sholud be a parentheses sign at the early of the line ?

I spent a lot of time on this but I couldn't solve it.Please Help me
Thank you very much for your help

jfb · October 18, 2019, 5:56pm

Why are you doing this? If the domains already exist in your blacklist, there is no advantage to duplicating them as regex entries. 13,000 blacklist entries is a lot easier for Pi-Hole to process than 13,000 regex filters.

secooonder · October 19, 2019, 4:06am

jfp and msatter thank you.

jfp,my list(blacklist.txt) has nothing to do with the /etc/pihole/black.list
i want to use regex.list.
for example,i want to block cnn.com
Blacklist.txt block exact cnn.com
regex.list block cnn.com,www.cnn.com,wwwwwwww.cnn.com
So, regex.list most effective than blacklist.list.
I think we agree on this

msatter you are true.
The correct way of writing this Wildcard Regex:

(^|\.)zeytinoyun\.com$

if I add one by one it will take too long.So ,i created a script.
My script ;
more pihole_regex_custom_list.sh
> #!/bin/bash
> input="blacklist.txt"
> while IFS= read -r line
> do
> echo "(^|.)${line//./.}$"
> done < "$input"

When i run my script,output is

$^|.)zeytinoyun.com

there's only one thing I want to ask.
what's wrong in my script ?

Thank you very much for your help

secooonder · October 20, 2019, 7:12am

msatter Thank you very much for your answer.
i can added to regex list.But another problem right now.

if i added from pihole web interface, my regex.list working.
if i added from command line, my regex.list not working.

i think,no permission problem.
-rw-rw-r-- 1 pihole www-data 160496 Eki 20 09:48 regex.list

i added from web interface "abcd.com.tr" .and i can not accessed to abcd.com.No problem.

pihole.log

root@johny:/etc/pihole# more /var/log/pihole.log| grep abcd.com.tr
Oct 20 09:22:26 dnsmasq[22394]: query[A] abcd.com.tr from 192.168.x.y
Oct 20 09:22:26 dnsmasq[22394]: /etc/pihole/regex.list abcd.com.tr is 0.0.0.0
Oct 20 09:22:39 dnsmasq[22394]: query[A] www.abcd.com.tr from 192.168.x.y
Oct 20 09:22:39 dnsmasq[22394]: /etc/pihole/regex.list www.abcd.com.tr is 0.0.0.0
Oct 20 09:24:48 dnsmasq[22394]: query[A] www.abcd.com.tr from 192.168.x.y
Oct 20 09:24:48 dnsmasq[22394]: /etc/pihole/regex.list www.abcd.com.tr is 0.0.0.0

But i added from command line "defg.com.tr" and i restarted pihole services,i can accessed to defg.com.tr

pihole.log
root@johny:/etc/pihole# more /var/log/pihole.log| grep defg.com.tr

Oct 20 10:04:35 dnsmasq[5891]: query[A] defg.com.tr from 192.168.x.y
Oct 20 10:04:35 dnsmasq[5891]: forwarded defg.com.tr to 212.c.d.e
Oct 20 10:04:35 dnsmasq[5891]: forwarded defg.com.tr to 212.g.h.f

i uninstalled pihole,and then i installed pihole.But i took same error.

When i runned pihole -d,i saw this;
*** [ DIAGNOSING ]: contents of /var/log/lighttpd

-rw-r--r-- 1 www-data www-data 414 Eki 20 09:41 /var/log/lighttpd/error.log
   2019-10-20 07:35:05: (server.c.1242) logfiles cycled UID = 0 PID = 14138
   2019-10-20 08:09:01: (mod_fastcgi.c.2702) FastCGI-stderr: PHP Warning:  file_put_contents(/etc/pihole/regex.list): failed to open stream: Permission denied in /var/www/html/admin/scripts/pi-hole/php/func.php on line 51
   2019-10-20 09:41:03: (server.c.1558) server stopped by UID = 0 PID = 1
   2019-10-20 09:41:04: (log.c.164) server started

But the log time is 08:09:01. The time is now 10:11:00. ?

I don't know if this error could be related to I can't add regex from command line.

Shortly ;

if i added from command line, my regex.list not working.

Can you help me,Thank you very much for your help

secooonder · October 20, 2019, 12:25pm

msatter

thank you very much for your patience.i have only one purpose.
i have a my own list (9800 line). i want to add all of it to regex.list.

root@johny:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.6 LTS
Release:        16.04
Codename:       xenial

root@johny:~# more blacklist.txt
televizyon-izle.gen.tr
hdjustintvizle.tv
izlemachd.com
sonmodamodelleri.com
dizimagazin.net
zeytinoyun.com
asoyun.net
magazincafe.com
gameofglam.com
.
.
.
9800 line

When i add form original command,i am not took an error
root@johny:~# pihole --regex televizyon-izle.gen.tr
[i] Adding televizyon-izle.gen.tr from regex list...

[i] Pi-hole blocking is enabled
[i] Using cached Event Horizon list...
[i] 1158034 unique domains trapped in the Event Horizon
[i] Number of blacklisted domains: 0
[i] Number of regex filters: 1

[✓] Restarting DNS service
[✓] DNS service is running
[✓] Pi-hole blocking is Enabled

root@johny:/etc/pihole# more regex.list
televizyon-izle.gen.tr

Oct 20 15:08:59 dnsmasq[26969]: query[A] www.televizyon-izle.gen.tr from 192.168.a.b
 Oct 20 15:08:59 dnsmasq[26969]: /etc/pihole/regex.list www.televizyon-izle.gen.tr is 0.0.0.0

No problem.

But ,When i copy my blacklist, i took an error

cp blacklist.txt /etc/pihole/regex.list

root@johny:~# ls -l /etc/pihole/regex.list
-rw-rw-r-- 1 pihole www-data 160496 Eki 20 15:11 /etc/pihole/regex.list

root@johny:/etc/pihole# more regex.list
televizyon-izle.gen.tr
hdjustintvizle.tv
izlemachd.com
sonmodamodelleri.com
dizimagazin.net
zeytinoyun.com
asoyun.net
magazincafe.com

i restarted pihole services.And log;

root@johny:/etc/pihole# more /var/log/pihole-FTL.log
[2019-10-20 15:13:03.621 27493] Compiled 9874 Regex filters and 0 whitelisted domains in 60.4 msec **(0 errors)**

But ;
When i query magazinkafe.com,the log is

Oct 20 15:15:37 dnsmasq[27493]: query[A] www.magazinkafe.com from 192.168.a.b
Oct 20 15:15:37 dnsmasq[27493]: forwarded www.magazinkafe.com to 212.g.h.j
Oct 20 15:15:38 dnsmasq[27493]: query[A] www.magazinkafe.com from 192.168.a.b

when i added the whole list (cp command ),The query bypassed regex.

I can't add my list on the web.

Because ,the addresses adding side by side :((

Sorry, I couldn't find a file with the regex command

root@johny:~# pihole --regex blacklist.txt
  [i] Adding blacklist.txt to regex list...

  [i] Pi-hole blocking is enabled
  [i] Using cached Event Horizon list...
  [i] 1158034 unique domains trapped in the Event Horizon
  [i] Number of blacklisted domains: 0
  [i] Number of regex filters: 3

  [✓] Restarting DNS service
  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled

This command insert the word(blacklist.txt),not the 9800 line in the file(blacklist.txt)

i am going crazy.

how do I add the list in my regex list.?

Thank you very much for your help

secooonder · October 20, 2019, 2:04pm

msatter thank you.

I copied my custom list from the windows computer, not linux computer.
the regex filter worked properly.
Thank you very much for your patience.

Gordon_Freeman · January 23, 2020, 9:01am

Hi, guys!
If I understood correctly the way for blocking 'yandex' and 'vkontakte' next resources:

https://mc.yandex.ru/metrika/tag.js
JavaScript-библиотеки — Технологии Яндекса
JavaScript-библиотеки — Технологии Яндекса
https://vkontakte.ru/js/api/openapi.js

(^|\.)yandex\.ru$
(^|\.)vkontakte\.ru$

Am I right or need to change something? Because regarding usual regex it should be:
.*yandex.ru.*
For checking I used https://regex101.com/

jfb · January 23, 2020, 1:14pm

The reliable and easy method is to enter yandex.ru as a wildcard and let Pi-Hole create the regex. In this case, Pi-Hole will create the regex you have shown, so you have it correct.

(^|\.)yandex\.ru$

Wolfpat · August 17, 2022, 2:26pm

i have a alpine docker, and i didn't want to install anything on it
so i came up with this trick

basically go to the devtools and make a variable called d, that contains the list of regexes, seperated by a new-line

then you use this command

d.split`\n`.map(e=>`"${e}"`).join(' ');

it basically turn it into a one line text that is acceptable by pihole's regex.

then all you do is
right click on the result text and press on "copy string content"(chrome).

then finally you go to your console and type

docker exec pihole_pihole_1 pihole --regex YOUR_COPIED_TEXT_HERE

note that pihole_pihole_1 is my container name,
you can find yours if you enter docker ps

i hope this made your life a little bit easier