It is a pity that pihole still can't use Easylist or DNSCRYPT lists as they include wildcards.
Im surprised that pihole doesn't include filter :
Please provide URLs for some of the lists you reference.
Continuing the discussion from Adding Easylist, DNSCRYPT and others lists that contain more than hosts:
The desired behaviour is that pihole will parse the list and reformat wildcards and or add them to wildcard / regex. It doesn't
Whats worse is that in some cases pihole will add a domain with the wildcard to gravity and as not supported messes up gravity.
This means that can't use such lists without parsing manually first and have to keep separate DNSCRYPT lists
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/dnscrypt/extra.txt
"-----head of gravity.list------
*.2mdn.net
*.a-msedge.net
*.adnxs.com
.ads.msads.net"
https://raw.githubusercontent.com/tomasko126/easylistczechandslovak/master/filters.txt
Not possible, dns is only domain blocker.
Maybe you like this?:
Thanks I will give that a try - looks interesting and useful.
I realise its domain blocker but expected that the list combine and cleanup would remove unsupported wildcard entries but they appear in gravity. I suspect that the combine & clean up functions aren't working well in general. I did a test with a big list and saw no noticeable change in gravity.
Pi-Hole uses hosts format for the gravity list, and if your subscribed blocklists are in hosts format, you will have no problems. The developers have added some ability for Pi-Hole to import other list types, but there are many lists available online and some are poorly formatted, formatted for a completely different ad-blocking method, etc.
Thanks. There are two items here. Pihole ability to import non hosts format and the ability to combine & clean up hosts format. I have noticed that large lists in hosts format are not not added to gravity (I think Pihole can't parse large lists) and when there are items in file like *.domain they are not removed.
Size of the list doesn't matter, we parse everything line by line. And *.domain is non-hosts by definition.
I tested with a large list 14m (hosts with no visible non hosts) and gravity only ended up with 1m. I couldn't figure out what the combine and cleanup function was doing but will look when have more time. I briefly checked some unique domains that wouldn't be affected by regex rules in case that was used in the cleanup function but they were not in gravity.
Agree *.domain is non-hosts but often hosts files have such entries in error in particular when users have both DNScrypt lists and pihole.
What is the URL of the large list you loaded?
I cant send that one but I just tested using another large one and see the same results
Are you using https://github.com/... as your list entry, or https://raw.githubusercontent.com/... as your list entry?
https://raw.githubusercontent.com/no-replies/blocklist/master/test
dschaper@nanopineo:/etc/pihole$ sudo curl -o three.list https://raw.githubusercontent.com/no-replies/blocklist/master/test
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 17.3M 100 17.3M 0 0 9354k 0 0:00:01 0:00:01 --:--:-- 9352k
dschaper@nanopineo:/etc/pihole$ wc -l three.list
909512 three.list
dschaper@nanopineo:/etc/pihole$ ls -la three.list
-rw-r--r-- 1 root root 18195920 Mar 22 12:12 three.list
Thanks for taking a look. This one isn't as large obviously but its easier to work with; it had 909513 and 867809 ended up in pihole. Is there a log to see what entries were discarded and why?
I can see that there are some without tld so they were probably discarded but not sure how pihole decides what to discard.
Things like the below malformed domains will be dropped.
zzm34y09-450ee1f6b71c92955a045db1bc49f9dfa17376a8-
zzm34y09-5e1139559b70106bb5ce43c9d86fcd6f67571c83-
zzm34y09-60d530789d46127fd64c5bd992b7bd76196bd540-
zzm34y09-695462b647b5efcd1954c376f1dcb9e40432f08b-
zzm34y09-6a8f942a46ed38d92f695857b87f090023a6ba5a-
zzm34y09-6ad61236ed6b7260a2e8103ad680f842d6abd650-
zzm34y09-710bf2f3640a07da463b64aaf428d4730671ad8b-
zzm34y09-72587d812458531cae2a9eb528cd4994cbc7fc6c-
zzm34y09-78617f4b8c55de5e1c38f67da8627a6079a6a5d4-
zzm34y09-7acfa8feeda04522f5e3c759bf25b157709d450b-
zzm34y09-7d6452de6b34eba35c5e5588467a3366fa5f1b25-
zzm34y09-7dec7e87e413534d1ff9baeedf20b52cf6b60e89-
zzm34y09-8d85400216ad8b8462d08e6dd3933c296c58a165-
zzm34y09-929324027394b28c626543a0ae47862c788aff80-
zzm34y09-9303379552b2dc487b9a27bf73fa6d6dde529547-
zzm34y09-9a43e0bda39b1bbd748d6eadb0acfef8069fac86-
zzm34y09-ab8d4de3f924466016b2c9d4944dc9d4ff089ff1-
zzm34y09-afd03661b9c2937aee409b650b67e9462e759ef7-
zzm34y09-b0ee927b247238545d1fbfbdcb47454869433cf6-
zzm34y09-b6e115af92dcbabb00d69f692b50276c40aa1a83-
zzm34y09-b9048c8f1327d6f75bd6c2f84b76c6cf5e9fffb2-
zzm34y09-c17d1f4939d52cea8c8c90568692c385306bf177-
zzm34y09-c29098448d0730311c09b82c87fab6da00100d6d-
zzm34y09-d289ce40246f210e59b2460c28c1ea508e035324-
zzm34y09-d2f3f1e3c083df9834d97720782910e14294d00a-
zzm34y09-d45957bed49183edc4ac2e1734099a585d18ac1b-
zzm34y09-d98bc2ac68c470a8147c014e45683529740c4a6a-
zzm34y09-e13b40bc135ff58f0379b3dc599b2a8b6dfa7df0-
zzm34y09-e2346f4e57399e81c07f80fc43de239c7f024725-
zzm34y09-e572d5c2e3112f1dc9f44f7713afd58cc391d16a-
zzm34y09-f46edf3cbf2c599e0e4e4aa7316f4289e719da83-
zzm34y09-f5dba2ef5a246c35d96a273bebfe018f03e27ae9-
zzm34y09-f5e9b578631c5ce104c11e0568641ae2de3c0009-
zzm34y09-fe358c1e17e77f87cb77a65287790de6d6b1ee72-
yep I saw that.
Its hard to debug without having the log of what was removed & why
Open up the blocklist files before you import them, and see what's in them. If they have junk in them, Pi-Hole will try to remove it but cannot remove all junk.
Here is the code for the gravity script - you can see for yourself how it works:
1 Like
Thanks I will try to get to the bottom of it. It's definitely missing some junk and removing some that it shouldn't.
You should use lists that are properly formatted for Pi-Hole; this will resolve the problem.
1 Like
Thanks but that doesn't resolve the problems.