Just sharing my blocklist; dbl.oisd.nl

Completed the run for 140K domains and found + 29K NXDOMAINS and will run tomorrow or later the re-check.

  [i] Pi-hole cleaner - reducing the number of domains in gravity.list

  [i] Creating regex cleaner file for gravity.list
  [â] 137 regex filters found
  [i] Applying filter, regex.clean to list.preEventHorizon
  [i] reducing the number of obsolete blocking domains.
  [i] 242865 (sub)domains are checked against regex.wildcard
  [â] 176302 (sub)domain left after applying filter, regex.clean

  [i] looking for sub.domains that can be used as wilcard when more than 20 of them
  [â] 258 wildcard domains found to be included in regex.list

  [i] Applying wildcard filtering
  [â] writing gravity.list with 138770 domains remaining

  [i] Applying NXDOMAIN filtering
  [â] writing gravity.list with 108914 domains remaining

Plot twist; I’ve checked the domains on my blocklist once before…
However… I still have the full list (raw unchecked domains)… over 4 million domains. I would be amazed if they’re all really dead.
I can provide you with a temp link, or mail it to you if you are interested.

That is huge list and it would takes weeks to check for NXDOMAIN domains. I use small lists and my the list I use is 450000 entries and when converting is ready, it is little over 100000 without loosing any of their workings.

I am now setting up the eco system to maintain the list. I have to collect the new domains, to be checked, and remove the removed domains from the NXDOMAIN list.

A consequence is that the blocklist can not vary to much in the number of lines. Because that could wipe out the build NXDOMAIN list. Or give a huge list to be checked.

I have all base files present so testing it will be easy…thanks to the utility comm.

ps. those 820 regex domains mentioned is really heavy on the processor so an other solution is needed. Maybe Mmotti’s way is more suitable for that.

The eco-system is also complete and new domains in blocklist are saved up and can be manually of automatic checked if put in a cron job.
Removed domains in the blocklist are also saved up and will be removed if present on the list with NXDOMAIN domains.

I have to some more testing and make it ‘user proof’ and like a old pump you first have to put water in to have it pumping water. You need a nxdomain.list so that this process is not stalling because it has to check lets say 4 million domains. :wink:

I will put it first live on my own Pihole and then put it here for the ones who want to use it.

You also can use the dig command with different DNS servers, like: dig www.google.com @1.1.1.1, www.google.com @8.8.4.4., www.google.com @8.8.8.8, etc.
In this case requests are spread.
DNS servers to be use: 1.1.1.1, 8.8.8.8, 8.8.4.4, 8.26.56.26 and 8.20.247.20 (Commodo), 208.67.222.222 and 208.67.220.220 (OpenDNS), 199.85.126.10 and 199.85.127.10 (Norton), OpenNIC (see https://www.opennic.org)
A list of public DNS servers: https://public-dns.info/. Pro country you can download a plain txt file with all their IP addresses of that country.

You have to make a routine that chose 1 of them from the list before doing the dig.
Using this ‘construction’ the PI can run at full speed.

I am using NXDOMAIN removing now live and having to still testing specific things. I had to adapted a lot to make it ‘user proof’ and is also near future proof for Pihole.

I am everytime impressed how fast, huge lists are checked or edited when using linux tools. And this on a RaspberryPI.

I was only checking for NXDOMAIN and I going to add SERVFAIL so that is also becovered.

Edited :smiley:
Gitlab: https://gitlab.com/ookangzheng/dbl-oisd-nl

I dont have gitlab account though you can host it ur self on gitlab and run a cron script to update every 24 hours ~~ :sunglasses:

I added Gitlab repo, Enjoy :sunglasses:

1 Like

Nice hosts list! i use the gitlab!

Can you add this hosts list to:
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
And
https://raw.githubusercontent.com/EnergizedProtection/block/master/extensions/xtreme/formats/domains.txt

The first is already included.
The second is not. Too many false positives.

Nice! and i found another good list (blocking the android tracks):
https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/android-tracking.txt

Not included because of it’s facebook entries.
Offtopic… You sure you wanna do this? Come up with 1 or 2 (new) lists you’ve found every time?

I have tried pihole for the first time with this host and there is something weird. I see "www.googletagservices.com Blocked (gravity)", but other side i see that this isn’t be blocked:
"www.googletagmanager.com OK (forwarded)"

I need to know why? Why dont you block this two google-trackers?

Thank you for pointing that out to me!
I seemed to be including a whitelist that had that entry in it.
Stopped including that whitelist.

Fixed. Please update gravity. :wink:

You need to block "www.google-analytics.com" to and maybe also "imasdk.googleapis.com"

They were whitelisted for a reason, but that seemed off.
Fixed, thanks :wink:

maybe you need to control the whitelist?

Stopped including the whitelists that had those entries in them.

Since this thread is now being used for support requests for this blocklist, I’ll close it to new comments with the OP’s concurrence.

The list maintainer has provided the following URL for support requests:

3 Likes