# I concatenated every BlockList I could find

**URL:** <https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184>\
**Category:** Customizing Pi-hole\
**Tags:** blacklisting, raspbian\
**Created:** [October 19, 2017, 3:59am UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184 "2017-10-19T03:59:36Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [October 19, 2017, 3:59am UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/1 "2017-10-19T03:59:36Z")

</div>

I concatenated every BlockList I could find then transforming the lists to start with 0.0.0.0 and 127.0.0.1 depending on source.  
It took Over 50 Man hours to create and modify this list with the bulk of the lists provided by @WaLLy3K

[https://smokingwheels.github.io/Pi-hole/allhosts](https://smokingwheels.github.io/Pi-hole/allhosts)  
Includes InBrowser Mining and sites that run them. You may have to WhiteList a few sites.

 ![Screenshot from 2017-10-19 11-54-33](https://discourse.pi-hole.net/uploads/default/original/2X/b/b555dbdb15534ea41ac801c8c07a93e19a3b2bfa.png)

I am running on 2 Pi-Holes currently for the main use of my YaCy Search Servers.  
I am running 1 on a Raspbian 3B it consumes around 2 Watts on the power bill it cheaper than a cloud and more convenient.

The Actual number on a clean Pi-Hole is 978022 Domains on BlockList I may update with a few more like the pic after testing.

Updated and bigger list.  
[https://ncloud.zaclys.com/index.php/s/MPYSjLkb8cE8gPN/download](https://ncloud.zaclys.com/index.php/s/MPYSjLkb8cE8gPN/download)

---

<div class="post-metadata">

**Author:** ![WaLLy3K](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/wally3k/32/649_2.png) [@WaLLy3K](https://discourse.pi-hole.net/u/WaLLy3K)\
**Post date:** [October 19, 2017, 4:12am UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/2 "2017-10-19T04:12:06Z")

</div>

When hosting stuff like blocklists on GitHub, please make sure you _do not_ use `raw.githubusercontent.com` links. Since these links don't provide the `last-modified` header, they will get re-downloaded _every single time_ `pihole -g` is run (or each week automatically) even if there are no new changes. That's a waste of time and bandwidth 🙂

Since you've created a [GitHub Pages Website](https://pages.github.com), you can use the following URL which ensures only _updates_ are downloaded: [https://smokingwheels.github.io/Pi-hole/allhosts](https://smokingwheels.github.io/Pi-hole/allhosts)

---

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [October 19, 2017, 4:19am UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/3 "2017-10-19T04:19:14Z")

</div>

Ok done I have changed it, Thank you.

There must be some sort of delay in github because I tried it and got a 404 error soon after the first push.

I will make a note of that for the future.

There are three lists now I separated my YaCy Search Engine from the main list.  
I have added another list and room for 1 more.

Right click **_Copy Link Location_** and paste into your Pi-Hole at /admin/settings.php?tab=blocklists then save.  
When complete Save and Update.

1. [https://smokingwheels.github.io/Pi-hole/allhosts](https://smokingwheels.github.io/Pi-hole/allhosts)

2. [https://smokingwheels.github.io/Pi-hole/searchengine](https://smokingwheels.github.io/Pi-hole/searchengine)

3. [https://smokingwheels.github.io/Pi-hole/yacysearchengine](https://smokingwheels.github.io/Pi-hole/yacysearchengine)

4. [https://smokingwheels.github.io/Pi-hole/searchengine1](https://smokingwheels.github.io/Pi-hole/searchengine1)

5. [https://smokingwheels.github.io/Pi-hole/fromlog](https://smokingwheels.github.io/Pi-hole/fromlog)

6. [https://smokingwheels.github.io/Pi-hole/fromlogtosh](https://smokingwheels.github.io/Pi-hole/fromlogtosh)

Added 2 Lists from QB64 Pihole log extractor.  
[https://smokingwheels.github.io/qb64/piholelogext.bas](https://smokingwheels.github.io/qb64/piholelogext.bas)

If you want to add something to my list please contact me Via Twitter @smokingwheels or here.

---

<div class="post-metadata">

**Author:** ![frankrpi3](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/frankrpi3/32/1142_2.png) [@frankrpi3](https://discourse.pi-hole.net/u/frankrpi3)\
**Post date:** [October 20, 2017, 9:11am UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/4 "2017-10-20T09:11:39Z")

</div>

Hi,

as I am using several of your lists from The _Big Blocklist Collection_:  
There are also some linking to raw.githubusercontent.com...  
So can we use them, or not?

---

<div class="post-metadata">

**Author:** ![WaLLy3K](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/wally3k/32/649_2.png) [@WaLLy3K](https://discourse.pi-hole.net/u/WaLLy3K)\
**Post date:** [October 20, 2017, 10:33am UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/5 "2017-10-20T10:33:33Z")

</div>

Of course you can, but it's worth keeping in mind that they will always be re-downloaded even if there are no new changes 🙂

Not really an issue with small lists, but can be problematic when downloading lists over a couple of MB.

---

<div class="post-metadata">

**Author:** ![herrwusel](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/herrwusel/32/258_2.png) [@herrwusel](https://discourse.pi-hole.net/u/herrwusel)\
**Post date:** [October 22, 2017, 12:53pm UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/6 "2017-10-22T12:53:17Z")

</div>

@smokingwheels: Thanks for this awesome list (and @WaLLy3K, too of course).

---

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [October 22, 2017, 1:34pm UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/7 "2017-10-22T13:34:52Z")

</div>

Cool thanks @herrwusel  
You may have some `WhiteListing ` to do.  
Do you run it on a Pi?

---

<div class="post-metadata">

**Author:** ![herrwusel](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/herrwusel/32/258_2.png) [@herrwusel](https://discourse.pi-hole.net/u/herrwusel)\
**Post date:** [October 23, 2017, 5:35pm UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/8 "2017-10-23T17:35:16Z")

</div>

Yes, it runs on a rp2 right now

---

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [October 23, 2017, 6:15pm UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/9 "2017-10-23T18:15:59Z")

</div>

Cool thats good to know.  
Is it Raspbian pixel or headless?

---

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [October 24, 2017, 4:49pm UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/10 "2017-10-24T16:49:01Z")

</div>

My block lists are growing and the access to to micro SD card is getting a little higher than I am happy with, I might have to migrate to a HDD.  
[https://www.maketecheasier.com/boot-up-raspberry-pi-3-external-hard-disk/](https://www.maketecheasier.com/boot-up-raspberry-pi-3-external-hard-disk/)

The tool to check disk access stats is iotop.

**_sudo apt-get iotop_**

Set interval 10 Seconds.  
**_sudo iotop -d 10_**

 ![Screenshot from 2017-10-25 00-00-38](https://discourse.pi-hole.net/uploads/default/original/2X/2/272ef32b3f6ca9650fc25bcf668ec58215e94371.png)

---

<div class="post-metadata">

**Author:** ![herrwusel](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/herrwusel/32/258_2.png) [@herrwusel](https://discourse.pi-hole.net/u/herrwusel)\
**Post date:** [October 26, 2017, 8:15am UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/11 "2017-10-26T08:15:47Z")

</div>

It's raspbian headless. I had to do some whitelisting for Youtube to play videos but nothing else so far

---

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [October 26, 2017, 8:41am UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/12 "2017-10-26T08:41:35Z")

</div>

Thank you for the feed back.

Could you share just the URL's to do with Youtube?  
It maybe a country thing.

---

<div class="post-metadata">

**Author:** ![herrwusel](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/herrwusel/32/258_2.png) [@herrwusel](https://discourse.pi-hole.net/u/herrwusel)\
**Post date:** [October 26, 2017, 5:44pm UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/13 "2017-10-26T17:44:42Z")

</div>

I should have been more specific. Only on Android devices youtube videos didn't work.  
I whitelisted:

> `android.clients.google.com`  
> `clients1.google.com`

---

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [October 26, 2017, 5:57pm UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/14 "2017-10-26T17:57:44Z")

</div>

Thank you.  
I have unlisted them.  
🕶

---

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [October 29, 2017, 4:02am UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/15 "2017-10-29T04:02:45Z")

</div>

I have released My Personal Pi-Hole log extractor for you to have a look at your own logs.

[https://smokingwheels.github.io/qb64/piholelogext.bas](https://smokingwheels.github.io/qb64/piholelogext.bas)

Some of the domains I have found are added in searchengine1, released as experimental at this stage.

[https://smokingwheels.github.io/Pi-hole/searchengine1](https://smokingwheels.github.io/Pi-hole/searchengine1)

Happy Hunting. 🐾

---

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [November 1, 2017, 1:07pm UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/16 "2017-11-01T13:07:54Z")

</div>

Does anyone have a `public/common WhiteList` they would to share?  
I am building a second list.

 ![Screenshot from 2017-11-01 21-06-23](https://discourse.pi-hole.net/uploads/default/original/2X/9/91bf89d9b6b5bc6fd7794db4e59454c46d2cb5a0.png)

---

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [November 2, 2017, 10:24am UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/17 "2017-11-02T10:24:18Z")

</div>

I have an the log file error rate down a bit on the 2nd lists about 0.07% of domains are have Bad Names.  
I have split my list because it is too big to have in one on **`Github.com`**  
I have unlisted ~280 domains so I would welcome any feedback on that.  
I have not tried to load it on my RaspberryPi yet but I have it running on an old laptop.  
The Total content of the lists is around 140 mb's.

```auto
https://smokingwheels.github.io/Pi-hole/0nov17
https://smokingwheels.github.io/Pi-hole/1nov17
https://smokingwheels.github.io/Pi-hole/2nov17
https://smokingwheels.github.io/Pi-hole/3nov17
https://smokingwheels.github.io/Pi-hole/4nov17
https://smokingwheels.github.io/Pi-hole/5nov17
https://smokingwheels.github.io/Pi-hole/6nov17
https://smokingwheels.github.io/Pi-hole/7nov17
https://smokingwheels.github.io/Pi-hole/8nov17
https://smokingwheels.github.io/Pi-hole/9nov17
https://smokingwheels.github.io/Pi-hole/10nov17
https://smokingwheels.github.io/Pi-hole/11nov17
https://smokingwheels.github.io/Pi-hole/12nov17
https://smokingwheels.github.io/Pi-hole/13nov17
https://smokingwheels.github.io/Pi-hole/14nov17
https://smokingwheels.github.io/Pi-hole/15nov17
https://smokingwheels.github.io/Pi-hole/16nov17
https://smokingwheels.github.io/Pi-hole/17nov17
https://smokingwheels.github.io/Pi-hole/18nov17
https://smokingwheels.github.io/Pi-hole/19nov17
https://smokingwheels.github.io/Pi-hole/20nov17
https://smokingwheels.github.io/Pi-hole/21nov17
https://smokingwheels.github.io/Pi-hole/22nov17

This list has been removed. But if it grows again it will come back. Errors have been reduced.
https://smokingwheels.github.io/Pi-hole/23nov17

These lists have strange symbols on, I am not sure if the Pi-hole will accept them. 
https://smokingwheels.github.io/Pi-hole/fromlog
https://smokingwheels.github.io/Pi-hole/fromlogtosh
```

---

<div class="post-metadata">

**Author:** ![HellboyPi](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/h/51bf81/32.png) [@HellboyPi](https://discourse.pi-hole.net/u/HellboyPi)\
**Post date:** [November 2, 2017, 9:35pm UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/18 "2017-11-02T21:35:07Z")

</div>

> **[GitHub - anudeepND/whitelist: A simple tool to add commonly white listed domains...](https://github.com/anudeepND/whitelist)**
>
> A simple tool to add commonly white listed domains to your Pi-Hole setup.

---

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [November 3, 2017, 12:41am UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/19 "2017-11-03T00:41:13Z")

</div>

Thank you for that.  
That is one I use.  
Do you know how often it gets an update in past history?

---

<div class="post-metadata">

**Author:** ![HellboyPi](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/h/51bf81/32.png) [@HellboyPi](https://discourse.pi-hole.net/u/HellboyPi)\
**Post date:** [November 3, 2017, 1:23am UTC](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184/20 "2017-11-03T01:23:50Z")

</div>

Pretty often. You can view the update history of every list on github. AnudeepND whitelist [example](https://github.com/anudeepND/whitelist/commits/24b394c60166a7b0d8ef7e1a95fb44e38e46466a/whitelist.txt).

[Next page](https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184.md?page=2)