# Need a host for 75 million blocklist please

**URL:** <https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992>\
**Category:** Customizing Pi-hole\
**Created:** [March 16, 2024, 9:50am UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992 "2024-03-16T09:50:35Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [March 16, 2024, 9:50am UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/1 "2024-03-16T09:50:35Z")

</div>

Looking for someone to host a 3 Gb yacyblocklist online to share with other pihole users please.

If anybody knows a site that I could use that would be good to.

 ![Screenshot from 2024-03-16 17-26-47](https://discourse.pi-hole.net/uploads/default/original/3X/2/7/274d3449399655e8be7c495b749162e6f2155eef.png)

---

<div class="post-metadata">

**Author:** ![chrislph](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/chrislph/32/32723_2.png) [@chrislph](https://discourse.pi-hole.net/u/chrislph)\
**Post date:** [March 16, 2024, 11:21am UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/2 "2024-03-16T11:21:56Z")

</div>

Out of interest, are you running on a Raspberry Pi? If so, how long does it take to run a Gravity update with that list?

---

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [March 16, 2024, 11:33am UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/3 "2024-03-16T11:33:14Z")

</div>

> [@chrislph](#):
>
> Raspberry Pi?

It takes approx 11 mins on a HP 360DL gen 8 server.  
I have loaded a 9 million block list on raspberry pi 3B, from memory, it took in the order of 1 hour so, going by that possibly 8 hours.

---

<div class="post-metadata">

**Author:** ![bkbartje](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/bkbartje/32/24655_2.png) [@bkbartje](https://discourse.pi-hole.net/u/bkbartje)\
**Post date:** [March 16, 2024, 6:55pm UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/4 "2024-03-16T18:55:06Z")

</div>

github?  
or is there a file limit  
nm:  
it's not for free a file that large

> **[About Git Large File Storage - GitHub Docs](https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-git-large-file-storage)**
>
> GitHub limits the size of files allowed in repositories. To track files beyond this limit, you can use Git Large File Storage.

---

<div class="post-metadata">

**Author:** ![buntspexxt](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/b/e9bcb4/32.png) [@buntspexxt](https://discourse.pi-hole.net/u/buntspexxt)\
**Post date:** [March 16, 2024, 7:20pm UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/5 "2024-03-16T19:20:02Z")

</div>

> [@smokingwheels](#):
>
> It takes approx 11 mins on a HP 360DL gen 8 server.  
> I have loaded a 9 million block list on raspberry pi 3B, from memory, it took in the order of 1 hour so, going by that possibly 8 hours.

75 million valid domains?  
Split it up into smaller files and put it on github.

---

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [March 16, 2024, 8:49pm UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/6 "2024-03-16T20:49:13Z")

</div>

> [@bkbartje](#):
>
> github

Yes that is possible but I would have to split it up into 4 segments.

> [@buntspexxt](#):
>
> 75 million valid domains?

About 8 million are invalid.

---

<div class="post-metadata">

**Author:** ![chrislph](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/chrislph/32/32723_2.png) [@chrislph](https://discourse.pi-hole.net/u/chrislph)\
**Post date:** [March 17, 2024, 5:03am UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/8 "2024-03-17T05:03:03Z")

</div>

> [@smokingwheels](#):
>
> possibly 8 hours

Would it not be simpler and quicker to do the following:

- Add a blacklist regex `.*` to block everything
- Let the network (attempt to) run for a short while
- Use the _ **Audit Log** _ to whitelist the top domains in order based on most requested where those domains are not in the standard blocklist.
- Iterate around this loop

This whitelist process can be automated, to some degree, by querying the blocked domains endpoint and interacting with the audit database directly.

This way only domains that are already not in the standard supplied list and which are being requested above a threshold would be allowed, and everything else would be blocked.

---

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [March 17, 2024, 10:14am UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/9 "2024-03-17T10:14:25Z")

</div>

> [@chrislph](#):
>
> Add a blacklist regex `.*` to block everything

Thanks for your time to comment @chrislph

Plenty gets through.

 ![Screenshot from 2024-03-17 18-05-44](https://discourse.pi-hole.net/uploads/default/original/3X/f/1/f178d5f03a1979a3909f0a2b1eb4999987011e4c.png)

---

<div class="post-metadata">

**Author:** ![Bucking\_Horn](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/bucking_horn/32/18719_2.png) [@Bucking\_Horn](https://discourse.pi-hole.net/u/Bucking_Horn)\
**Post date:** [March 17, 2024, 11:23am UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/10 "2024-03-17T11:23:38Z")

</div>

That chart isn't very informational, as it doesn't have any details on domains (e.g. could be always the same domain).  
Also, we don't know what's on your list of allowed domains (I recall you mentioning having to whitelist a lot of domains from another post).

---

<div class="post-metadata">

**Author:** ![whole](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/w/5f8ce5/32.png) [@whole](https://discourse.pi-hole.net/u/whole)\
**Post date:** [March 20, 2024, 8:32pm UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/11 "2024-03-20T20:32:09Z")

</div>

I feel left out, I only have 11,200,000 😉

---

<div class="post-metadata">

**Author:** ![whole](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/w/5f8ce5/32.png) [@whole](https://discourse.pi-hole.net/u/whole)\
**Post date:** [March 20, 2024, 8:39pm UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/12 "2024-03-20T20:39:05Z")

</div>

> [@bkbartje](#):
>
> github?  
> or is there a file limit  
> nm:  
> it's not for free a file that large

Gitlab has a free tier 5GB

> **[Pricing](https://about.gitlab.com/pricing/)**
>
> See pricing for GitLab's DevSecOps Platform, which combines the ability to develop, secure, and operate software in a single application.

---

<div class="post-metadata">

**Author:** ![bkbartje](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/bkbartje/32/24655_2.png) [@bkbartje](https://discourse.pi-hole.net/u/bkbartje)\
**Post date:** [March 20, 2024, 8:51pm UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/13 "2024-03-20T20:51:57Z")

</div>

that's total. not per file

---

<div class="post-metadata">

**Author:** ![whole](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/w/5f8ce5/32.png) [@whole](https://discourse.pi-hole.net/u/whole)\
**Post date:** [March 20, 2024, 9:06pm UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/14 "2024-03-20T21:06:03Z")

</div>

Totally missed that, what is the "per file limit" ?

Would a free R2 bucket work? 'Maximum upload size 5 GiB'

[https://developers.cloudflare.com/r2/reference/limits/](https://developers.cloudflare.com/r2/reference/limits/)

[https://www.cloudflare.com/developer-platform/r2/](https://www.cloudflare.com/developer-platform/r2/)

---

<div class="post-metadata">

**Author:** ![bkbartje](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/bkbartje/32/24655_2.png) [@bkbartje](https://discourse.pi-hole.net/u/bkbartje)\
**Post date:** [March 20, 2024, 9:30pm UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/18 "2024-03-20T21:30:27Z")

</div>

> [@whole](#):
>
> Totally missed that, what is the "per file limit" ?

It's in the link

> [@Need a host for 75 million blocklist please](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/4):
>
> github? or is there a file limit nm: it's not for free a file that large [https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-git-large-file-storage](https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-git-large-file-storage)

but 2GB for free on large file storage, 100MB for normal files.

---

<div class="post-metadata">

**Author:** ![whole](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/w/5f8ce5/32.png) [@whole](https://discourse.pi-hole.net/u/whole)\
**Post date:** [March 20, 2024, 9:46pm UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/22 "2024-03-20T21:46:54Z")

</div>

> [@bkbartje](#):
>
> > [@whole](#):
> >
> > Totally missed that, what is the "per file limit" ?
> 
> It's in the link

Can you point out the file limit? I can't see it in the link

 ![image](https://discourse.pi-hole.net/uploads/default/original/3X/5/5/554688c324d2363ce4d113e0ff1617e05bb16d22.png)

---

<div class="post-metadata">

**Author:** ![smokingwheels](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/smokingwheels/32/2029_2.png) [@smokingwheels](https://discourse.pi-hole.net/u/smokingwheels)\
**Post date:** [March 20, 2024, 10:26pm UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/30 "2024-03-20T22:26:26Z")

</div>

See how this link goes for traffic.  
The list I have been talking about and using with Yacy.

[http://gts.undo.it/index.php/s/55aEx6yESmCFS8N/download](http://gts.undo.it/index.php/s/55aEx6yESmCFS8N/download)

A backup site you have to download it and store it in the pihole web server folder eg /var/www/html  
then point it to it in the adlists.  
[https://u.pcloud.link/publink/show?code=XZdKIF0Z9CvvFQCEO3uVgQkevMRadFl7NuKX](https://u.pcloud.link/publink/show?code=XZdKIF0Z9CvvFQCEO3uVgQkevMRadFl7NuKX)

---

<div class="post-metadata">

**Author:** ![chrislph](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/chrislph/32/32723_2.png) [@chrislph](https://discourse.pi-hole.net/u/chrislph)\
**Post date:** [March 20, 2024, 11:19pm UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/34 "2024-03-20T23:19:36Z")

</div>

Having a look through the list, there are loads of valid, important domains on there like GitHub's and Pi-hole's domains, Debian's and Ubuntu's domains, plus entries for `localhost` and `pi.hole`, as well as loads of non-domain entries that will just waste Pi-hole's time and energy and needlessly extend the update window.

Some of these, especially the valid domains, may impact functionality on the network, including for Pi-hole itself and its OS. For example systems will be unable to access lists or repos on GitHub.

Using this list is likely to lead to unexpected instability and failure modes on the network, largely due to indiscriminate overblocking. It's not suitable to be reliably used as is.

I'd suggest a new project to prune the hell out of the list – remove the millions of non-domain entries, collate the valid entries into genuine blocklists that should be blocked. But even then you're essentially repeating work that's already been done when someone finds a curated list for a given service (eg TikTok, Facebook) already out there.

---

<div class="post-metadata">

**Author:** ![whole](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/w/5f8ce5/32.png) [@whole](https://discourse.pi-hole.net/u/whole)\
**Post date:** [March 21, 2024, 11:02pm UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/36 "2024-03-21T23:02:04Z")

</div>

or maybe not 🕶

@smokingwheels I'd rather this list grow to maybe 100 million

---

<div class="post-metadata">

**Author:** ![Bucking\_Horn](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/bucking_horn/32/18719_2.png) [@Bucking\_Horn](https://discourse.pi-hole.net/u/Bucking_Horn)\
**Post date:** [August 15, 2024, 10:09am UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/40 "2024-08-15T10:09:36Z")

</div>

A post was split to a new topic: [Issues when loading blocklist](https://discourse.pi-hole.net/t/issues-when-loading-blocklist/71767)

---

<div class="post-metadata">

**Author:** ![vasundhar](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/vasundhar/32/8965_2.png) [@vasundhar](https://discourse.pi-hole.net/u/vasundhar)\
**Post date:** [August 16, 2024, 6:35am UTC](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992/41 "2024-08-16T06:35:07Z")

</div>

![image](https://discourse.pi-hole.net/uploads/default/original/3X/e/7/e72f44b9beae30ddab91a7632870ccf75543db06.png)

[Next page](https://discourse.pi-hole.net/t/need-a-host-for-75-million-blocklist-please/68992.md?page=2)