There are a few block lists out there that pretty much throw in everything but the kitchen sink for domains to block, and aren't really tailored to work with a DNS-based solution like Pi-Hole.
In my experience, one of the worst offenders is https://adblock.mahakala.is. This blocks a lot of block list sites, as well as many reputable commercial sites like walmart.com (not just the ad-serving subdomains), etc. I deleted it shortly after I started using it.
I am trying to make pihole as automatic as I can, using one of WaLLy3K blacklists and anudeepND whitelist, downloading both lists automatically and running pihole -g. This works until one of WaLLy3K's lists starts blocking other lists.
I understand you don't want to add whole domains (like s3.amazonaws.com) to the whitelist, but I have no problem allowing a specific list like https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt from a well curated Blocklist Collection.
--dns-servers 1.1.1.1 could be added to the curl command to allow the lookup of the entries in the blocklist collection. Unfortunately, the --dns-servers option is not available on Raspbian:
$ curl pi-hole.net --dns-servers 1.1.1.1
curl: (4) A requested feature, protocol or option was not found built-in in this libcurl due to a build-time decision.
Any other ways you can think of (other then disabling pihole temporarily) to allow the download of the blocklist entries?
I am trying to whitelist the entries in the blocklist. Not statically(manually), but dynamically, depending on the content of the blocklist collection.
This works pulling the file, even as hosts-file.net is blacklisted
Are you trying to whitelist the list domains though? That seems to be what you are trying to do. The best way to do that is to just whitelist them manually.
In addition to the other comments, having a pre-populated whitelist also confused some users and some thought we were showing favoritism. So those were a couple other reasons we took them out.
No, I'm not trying to whitelist the domains, I'm trying to find a way to download the url's in a blocklist regardless of the current blacklist or whitelist.
My blocklists now update even if the URL is blocked by pi-hole:
$ pihole -g
[i] Neutrino emissions detected...
[✓] Pulling blocklist source list into range
[i] Target: hosts-file.net (grm.txt)
[✗] Status: hosts-file.net is currently blocked by pi-hole. Circumventing pi-hole and trying again
[✓] Status: Retrieval successful
[i] Target: reddestdream.github.io (minimalhosts)
[✓] Status: No changes detected
Thanks for your help. I tried exactly what you said, but I'm still getting the same exact error. I'm completely ignorant so any other thoughts would be helpful.