one question if a block list is in html format would that cause it to flag, since mvps hosts was in that format and i caught it last night, and got their no html list.
Im also thinking that pi doesn't like media fire for list hosting i removed old lists id made and uploaded there.
Hosts format is the only real format to use. The problem with the mediafire is that when you curl that URL you do not get the file, you get mediafire's timer page.
curl -sSL http://www.mediafire.com/file/lc6746hl121wgd2/pac_list-1.txt to see what I mean.
There isn't really an html format, if there's html code seen when you curl then you're doing it wrong.
Wooott, i think i nailed it after removing the links that i had with media fire, im starting to think that if the block list downloads as a text file as opposed to opening in a browser this error occurs i find that list that does it.
So essentially what im saying is that if a block list doesn't open in a browser and it opens to a mediafire download link pi will trip, because after i removed the lists there was link 5 and i corrected the pgl.yoyo.org hosts from html list to the no html list this is the result.
pgl.yoyo.org
html
https://pgl.yoyo.org/as/serverlist.php?hostformat=hosts
non html
https://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml
i think this is the list causing the last error its missing the .txt extension when i checked it
https://reddestdream.github.io/Projects/MinimalHosts/etc/MinimalHostsBlocker/minimalhosts
this is the correct url for the git hub link, to the list.
https://raw.githubusercontent.com/ReddestDream/reddestdream.github.io/master/Projects/MinimalHosts/etc/MinimalHostsBlocker/minimalhosts
since i removed my mediafire links the error for the most part has cleared.
i meant this list, when i was talking about html.
https://pgl.yoyo.org/as/serverlist.php?hostformat=hosts
probably explains it, keep in mine i made this during an early version of pi-hole.
i should probably adjust the lists to the correct format, remove duplicates and upload to git hub.
The way pihole gets the lists is via curl. Really, run the command I pasted ( curl -sSL http://www.mediafire.com/file/lc6746hl121wgd2/pac_list-1.txt).
It doesn't matter what things look like in your browser or what pgl.yoyo.org looks like in a browser or if mediafire redirects you to the file. If it's broken when curld then it wont work.
Case example:
curl https://pgl.yoyo.org/as/serverlist.php?hostformat=hosts | head -n 20
<html>
<head>
<meta name="author" value="Peter Lowe; pgl@yoyo.org">
<meta name="description" value="Blocklist of hostnames and domains for blocking ads, trackers and others (format: hosts -- in hosts file format)">
<meta name="keywords" value="ad blocking, blocking ads, ad servers, trackers, ads, banners, hosts file, privacy">
<link rel=alternate type="application/rss+xml" title="Blocklist of hostnames and domains for blocking ads, trackers and others (format: hosts -- in hosts file format) (RSS - last 50 added)" href="rss/1.0/adservers.rss">
<title>
Blocklist of hostnames and domains for blocking ads, trackers and others (format: hosts -- in hosts file format)</title>
<link rel='stylesheet' href='/css/pgl.css' type='text/css'></head>
<body>
<p> [ <b><a href="/adservers/">view the list in a different format</a></b>
| <a href="index.php">about this list</a>
| <a href="iplist.php">view the blocklist of IP addresses</a>
Okay, what should that mean to me, I'm not seeing the point.
so question would a list like this work with pihole i disabled it till i can hit the list that is causing issues, not sure if this is too much of a mess.
https://raw.githubusercontent.com/lewisje/jansal/master/adblock/hosts
No it won't work. curl it yourself to see.
Go back to the stock list setup. Add back the lists you want one at a time until you see issues. If that is a long process then honestly it should be. Adding 70+ lists that no one knows the status of is just a bad idea.
i looked at it already just wanted to confirm that it was off the table, thank you for that, im just in the process of purging garbage lists, and only keeping compatible ones.
0909148.com;1
0909471.com;1
0909582.com;1
0909693.com;1
0909704.com;1
0918amazing.us;1
0918big.us;1
0918instructive.us;1
0918unusual.us;1
from this list i think its the culprit, probably needs parsing logic to remove the ;1, i found it duplicated oddly
https://joewein.net/dl/bl/dom-bl-base.txt
according to your commands this is the list, i just doing a gravity with it disabled
https://raw.githubusercontent.com/mkb2091/blockconvert/master/output/hosts.txt
root@WonderBoom2:/home/pi# head -n 7775252 /tmp/tmp.a9irUejwu1.gravity | tail -n 10
entranceemployees.com,155
entrancemine.tk,155
entranceprepmaterials.blogspot.com,155
entrar-fb.vacau.com,155
entrata.qualtrics.com,155
entrateluwian.host,155
entrav.ru,155
entre-culturas.cl,155
entre-potes.mon-application.com,155
entrecard.s3.amazonaws.com,155
And after 5.0 is released they will find that the lists they were using and were happy with didn't really do anything in Pi-hole v4, adding extra lists doesn't do much of anything. They will learn that the lists they use are not proper and need to be fixed or no longer used. I do not think that manipulating lists and filtering out garbage is the proper response. That is a band-aid response.
I'll be blunt. You're using garbage lists and lists that are not compatible with Pi-hole. Stop chasing insane numbers of blocked domains and look at the user experience. Start with the default lists and if you see ads then that is the time to add more lists.
All of this is self-inflicted. You're causing your own headaches. Do you need to add a list with a single domain on it? https://raw.githubusercontent.com/broerict/blacklists/master/phishing
The only reason i added other lists was to get rid of some of the pesky ads that were bothering me, as for insane numbers of blocked domains i could care less.
Some of the lists i added in the past had more domains on them at the time of adding but i never checked them, ive been so far removing lists that have become inactive or as of now are not found.
If my self and other users don't exactly have a measure, of determining compatibility of lists, then the experience of trial and error can be considered self inflicted, as for headaches that is why im trying to fix the problem, that's the point of me purging incompatible lists.
Look at the end of the day its a leaning experience for me, so that in future i don't make the same mistakes again, if i knew that adding certain lists would cause a major problem to ad certain lists then i would have never added them in the first place, and i would save my self issues, i was already entertaining the idea of downsizing the amount of lists i had in the first place.
That's why we added the capability to blacklist domains. And added on that capability with regex blocking. Then expanded that to CNAME block as well. If there are a few domains that are problematic then add them to your blacklist. There's no need to add more lists that can change, go stale, stop being hosted or any other number of issues that can come up with things not under your control.
Edit: Note that none of these lists actually worked in version 4, the changes now are that instead of silently accepting everything, now bad data causes an immediate response.
How about a warning in gravity that tells the user which lists need to be removed, or an automatic disabling of incompatible lists.
if i remember correctly there use to be a warning for lists that weren't compatible, in v4, during gravity.
Which was one of the most awesome additions, ive been reading over the documentation on trying to lean how to use the tools correctly, the best outcome of this whole situation is i can now warn my other friends to check their pi-hole lists so the don't have the same issues.
Ive got one who wants me to help them set up a pi-hole install another who is probably going to replace his sd card and rebuild his block lists, and another who will rebuild once he gets a pi 4.
On the up side, v5 is extremely responsive, also @DL6ER his fix worked since its no longer throwing up a database lock error and the -2 error is gone.
Also thank you for the help since this purge is doing wonders for my pi-hole, it just shocked me that, going form a rock 64 to a pi4 both fresh installs there was errors.
If you add a list and it breaks then that's the list to remove.
At some point you have to take responsibility for your setup. This is a DNS server and a vital part of your network. Hiding things is not the proper response, in my opinion.
The gravity output will say if there are errors, but no one really reads that output. I think you would have seen the errors for lists that went 404 or other errors if you had been reviewing them.
Great! Glad that it's working better for you and you've learned more about how things work. Even better that you're willing to share that information with others to help with their experiences.
In this case, they should screen their lists more carefully and not pick up junk from the internet.
