Within the repo is a directory titled 'parsed' I should have been more clear. I already parsed the lists.
Just checked and my automatic script ran without issue!
I parsed all the lists separately on purpose.
Ah I see, so we have to add each of those lists manually to the lists.lst with Nano?
Add the parsed list directly to pihole either with the webui, or
sudo nano /etc/pihole/adlists.list
Will adding just https://github.com/deathbybandaid/piholeparser/tree/master/parsed URL to adlists.list add all the parsed lists or we still need to add them one by one?
I need to update my README.md, I just ran out of time last night.
You won't need the tool to process the lists,,, just the lists.
I did all the heavy lifting.
One
By
One
I made a choice to not merge all of the lists into one. These are simply the original list maintainers work, but filtered/parsed.
That and the way it is setup, I can easily add more lists to parse.
Well that was no fun, but my contribution is this pastebin of all the URLs in that parsed list you created
One by one (My eyes hurt now lol)
Thanks, I've been on mobile and didn't have the chance.
https://github.com/deathbybandaid/piholeparser/blob/master/All%20the%20lists.txt
There may be more lists to come.
Added all those lists to my pihole but it only seems to have added a few more more blocks, less than 50
Can you post the URLs you're using to get 1.6m blocked?
This is what I'm using,,,, everything I use is within the piadvanced script,,,, consult my modules regarding installing the cronjobs and adding the lists to the files. (I'll be on mobile for most of the day.)
The Wally3k Adlists
https://github.com/deathbybandaid/wally3k-adlists.list
Some Youtube adblocking
https://github.com/HenningVanRaumle/pihole-ytadblock
More Youtube Adblocking
(nano the files to change directory if not in /etc/piadvanced)
https://github.com/deathbybandaid/piadvanced/tree/master/piholetweaks/youtubeadblock
Adguard
https://github.com/deathbybandaid/piadvanced/blob/master/piholetweaks/adguard.sh
This is a php parser, put it in your /var/www/html/admin/ folder
https://github.com/deathbybandaid/piadvanced/blob/master/piholetweaks/parser.php
I also use this to remove lists once a week, to re-download them. Some Lists don't seem to correctly distinguish if they have changed or not,,, so this enables a fresh download.
https://github.com/deathbybandaid/piadvanced/blob/master/piholetweaks/piholefreshlists.sh
1 Like
Well I've removed just about all the lists from the OP / parsed github, seems like every day I was having to whitelist yet another normal website, gmail, tech forums, kickstarter, yahoo mail (Don't use it, but discovered it was blocked) and other seemingly harmless websites such as Amazon
Got sick of having to whitelist constantly, and the gf complaining yet another X app stopped working, or wouldn't let her log in, or a site wasn't working
Default set of lists with a couple of others and back to 108k blocked domains now
https://github.com/chadmayfield/pihole-blocklists
This one really increased the amount I block!
There's an occasional domain that I have to whitelist,, but nothing has been super inconvenient for my wife,,, so it must be alright!
2 Likes
Holy moly, didn't realise there were quite as many pR0n sites out there lol
There's only me and the gf here so no worries about kids accessing things they shouldn't, ads are the main thing I want rid of, and facebook, but she uses that so I can't, or it would be wildcard blocked
if that's true we should really be okay haha
Don't know what's going on with mine, you have over 3m blocked and no issues other than the odd domain, I have added again only up to 700k and it's blocking things like mail.live.com, and these lists I am adding are malware domain lists recommended by people already using them (Can't find the site atm)
Why would live.com be on any list at all? Some very paranoid people tinfoiling about MS Win 10 telemetry spying to block an email domain and add it to a malware filter list
Hey guys
I am using some of you'r filter lists and it's working great until now
i tried to customize my word press site with the costumizer
the problem is the costumizer does not load up it's just stuck
i tried to white-list domains that where blocked from my pi-hole but it still does not work
thanks for any infos
One of the reasons we operate like we do with regards to lists is the copyright aspects to lists. Many of the list providers have copyrighted their works, and so we must download the entire lists with included copyright headers and store them in the /etc/pihole directory in whole and unaltered. I'm concerned that this amalgamated list may be in violation of copyright rules, do you provide the unaltered lists along with the parsed output from all the lists?
For example: Creative Commons — Attribution-NonCommercial-ShareAlike 3.0 Unported — CC BY-NC-SA 3.0 for ABPJapanese3rdPartySNSFilters.txt
Yes, all unaltered lists are within the mirroredlists directory.
Many of the lists end up becoming nothing after running through the parser.
As soon as my current test run is complete, I'll update my README.md with a disclaimer at the bottom:
Disclaimer
All "Original Unaltered Lists" are located within the mirroredlists directory.
After going through the parser, many lists contain zero lines and are deleted.
The filtered copies are located in the parsed directory, with filenames to reflect the Original Creators work/effort.