Yes you are right
So i sorted by name
I can not find duplicated list entrys
Here the sorted list
<List of Lists that only mods can see now.>
Lets try this
regards
Maybe you have already some adlists configured and now one of those you try to add is a duplicate?
You could also try to add them in smaller chunks to see which one is the duplicate
List of configured adlists
= It is empty
I have completely deleted the list before the import.
There are no entries.
Each line has a space at the end.
No URL is double
Hi i solved it
You have to put it in ONE LINE !!! (incl. spaces)
The LIST look like this
<List of Lists that only mods can see now.>
ONE LINE
Than it works - it is SOLVED for PHOLE 5
Command = pihole -g in SSH you can update the lists
or in web gui under TOOLS / UPDATE GRAVITY
Regards
The piHole can import lists with just
domain ?
Normaly the host list looks like
0.0.0.0 domain
Like the KADhosts_without_controversies.txt from https://firebog.net/ .
But what if there is another IP Address related to a domain and not 0.0.0.0, this cloud be harmful . Or is piHole replacing the IP anyway?
Yes. Pi-hole strips the leading IP, which is not required in the gravity list. Many lists are published with the leading IP 0.0.0.0 or 127.0.0.1 so the list can be used directly in the hosts file on an operating system, which does require an IP to domain name match on each line.
2 Likes
I tried the "TR-PhishingList" list, there where no errors on your pihole?
There are obviously umlaute äö ...
The new pi5 Version can not understand it, or the list is broken - wrong encoded?
[i] Target: https://raw.githubusercontent.com/HorusTeknoloji/TR-PhishingList/master/url-lists.txt
[✓] Status: Retrieval successful
[i] Received 221316 domains, 19 domains invalid!
Sample of invalid domains:
- bitcoin-f%c3%bcr-anf%c3%a4nger.de
- die-mobilit%c3%a4tsdesigner.de
- e-%e2%80%8brech%c2%adt24.de
- fpn81171321mp.hol.es?
- gl%c3%bcckscard.at
I think this was not the case, as shown in the screen capture posted with the final result.
The new gravity import routine for Pi-hole V5 will skip domains that do not meet the HOSTS requirements, and will report how many domains were loaded. If all domains were not loaded, it will also report how many domains were not loaded.
The worst offender in the group that @stone1978 loaded was:
[i] Target: https://www.joewein.net/dl/bl/dom-bl-base.txt
[✓] Status: Retrieval successful
[i] Received 71444 domains, 18010 domains invalid!
Sample of invalid domains:
- 0-as69vh7k2.biz;1
- 001linker.net;1
- 0034ufow.com;1
- 00hotmlsecure.live;419
- 01aj78cz5spx-9n3.com;1
Several of the other lists had a few domains rejected, but not near this many. I wouldn't subscribe to this list.
1 Like
The Adlist Group Management page will also accept entries separated by line breaks. If you go to the Wally3K ticked lists page, select all, copy to your clipboard, you should be able to paste the contents of the clipboard directly into the Address: window and hit Add.
I think the problem people are encountering is that the whole command seems to file when one of the X listes supplied is a duplicate.
Would be sensible to just ignore that one entry and add the rest, imho.
Feature requests are always welcomed.
Just joining in. Experiencing the same discomfort here. It is indeed about inputting multiple lists at once, but also, about 1 duplicate list stopping the whole processing instead of just being skipped. This is also a case of regression in 5.0, since 4.x did this before.
So I took the liberty to dive into it real quick, code wise:
File containing the HTML interface:
File containing the JS events:
File processing the inputted adlists:
Relevant action: add_adlist
Starting at line: 660
Culprits:
- Possible confusion
- HTML placeholder shows "http://..., https://..., file://...". Note the commas. Users might think the comma needs to be actually there.
- The actual code does: $addresses = explode(' ', trim($_POST['address'])); which means it splits multiple domains by spaces.
- Solution: it would be better not to split on space, since web addresses can contain non-encoded spaces (non %20). Best is to split on \n AND \r (to make it cross platform compatible, *nix/win) and leaving out any empty array elements it might result in. This solution works best with a textarea instead of the input now.
- Terminating all execution
- Further code (lines 676-692) executes statements to populate the database. These throw new Exception(). This halts further processing.
- Solution: populate an empty array with each exception encountered, but continue processing. At the end, return the array showing in the popup in the interface. No need for terminating the whole process.
I'm currently about to grab a bite to eat, otherwise I would have written the code myself. If nobody does it before I get a chance to, I still will. It would a a nice addition to V5.
Yes, that will work. It uses spaces between each (simple) address.
This is a different problem: you are entering addresses of lists you already have. Right now, you can only enter new addresses. But as said in my post above, this will be changed so you can have addresses in there that you already have without stopping. If not for me, some of us programmers will add this asap.
that is correct
correct. for now. again, I hope we can offer a larger field again soon that will accept multiple lines. hang in.
Pihole is getting rid of that IP anyways. It will block the domain name. No matter what IP is in front. Normally you only add domain names. But pihole can work with lists that have an IP in front to some degree, this style is called "host file"-style and comes from a long time ago. But it can still be used today. Anyways, do not worry about it. Pihole got your back and does it right.
the domain "fpn81171321mp.hol.es?" contains a question mark at the end, which is not valid. The other domains seem valid, this might indeed be a URL encoding problem.
Please note, the Adlist Group Management is ONLY meant to supply lists to. Lists having domains that you want to block. Do not add domains there which you want to block. You can add domain to block under "Blacklist". The Adlist page is for addresses leading to lists that contain many domains to block.
Agreed
Since I ran into this myself, I just might send a pull request or at least paste the desired code changes for all this myself.
I am aware of this. The original question was how to bulk add adlists, not domains.
Bulk addition of domains is also supported in the same way, but with domains only, not URLS that would be added to the adlist.
no, i know you know @jfb 
this was just to clarify for other readers that might stumble upon this thread.
This would be really great.
This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.