When updating gravity, I get the following output:
[i] Number of gravity domains: 851836 (700147 unique domains)
[i] Number of exact blacklisted domains: 322
[i] Number of regex blacklist filters: 176
[i] Number of exact whitelisted domains: 856
[i] Number of regex whitelist filters: 1
I recently did a full query log analysis sorted by domain, to find additional domains to blacklist from network activity. At least some of the domains I blacklisted were already visible in my gravity lists, as confirmed by a search from the "Query Lists" interface. I'm curious if some of the traffic was allowed because of a less-than-desired (malicious?) whitelisted domain in one of the block lists I'm pointing to. Of course, this is all under the premise that blocklists also can whitelist domains, which is an assumption I'm making based on the output above. I only have <300 total manually entered filters, a mix of blacklist and whitelist entries.
I'd like to confirm this behavior by seeing an explicit list of whitelisted domains/filters provided by my subscribed lists, so that I may determine if I should remove some of these lists or identify other filter candidates to block manually. There are certainly quite a few whitelisted domains, and that concerns me. I understand that some whitelisting is important to preserve functionality, and am not suggesting all whitelisted entries are "malicious."
What methods can I use (preferably from the GUI, but I can go to the CLI if necessary) to get a list of whitelisted domains and their source list?
Blocklists are exactly that - a list of domains to be blocked. Pi-hole reads all your block lists (adlists) and puts the domains from those lists in gravity to be blocked.
Whitelist entries are domains that you have added to your local Pi-hole. Adding a domain to the whitelist makes the domain gravity proof - regardless of whether it appears on your blacklist or in a block list, the whitelist will take precedence and the domain will not be blocked.
This doesn't exist, because subscribed lists are not for whitelisting, they are domains to be blocked. Pi-hole does not have a feature to subscribe to public whitelists.
Please upload a debug log and post just the token generated by
pihole -d
allowing to upload when prompted, or do it through the Web interface:
Ok, that certainly clears up a lot. However, I know I haven't created 856 whitelists myself (my Whitelist Management page shows 179 total entries), which caused me to question how it worked. What's confusing to me is why some of my local requests that hit domains found in a Query Lists search weren't caught by gravity. If there's no way for these lists to create whitelist entries, I have no reason for concern.
I'm not following you here. Please provide some examples of domains that you feel should have been blocked with the following command:
pihole -q -exact domain-name-here
Example would look like this:
pihole -q -exact s.youtube.com
Exact match found in exact whitelist
s.youtube.com
Exact match for s.youtube.com found in:
- https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
Note that you have a number of management groups, so the blocking will be unique to each group. All of your adlists are assigned to the default group only, so they aren't effective for clients in other groups.
That's exactly what I was missing! I just recently began using the group management feature and didn't realize group assignment also applied to adlists. I will make the change now, and am confident this will resolve my issue.
While the feature could use some additional functionality, it is very powerful and something I've been looking for. On that note, what's the best way to formally request features, so that I follow process?
Open a new topic and make it a feature request. As a new forum user, you may not have yet gained the privileges to do that.
As to your other question, I believe the count is being inflated by the number of groups each whitelist entry is assigned to. Making a whitelist entry for 5 groups is 5 whitelist entries, even though you have the domain specified only once.
Please understand anything listed below is meant to be constructive, and I appreciate the effort the entire team has gone through to deliver this project!
Specifically with group assignment, a dedicated "all" group (e.g. -1 or some other unique ID) might be a nice feature, to ensure that any newly created filters, Groups, or Clients could automatically apply to all relevant objects, if desired.
This would be a distinct difference than selecting "All" in the interface, which essentially enumerates all individual groups. These two variations would not be mutually exclusive, but I could see it causing some confusion if not designed/documented clearly.
The reason this matters is two-fold. Since I am new to group management, my needs may change, causing me to create new groups. Since there is no easy way to mass update entries to include additional groups, I would have to manually modify each entry to include the new group(s). I'm aware it is possible to run queries against the database to achieve this and has probably been addressed dozens of times before, but I'd rather not have to dig so far into the guts of the service to make this otherwise simple change.
Another example is manual blacklisting from the Query Log. By default, selecting Blacklist (or Whitelist) in the entry applies the filter only to the Default group, but none of my managed Clients currently belong to this group as per my desire for granular control. This then requires me to go back to Domains under Group Management to adjust membership for each added entry.
If objects could be automatically applied to all future objects in such a manner, it would obviate the need for these adjustments and allow for immediate protection.
For DNS Records, it would be nice to be able to perform an inline modification/update to an existing entry with a different domain name or IP, as opposed to having to delete and recreate.
This becomes useful when a DNS name change is needed (e.g. by preference or just a typo), or if the client's IP changes. I recently reorganized the IP addresses in my subnet, triggering the need to touch almost every DNS record in the list.
For the Local DNS editing question, you can edit file /etc/pihole/custom.list and search/replace there. That's the source file for the Local DNS records.