[Request] Add "Filter Whitelisted" checkbox to Query Log

I get tons of queries from whitelisted domains in my query log. So much so that even using the "All" settings I sometimes have to go through pages of the same results to get to something interesting.

Some of it is caused for example by using the Atom editor which will poll the "atom.io" domain repeatedly for updates etc. This causes entire pages of the query log to be filled with results that I know are irrelevant.

Once I have whitelisted a domain I would like to have the option to (permanently) exclude it from the Query log. Ideally a Settings option that allows me to choose a default (e.g. "Exclude whitelistes items from Query Log") which results in a checked box on the Query Log page. If I do not wish to set this option as default I could still enable the option on the fly when browsing the Query Log.

If you give me a hint on how this could be done I'd give it a shot implementing it.

EDIT:
This is NOT covered by the exclusion options for the API "Top x" display. On an admin level this would be much more useful than excluding entries from the graphs.

You can filter the query log using the search box at the top. For excluding certain domains from the log that would need to be added to the code that returns the log (getAllQueries).

I know I can manually filter for URLs that are on the whitelist but this doesn’t allow for easy scanning of the list. I’ll have a look at the code that does the queries and how you could compare and filter against whitelisted stuff.

What kind of format would be easier to scan than simply searching on the query log for not blocked queries? Do you want to see only whitelisted queries, or do you mean whitelisted as in not blocked?

I mean “not whitelisted and not blocked”.

I get hundreds if not thousands of “not blocked queries” that are all valid and I know them to be unharmful.

It would be awesome to simply be able to add certain domains to the whitelist and not have them show up anymore. Some examples:

github.com (I use it a lot, so there are tons of legit queries for this)
yammer.com (I use this for work, same reason)
Various CDNs that I can identify to belong to sites I use
Akamai
The Ad List servers (which are automatically added to the whitelist)
Various Video Streaming services and their CDNs
etc.

Now look at a typical Query Log. Let’s say there are the 113 domains I blocked through wildcards, about two dozen which I blocked through exact matches, let’s assume 3 harmful domains that I might want to filter and finally about 3000-5000 requests for various iterations and varieties of the things I mentioned above. Now multiply this by every device on your network. For me it’s three smartphones, two tablets, a Desktop, a Laptop, several gaming consoles and other entertainment electronics. All of them cause queries. So I have to scan (with my eyes) through tens of thousands of queries of legitimate and unharmful stuff to find the malicious ones.

Now I have to scroll through dozens of pages of the same query types to “maybe” spot one of the three harmful ones.

It would be MUCH easier to simply add all the domains that I have positively identified as being “not harmful” to a permanent whitelist and simply drop them from the Query Log (by selecting that option) altogether. Ideally this would have a second option dropping all known blocked entries from the list. What should remain is the much smaller subset of:

Domains that I have not yet whitelisted
Domains that I have not yet blocked

And in the example probably a handful of pages to scan through.

TL;DR: If I know a domain is safe I want to be able to whitelist it and no longer list it in the query log unless explicitly asking for it.

Can’t you use the settings page to not show the unwanted domains?

@telekrmor hat only applies to the graphs. Also this does not improve usage of the Query Log to discover new potential black/whitelist items.

I came here looking for this same feature, just installed pi-hole today, love it already.

I think this is what you want.

No, this only excludes whitelist items from the “Top” list which I don’t use at all. I want the query log to be more useful.

I’m confused: Why doesn’t this options address your request?..

He doesn’t want to see any of the clients that he has white listed in the query logs. He still wants those clients to go through pi.hole. Just doesn’t want to see their logs unless he unticks a “hide whitelisted clients” button

Oh, I got confused by the other answers. Rereading the two posts from the initial posters I see this now.

@r0ckarong @derailius Would agree that simply extending the filtering from Top Lists also to Query Log or would we absolutely have to implement another filtering layer for that?

Just extending the filter to include the query log would be perfect in my opinion.

Edit: if you don’t want to see whitelisted clients in queries I can’t imagine you would want to see them in top lists.

This will make it easier when reading the queries. A lot less will have to load too. Also thanks for everything!

I see it from a different perspective. From your point of view programmatically it’s “another filtering layer” for the user it’s “another list I have to maintain”. Since the "exclude from Top Lists is a totally different list than “Stuff I already know is safe (whitelist)” this is another “layer” of though you need to invest to easily maintain your filter.

To me it would make more logical sense to remove all whitelisted stuff from things like top lists and query log (when the option is chosen). This would make most “exclude from top” lists less long anyway and then they would only actually need to show domains that are not yet determined to be safe. If you know you have something constantly on the top list that is unsafe and needs to be there then (to me) would be the extra layer of “filter this from top as well”.

In short: Extending the “exclude these” logic from Top lists to query log would be an improvement, sort of a workaround but not really the solution to the overall problem of disconnect between the filters and display lists.

Query Log = Whitelisted Items + Blacklisted Items + Top Lists Meta Data + Excluded From Top List
Top List = Whitelisted Items + Blacklisted Items + Top Lists Metadata

What I as a user want to be able to maintain is
A) Whitelisted Items
B) Blacklisted Items

What I am looking for would be something like an “Audit Log” that shows only what I have not yet black or whitelisted so I can make an analysis and determine if it can go on either list.

1 Like

Since I would have to realize this in FTL (in a rather critical part of the code), I wanted to be sure to understand what I would have to do. Even adding “another filtering layer” is not something that can be done without a noticeable amount of effort.

So let me come to the idea of an Audit log (stepping away from changing the Query Log page at all for the moment):

It would be (more or less) easily doable to add an Audit log in the sense of a table, that looks maybe something link this:

-----------------------------------------------------------------------------
| Permitted queries               | Blocked queries                         |
|---------------------------------|-----------------------------------------|
| google.de    [Hide] [Blacklist] | googleadservices.com [Hide] [Whitelist] |
| ebay.com     [Hide] [Blacklist] | somebadguy.com       [Hide] [Whitelist] |
| amazon.co.uk [Hide] [Blacklist] | *.microsoft.com      [Hide] [Whitelist] |
| ...                             | ...                                     |
-----------------------------------------------------------------------------

Once, you click on Hide, it will be removed from this table and saved in some list, so that it wouldn’t be shown in the future. However, it will also give you the option to White-/Blacklist this domain immediately (and hide it afterwards as well).

Is this going into the direction you are thinking about or am I missing something fundamental?

Yes that sounds a lot like what I’m looking for to improve usage of the filter.

In this case I don’t see any change to the Settings page.

Agreed, if this is implemented as a separate page from the Query log it doesn’t need extra settings (for now).

I will branch this off to a new feature request so it is easier for people to find and vote for the Audit Log page.