Yeah maybe....
domain_blocked_by_deep_cname_inspection is a bit lengthily..
additional_info
This is about what I had in mind earlier as it can be recycled for other things in the future. Its meaning will then depend on the status
You could, when ecexuting pihole -q -exact <domain>, lookup (sqlite3) existing entries for <domain> in the database and check if there is a ! null entry (new database field) for that domain, display the entry when something is found
something like:
pihole -q fonts.gstatic.com
Match found in regex blacklist
.*;querytype=!A
CNAME match found in database
fonts.gstatic.com is CNAME for gstaticadssl.l.google.com
This is a good idea, however, I immediately have two objections, maybe you can convince me that they are minor. This is not at all to play down your idea.
- Performance
The new field has no index on it. That mean that such a scan means reading the entire table from disk in a full-text search. This can easily take minutes. You may argue that we can add an index on the newly added column, however, this is also not an optimal idea as there is no way the database knows that the column is "new" (as in all rows have aNULLcolumn) so at least the index creation would again ask for an entire full-text database operation. - Reliability
Such a lookup will only show things which happened in the past. Queries which would be blocked by CNAME inspection but we haven't seen them so far would not be shown.
I prepared a new branch new/cname_inspection_logging for this feature request.
The CNAME details are now, in addition to the already existing display in the Query Log:
also stored in the database:
sqlite> SELECT * FROM queries WHERE additional_info NOT NULL;
id |timestamp |type |status |domain |client |forward |additional_info
8747483|1594668331|1 |11 |fonts.gstatic.com|192.168.2.223| |gstaticadssl.l.google.com
and also in pihole.log:
Jul 13 21:25:31 dnsmasq[23773]: query[A] fonts.gstatic.com from 192.168.2.223
Jul 13 21:25:31 dnsmasq[23773]: forwarded fonts.gstatic.com to 127.0.0.1
Jul 13 21:25:31 dnsmasq[23773]: reply fonts.gstatic.com is <CNAME>
Jul 13 21:25:31 dnsmasq[23773]: reply gstaticadssl.l.google.com is blocked during CNAME inspection
What is not done so far is restoring CNAME information from the database into Pi-hole's memory after an FTL restart. I will have to think a bit about how to do this (it may be that the domain initially causing the block is not known to FTL during history parsing because it was seen more than 24 hours ago).
does that branch also include the modifications for
- pihole checkout ftl new/tre-regex (Approximative matching branch)
- allow certain duplicate domains
I'm still testing these, aborting my test for another test doesn't look smart.
For having manageable and reviewable pull requests, I separate new features from each other, so
-
new/tre-regex: No, there is still an open PR for this -
tweak/improve_unique_domains: Yes, because it is already contained indevelopment
1 Like
I will not be testing the new branch yet, I want to complete my test on .*;querytype=!A first, this requires a lot of time (different devices). I'm sure however, there are sufficient interested users to test this, as it possibly resolves a problem for many.
Would it be possible to let pihole -q -exact <domain> perform the equivalent of a dig to that domain, this would expose a possible deep CNAME match and make the information available to show in the result of pihole -q. I assume pihole-FTL can retrieve DNS information, without using an external command (dig)?
I'm struggling to see the need for any addition to pihole -q for this one. If you see a domain blocked by CNAME on the query log, surely you would query the deep CNAME match, and not the actual requested domain?
I thought the issue here was that the information about the deep CNAME match was not persisted across reboots/restarts of pihole-FTL?
I'm not sure what there is to be gained from adding complexity to a function who's job it is to check the downloaded lists for matches of input... 
Edit: though reading the request I see this is about not visiting the web interface to see this information. Original point still stands though, it's increased complexity for very little gain
1 Like
You may recall I already mentioned that information is no longer available, if you restart pihole-FTL, as you also mentioned.
query.sh will already need a change, due to pihole checkout ftl new/tre-regex, see here.
Adding the deep CNAME inspection info is the original feature request...
This is now done as well.
No offense here, appreciating all the great work the developers are doing, just trying to draw the complete picture...
Another reason why having the info available in pihole -q:
The deep CNAME inspection info is Not available when looking at the Long Term Data / query Log.
It will be, though, which makes the pihole -q addition moot. As @DL6ER mentioned above, there's no real performant way to add this to pihole -q, and adding dig-like functionality begins to creep away from the scope of the script... which is, quite simply, to determine whether or not a domain is included on one of the blocklists, or is affected by regex rules. We would have to check each time pihole -q is run "is this domain a CNAME for some other domain that may or may not be on the blocklists?"
But like I said, this information will be available already on the query log (yes, I know it isn't right now after a restart)
Linking for reference, as it is related to CNAME
This issue ticket should be transferred to the FTL repo. I looked through the CNAME code and tried to find the bug but couldn't.
I think the ticket is fine in AdminLTE.
The problem is that the actual domains blocked by deep CNAME inspection were reported in the dashboard "Top Blocked Domains" and on click the query log is empty. This was (so far) due to the query string (containing the actual domain) didn't match any domain.
Edit: more accuracy about which domain is shown on dashboard.
No. The dashboard shows 1:1 what it receives through the API from FTL. Wrong display on the dashboard means wrong data given by FTL.
The dashboard is fine. It reports the actual blocked domain. The link to the query log (query sting) does not return any result (so far) for this domain.
Yes, and my point is that this is wrong. Pi-hole should not show the actually blocked domain (because the user never tried to access this page directly) but rather the originally requested CNAME. Hence, in my world, showing this domain is wrong and the CNAME should be shown instead.
So, instead of settingsfd-geo.trafficmanager.net as top entry, there should be settings.data.microsoft.com
The link will then also work as expected and this is honestly how I think it would be best and likely how the devs wanted to have it (but then some bug came in their way).
1 Like
Thanks for your more detailed explanation. I get your point now.
I'm not sure if I prefer 'What you see is what you meant' over 'What you see is what really happend' but (most?) users might be irritated by the latter. Fixing the query link would be one solution to reduce the confusion - the other would be changing the domain displayed at the dashboard as you suggest.
Let's see what the devs intended.