With the addition of "Deep CNAME inspection" in Pi-hole Version 5, the Pi-hole recently started blocking DNS requests cloaked behind CNAME, but does not show which domain caused the block in pihole -t or pihole -q.
For example, trying to query the german satire website "Der Postillon" using pihole -q der-postillon.com returns, that the domain has not been found within the block lists. When looking into the query using pihole -t, the query returns:
16:15:52: query[A] www.der-postillon.com from 192.168.178.1
16:15:52: forwarded www.der-postillon.com to 127.0.0.1
16:15:52: validation result is INSECURE
16:15:52: reply www.der-postillon.com is <CNAME>
In order to determine what caused the block of the domain, it was required to visit the query log of the WebUI.
I was trying to check if something similar was requested in advance within the period of the Beta v5 period or in the last couple of hours. I want to apologize if there is a duplicae topic for that.
Oh yeah, that totally makes sense. I need to apologize i guess.
I like the Web Interface, but is there a strategy to discover blocked CNAMEs over CLI available or atleast planned in the future?^^
There is a way - but not an easy one. You can set DEBUG_QUERIES=true in /etc/pihole/pihole-FTL.conf and restart FTL. In /var/log/pihole-FTL.log you will find all queries with debug outupt - including CNAME. Here is a sample output which you might use in some kind of script to get what you want.
[2020-05-24 19:31:36.607 14149] **** new UDP query[A] "logger.yp.ca" from eth0:10.0.10.136 (ID 17, FTL 15909, src/dnsmasq/forward.c:1549)
[2020-05-24 19:31:36.607 14149] logger.yp.ca is known as not to be blocked
[2020-05-24 19:31:36.608 14149] **** forwarded logger.yp.ca to 127.0.0.1 (ID 17, src/dnsmasq/forward.c:549)
[2020-05-24 19:31:36.608 14149] logger.yp.ca is known as not to be blocked
[2020-05-24 19:31:36.609 14149] CNAME logger.yp.ca
[2020-05-24 19:31:36.609 14149] **** got reply logger.yp.ca is (CNAME) (ID 17, src/dnsmasq/cache.c:480)
[2020-05-24 19:31:36.609 14149] ypg.tagcommander.com is known as not to be blocked
[2020-05-24 19:31:36.609 14149] CNAME logger.yp.ca ---> ypg.tagcommander.com
[2020-05-24 19:31:36.609 14149] **** got reply ypg.tagcommander.com is (CNAME) (ID 17, src/dnsmasq/cache.c:480)
[2020-05-24 19:31:36.610 14149] ypg.aws.tagcommander.com is known as gravity blocked
[2020-05-24 19:31:36.610 14149] CNAME ypg.tagcommander.com ---> ypg.aws.tagcommander.com
pi@raspberrypi:~ $ pihole -q fonts.gstatic.com
[i] No results found for fonts.gstatic.com within the block lists
gstaticadssl.l.google.com is blocked
pi@raspberrypi:~ $ pihole -q gstaticadssl.l.google.com
Match found in https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt:
gstaticadssl.l.google.com
in the pihole query log, I see (deep CNAME inspection)
however, in the pihole log, I cannot find any reference togstaticadssl.l.google.com, only to fonts.gstatic.com
Jul 2 09:52:54 dnsmasq[27254]: query[AAAA] fonts.gstatic.com from 192.168.2.228
Jul 2 09:52:54 dnsmasq[27254]: forwarded fonts.gstatic.com to fdaa:bbcc:ddee:2::5552
Jul 2 09:52:54 dnsmasq[27254]: reply fonts.gstatic.com is <CNAME>
is it possible to add log lines for the domain entry, causing blocking (gstaticadssl.l.google.com)?
I use this little script to find blocked entries quickly, but, items blocked, due to deep CNAME inspection, never show, due to the missing log entries.
#!/bin/bash
logfile=/var/log/pihole.log
tail -F $logfile | grep --line-buffered "is 0.0.0.0\|is ::" | \
while read x; do
echo "$x"
done
The solution proposed in that topic ( DEBUG_QUERIES=true in /etc/pihole/pihole-FTL.conf) requires analyzing a different log, and will exponentially increase the number of writes on the SD card (raspberry pi users).
The question is specifically asking for added log entries in pihole.log, without changing any system (pihole) settings (activating additional logging).
The amount of data collected by setting DEBUG_QUERIES=true is far larger than adding a single line to pihole.log, when, for example gstaticadssl.l.google.com is blocked, due to deep CNAME inspection.
An additional reason for this request:
When pihole-FTL is running, a blocked entry (deep CNAME inspection) looks like this (actually blocked domain name is available)
I'm sorry to keep at this, I hope you know I respect your work.
I'm sure the reasons for NOT being able to add an extra log entry to pihole.log are valid, however, for most users, it's really hard to solve problems, caused by deep CNAME inspection.
Would it be possible to create an additional log (or database table), that only contain entries, NOT in the pihole log? The web interface could than have an additional menu entry (CNAME blocks or something), that only contains this information. Pihole-FTL knows this information, since you are able to display it in the regular query log.
I kind of get what you're saying, jpg, but it's probably not as straightforward as that. Besides which, is a pretty niche case to have a whole additional page/menu item for. I'm still yet to see a single domain blocked by CNAME on my production Pi-hole
That all said, I got to thinking.. and so have a thought that I'd like @DL6ER's input on. I might be missing something here, being not overly versed with the inner workings of dnsmasq, but is there anything stopping us from adding an additional (nullable) column to the queries table in pihole-FTL.db?
For the vast majority of queries, this column will be empty, but in the case of a domain blocked in this way, we could store the actual blocked domain in this additional column? What does it currently look like to be able to display it on the page in the first place?
From my very loose understanding of what is actually going on behind the scenes here, we keep the information in memory as long as FTL is running, but once FTL is restarted, then that information is lost because there is nowhere to store it, right?
edit:
having looked a bit closer at the FTL code.. I see that whilst running the information is stored as an ID here CNAME_domainID... beyond all this it looks like to resolve that ID to a domain name, one needs to start looking at shared memory and that's where my head starts to feel sore.
second edit to add:
I'm sure this has all been thought of before - and there is probably a very good reason why we're not doing it.
I appreciate your comment(s), unfortunately, I fail to understand why it would be possible to display something (an actual domain name) in the web interface (query log) and not being able to write the same information in a log or a database table.
As I said, if it isn't that we just haven't thought of doing it before, then there is probably a good reason behind it. Let's see what those with a more intimate knowledge of the code have to say.
The reason is simple: When the database was designed (years ago) there wasn't a thing like CNAME blocking. Hence, there is (currently!) no field in the database where this additional information can be stored in.
No, the major reasons why it's not there is likely because (a) nobody asked for it and (b) I haven't thought of it myself.
Yes.
Since this feature request (originally posted by @PureFallen) asks for two things:
Add logging to pihole -q
Add logging to pihole -t
My summary is:
Is not really possible as CNAME blocking depends on external information (the entire path of CNAMEs) and, hence, is only possible at real time as (a) the paths may change upstream and (b) it would be necessary to query all domains in the entire Internet to find out if they contain a CNAME element which is blocked. It is not possible to ask a DNS server something like "tell me all domains where CNAME x.y.z is somewhere in the path".
This we can do, I will have to research how to do it without hooking into the dnsmasq code even more, but it will surely doable somehow.
In addition to this, we will add a new column containing (TEXT type, allowing and defaulting to NULL as @PromoFaux suggested). I'm open for suggestions for a name. cname doesn't seem appropriate, maybe actually_blocked or something like that.
