[DNSSEC] Remote client DNS queries INSECURE but local or dig ones SECURE

Tojaj · December 20, 2022, 12:10pm

The issue I am facing:
When a (remote) Pi-hole client (192.168.1.102) ask for DNS resolution, queries are marked as INSECURE.
When I ask for the same domain locally from the system where Pi-hole runs (localhost), the query is marked as SECURE.
When I use dig to from the client (192.168.1.102), the query is marked as SECURE as well.

The log above shows:

2022-12-20 12:53:52 - "$ ping seznam.cz" from Pi-hole system (localhost) - Status SECURE
2022-12-20 12:53:55 - "$ ping seznam.cz" from client (192.168.1.102) - Status INSECURE
2022-12-20 12:53:58 - "$ dig seznam.cz" from client (192.168.1.102) - Status SECURE

I have suspicion that the reason may be cased by the way how the client's system does the name resolution (it's Fedora 36 with systemd-resolved).
My theory is that when regular ping is run, systemd-resolved is used and it leads to the INSECURE status (as I see the INSECURE status also when I try to access the seznam.cz via web browser). But when I use dig, dig does the DNS query by itself not really relying on the systemd-resolved (?). But this is just a theory and I had no idea why would the way client asks Pi-hole for name resolution could affect the status to become INSECURE. My assumption of Pi-hole DNSSEC validation feature is that Pi-hole should do the DNSSEC validation if the record is signed no-matter what (no-matter how the client asks).

Anyone any idea?

Details about my system:

Pi-hole is running on RPi Zero with IP 192.168.1.224, listening on the default port 53.
Pi-hole has "Use DNSSEC" enabled.
Pi-hole has only one upstream DNS server configured (and none of the predefined is enabled).
The only DNS configured in the Pi-hole configuration is "127.0.0.1#54".
I'm using DoT (DNS over TLS) via Stubby running locally on the same RPi and listening on port 54, that's why the queries in the log are answered by the localhost#54 DNS on the screenshot above.
The Stubby is configured to use Open DNSSEC Validating Resolvers from CZ.NIC (Czech Domain Administrator) via the DoT [1].
The Stubby is also configured to validate DNSSEC.

Testing DNSSEC on the Pi-hole system via Delv and Dig confirms that answers from the Pi-hole as well as from the Stubby are DNSSEC validated:

Using Dig from the client in question (192.168.1.102), shows that the answer is DNSSEC validated:

See "ad" (Authenticated Data) flag in the header response, "do" (DNSSEC OK) flag and the RRSIG record all present in the dig output on the client.

[1] More info about the CZ.NIC resolvers including Stubby config for DoT is available here: CZ.NIC - Otevřené DNSSEC Validující Resolvery

s.beimer · December 21, 2022, 11:14am

I get similar results with midov.pl (or .pl in general). Looking at dnsviz it seems as if the dnssec setup is changing quite often at the moment ...
Look here: DNSSEC resolving problem with .pl domains - #16 by s.beimer

s.beimer · December 24, 2022, 5:14pm

I have no problems anymore with latest pihole and with pl domains.

Tojaj · December 24, 2022, 8:15pm

I have just updated:

  [✓] Update complete! 

  Current Pi-hole version is v5.14.2
  Current AdminLTE version is v5.18
  Current FTL version is v5.20

I still see the issue for that Linux (Fedora 37) host, its queries are always INSECURE
But queries from an Android phone or Samsung TV are returning SECURE.

Maybe the problem is not in the Pi-hole but in that Linux client.

DL6ER · December 26, 2022, 2:21pm

This is indeed a strange behavior. I did some tests on the domain seznam.cz myself here as I have a Fedora 37 system available. However, both dig and ping result in SECURE for me. ping is installed from iputils (version 20221126-1.fc37) on my system.

As a first test, could you check if you still see the same error when using a different DNSSEC-capable upstream server for your Pi-hole (such as Google) only momentarily during this test?

How much traffic is there on your system? If there isn't too much, you could also record your network traffic on the Pi-hole only for the time you are doing the two tests (one dig from local (SECURE) and then ping on the client (INSECURE)) so we can compare the queries at low-level. This can be done either from within Pi-hole or using tcpdump. I'll provide instructions if you consider this possible in your situation.

Tojaj · December 27, 2022, 6:36pm

Ok, I've just done some testing.

In the first part, I kept my current configuration (with local stubby on port 54 as the only DNS configured in the Pi hole).
In the second part, I used the google DNS (as predefined in the Pi hole config).

I did the testing with the Fedora 37 host (192.168.1.102) and with Android phone (192.168.1.114).

On the Fedora 37 (192.168.1.102), I run these three commands - "ping -c1 seznam.cz; dig seznam.cz; dig seznam.cz +dnssec"
On the Android phone (192.168.1.114), I was loading the seznam.cz homepage in web browser.

[ORIGINAL CONFIG]

[Original config] Android phone (192.168.1.114) - SECURE

Capture command:

pi@pihole:~/tcpdump $ sudo tcpdump -n -s 1500 udp and port 53 and host 192.168.1.114 -w dns-traffic-rpi-pi-hole-androidphone.pcap
tcpdump: listening on wlan0, link-type EN10MB (Ethernet), capture size 1500 bytes
^C32 packets captured
41 packets received by filter
0 packets dropped by kernel

PCAP file:
dns-traffic-rpi-pi-hole-androidphone.pcap (3.9 KB)

[Original config] Fedora 37 (192.168.1.102) - INSECURE

[tojaj@thinkpad ~]$ ping -c1 seznam.cz; dig seznam.cz; dig seznam.cz +dnssec
PING seznam.cz (77.75.77.222) 56(84) bytes of data.
64 bytes from www.seznam.cz (77.75.77.222): icmp_seq=1 ttl=54 time=8.46 ms

--- seznam.cz ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 8.457/8.457/8.457/0.000 ms

; <<>> DiG 9.18.8 <<>> seznam.cz
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26911
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;seznam.cz.			IN	A

;; ANSWER SECTION:
seznam.cz.		28	IN	A	77.75.77.222
seznam.cz.		28	IN	A	77.75.79.222

;; Query time: 1 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Dec 27 19:13:00 CET 2022
;; MSG SIZE  rcvd: 70


; <<>> DiG 9.18.8 <<>> seznam.cz +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20214
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 65494
;; QUESTION SECTION:
;seznam.cz.			IN	A

;; ANSWER SECTION:
seznam.cz.		28	IN	A	77.75.77.222
seznam.cz.		28	IN	A	77.75.79.222
seznam.cz.		28	IN	RRSIG	A 13 2 30 20230108070320 20221225053320 6277 seznam.cz. PkqA7Ify4DsgWxSKp2kSCpmNILHSZm9s5tTZ+pKssrRif3/CgJw2onZv JFgauZ5Jr5ABmCtBlWFGjpnkl8KtWw==

;; Query time: 1 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Dec 27 19:13:00 CET 2022
;; MSG SIZE  rcvd: 175

Note that for all three commands, there was only one DNS query (I believe this may be ok).

Capture command:

pi@pihole:~/tcpdump $ sudo tcpdump -n -s 1500 udp and port 53 and host 192.168.1.102 -w dns-traffic-rpi-pi-hole-fedora37.pcap
tcpdump: listening on wlan0, link-type EN10MB (Ethernet), capture size 1500 bytes
^C16 packets captured
23 packets received by filter
0 packets dropped by kernel

PCAP file:
dns-traffic-rpi-pi-hole-fedora37.pcap (3.5 KB)

[GOOGLE DNS]

[Google DNS] Android phone (192.168.1.114) - Mix od SECURE and INSECURE

Capture command:

pi@pihole:~/tcpdump $ sudo tcpdump -n -s 1500 udp and port 53 and host 192.168.1.114 -w dns-traffic-rpi-pi-hole-google-androidphone.pcap
tcpdump: listening on wlan0, link-type EN10MB (Ethernet), capture size 1500 bytes
^C78 packets captured
96 packets received by filter
0 packets dropped by kernel

PCAP file:
dns-traffic-rpi-pi-hole-google-androidphone.pcap (8.4 KB)

[Google DNS] Fedora 37 (192.168.1.102) - INSECURE

[tojaj@thinkpad ~]$ ping -c1 seznam.cz; dig seznam.cz; dig seznam.cz +dnssec
PING seznam.cz (77.75.77.222) 56(84) bytes of data.
64 bytes from www.seznam.cz (77.75.77.222): icmp_seq=1 ttl=54 time=11.5 ms

--- seznam.cz ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 11.454/11.454/11.454/0.000 ms

; <<>> DiG 9.18.8 <<>> seznam.cz
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44420
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;seznam.cz.			IN	A

;; ANSWER SECTION:
seznam.cz.		6	IN	A	77.75.77.222
seznam.cz.		6	IN	A	77.75.79.222

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Dec 27 19:28:23 CET 2022
;; MSG SIZE  rcvd: 70


; <<>> DiG 9.18.8 <<>> seznam.cz +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28824
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 65494
;; QUESTION SECTION:
;seznam.cz.			IN	A

;; ANSWER SECTION:
seznam.cz.		6	IN	A	77.75.79.222
seznam.cz.		6	IN	A	77.75.77.222
seznam.cz.		6	IN	RRSIG	A 13 2 30 20230108070320 20221225053320 6277 seznam.cz. PkqA7Ify4DsgWxSKp2kSCpmNILHSZm9s5tTZ+pKssrRif3/CgJw2onZv JFgauZ5Jr5ABmCtBlWFGjpnkl8KtWw==

;; Query time: 1 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Dec 27 19:28:23 CET 2022
;; MSG SIZE  rcvd: 175

Note that for all three commands, there was again only one DNS query.

Capture command:

pi@pihole:~/tcpdump $ sudo tcpdump -n -s 1500 udp and port 53 and host 192.168.1.102 -w dns-traffic-rpi-pi-hole-google-fedora37.pcap
tcpdump: listening on wlan0, link-type EN10MB (Ethernet), capture size 1500 bytes

PCAP file:
dns-traffic-rpi-pi-hole-google-fedora37.pcap (2.5 KB)

DL6ER · December 28, 2022, 2:26pm

Okay, thanks for that. The answer is indeed found in the pcap files:

The Android phone says: DNSSEC validation is necessary, non-authenticated data is unacceptable (emphasis on the highlighted line):

This is the default behavior of virtually any DNS client on this planet, well ... except ... your Fedora, apparently.

Your Fedora client says: DNSSEC validation is optional, non-authenticated data is acceptable:

This bit is called the "CD bit". It is defined in RFC 2535 Domain Name System Security Extensions, Section 6.1:

The CD (checking disabled) bit indicates in a query that Pending (non-authenticated) data is acceptable to the resolver sending the query.

Even more, the bind documentation says some words about it, but it is best summarized in man dig :

+[no]cdflag
Set or do not set the CD (checking disabled) bit in the query. This requests the server to not perform DNSSEC validation of responses.

(emphasis on the last sentence)

Indeed, we can now reproduce this behavior reliably just by setting the CD bit explicitly (this dig command is even reproducing the exact query of your Fedora client hence the other options):

dig seznam.cz +cdflag +bufsize=1472 +nocookie +dnssec +ednsopt=5:0507080a0d0e0f +ednsopt=6:010204 +ednsopt=7:01

Code-wise is this realized by:

github.com/pi-hole/FTL

src/dnsmasq/forward.c

cba74934b


      
          if (header->hb4 & HB4_CD)
            fwd_flags |= FREC_CHECKING_DISABLED;

and then

github.com/pi-hole/FTL

src/dnsmasq/forward.c

cba74934b


      
              server->mma_latency = (dnsmasq_milliseconds() - forward->forward_timestamp) * 128; /* init */
            else
              server->mma_latency += dnsmasq_milliseconds() - forward->forward_timestamp - server->query_latency;
            /* denominator controls how many queries we average over. */
            server->query_latency = server->mma_latency/128;
            
            
          #ifdef HAVE_DNSSEC
            if ((forward->sentto->flags & SERV_DO_DNSSEC) && 
                option_bool(OPT_DNSSEC_VALID) &&
                !(forward->flags & FREC_CHECKING_DISABLED))
              dnssec_validate(forward, header, n, STAT_OK, now);
            else
          #endif
              return_reply(now, forward, header, n, STAT_OK); 
          }
          
          static void return_reply(time_t now, struct frec *forward, struct dns_header *header, ssize_t n, int status)
          {
            int check_rebind = 0, no_cache_dnssec = 0, cache_secure = 0, bogusanswer = 0;
            size_t nn;

TL;DR: Everything is behaving as expected and designed. The client excplicitly requested data not to be checked - which is nothing else than INSECURE in Pi-hole

This can be understood as a measure to reduce DNS latency while throwing caution to the winds. DNS data will be returned faster (no validation is done + no DNSSEC chain is validated, triggering possibly further queries along the way) Interestingly, my local Fedora system 37 doesn't do this.

Tojaj · December 29, 2022, 12:03pm

@DL6ER thank you!

My resolved config on the Fedora was:

$ grep -i dnssec /etc/systemd/resolved.conf
DNSSEC=allow-downgrade

A man page for the config file:

$ man resolved.conf

says:

If set to "allow-downgrade" DNSSEC validation is attempted, but if the server does not support DNSSEC properly, DNSSEC mode is automatically disabled.

Anyway, I went and restored the default resolved.conf file:

$ rpm -qf /etc/systemd/resolved.conf
systemd-resolved-251.10-588.fc37.x86_64
$ sudo rm /etc/systemd/resolved.conf
$ sudo dnf reinstall -y systemd-resolved
$ rpm -V systemd-resolved
# No output here, which confirms that rpm version of the config file is installed now.
$ sudo reboot

After the restore and reboot, the answers for Fedora host are SECURE:

Note that the original resolved config has DNSSEC disabled by default:

$ grep -i dnssec /etc/systemd/resolved.conf
#DNSSEC=no

(Usually the commented values are the defaults that are in use)

The man page for the resolved.conf confirms this as it says that DNSSEC option:

Defaults to "no".

Runtime resolved status also says that DNSSEC is not enabled/unsupported:

$ resolvectl status
..
Link 3 (wlp58s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.1.224
       DNS Servers: 192.168.1.224
..

TL;DR: The default systemd-resolved configuration that has DNSSEC disabled (DNSSEC=no) leads to "SECURE" queries in the Pi hole log, while custom configuration where DNSSEC is enabled (either via DNSSEC=allow-downgrade or via DNSSEC=yes) leads to "INSECURE" queries in the Pi hole log.

Side note:
I checked the "DNSSEC=yes" resolved behavior on the packet level with Wireshark the same way as @DL6ER:

I confirm to see that resolved says non-auth data are acceptable, however I noticed, that resolved is also asking for DNSKEY records so my assumption is that even when resolved is asking DNS to provide non-auth response, it (resolved) does the DNSSEC validation locally on its own, see the "Data is authenticated: yes" here:

$ resolvectl query seznam.cz
seznam.cz: 2a02:598:a::79:222                  -- link: wlp58s0
           2a02:598:2::1222                    -- link: wlp58s0
           77.75.77.222                        -- link: wlp58s0
           77.75.79.222                        -- link: wlp58s0

-- Information acquired via protocol DNS in 436.9ms.
-- Data is authenticated: yes; Data was acquired via local or encrypted transport: no
-- Data from: network

(This above query was logged in the Pi hole log as INSECURE although resolved says it's authenticated)

system · January 19, 2023, 12:04pm

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.