Troubleshooting DNSSEC validation ABANDONED with my own domain

The issue I am facing:

  • I own a real domain.
  • I'm running Pi-Hole and CoreDNS within my LAN, with CoreDNS configured as the upstream resolver for Pi-Hole.
  • CoreDNS serves a zone for my domain containing A records pointing to internal IP addresses. It's configured to sign this zone with DNSSEC keys I've generated and saved, then to serve the signed zone file. It forwards requests for any other zone to Cloudflare's public resolver.
  • DNSSEC is enabled in Pi-Hole.
  • At my registrar, the nameservers for my domain point to Cloudflare and in Cloudflare I have records pointing to public IP addresses (the idea is that mydomain.com should resolve to a public IP address in general, but should instead resolve to a LAN IP address for clients within my LAN). DNSSEC is also enabled in Cloudflare and multiple DS records corresponding to the key used by Cloudflare and the key used by my private internal zone are both present at my registrar.
  • Although queries which are forwarded to Cloudflare are working fine and the responses for my domain look correct when I query CoreDNS directly (e.g. with dig), queries to Pi-Hole for my domain return SERVFAIL and in Pi-Hole's logs I see validation mydomain.com is ABANDONED.
  • The public part of the setup also works fine (I can resolve my domain to the public IP addresses without issue if I'm outside my LAN).
  • Also, if I disable DNSSEC in Pi-Hole, everything works. So the issue is definitely specific to Pi-Hole validating DNSSEC.

The setup:

Zone file:

$ORIGIN mydomain.com.

mydomain.com.			300	    IN	SOA	    mydomain.com. root.mydomain.com. 2274749221 10000 2400 604800 3600
mydomain.com.			300	    IN	NS	    mydomain.com.

$INCLUDE keys/Kmydomain.com.+<...>.key
$INCLUDE keys/Kmydomain.com.+<...>.ds

@                       IN  A       192.168.0.2
*                       IN  CNAME   @

CoreDNS Corefile:

mydomain.com {
    file db.mydomain.com.signed

    sign mydomain.com.zone {
        key file keys/Kmydomain.com<...>
        directory .
    }

    errors
    log
}

. {
    any
    forward . tls://1.1.1.1 tls://1.0.0.1 {
        tls_servername cloudflare-dns.com
    }
    errors
    log
}

As mentioned, if I query CoreDNS I receive the records as expected, containing the RRSIGs. But Pi-Hole's logs show:

Apr  8 12:09:57 dnsmasq[469]: query[A] mydomain.com from <Some client IP>
Apr  8 12:09:57 dnsmasq[469]: forwarded mydomain.com to <CoreDNS IP>
Apr  8 12:09:57 dnsmasq[469]: dnssec-query[DS] mydomain.com to  <CoreDNS IP>
Apr  8 12:09:57 dnsmasq[469]: validation mydomain.com is ABANDONED
Apr  8 12:09:57 dnsmasq[469]: reply mydomain.com is 192.168.0.2

I'm really not sure why Pi-Hole is abandoning the validation. CoreDNS logs NOERROR for the A and DS queries and indeed, if I query it directly, I get the DS record as expected. Is there a way to get more info from Pi-Hole about what is causing it to abandon the validation? Or is there something obvious I might be missing?

Thanks!

Do I read you right: It looks like the log would imply that Pi-hole does correctly resolve your domain, despite of abandoning DNSSEC validation, so your concern is about removing the warning?

Does tracing DNSSEC with dig reveal any abnormalities?

dig mydomain.com @<Pi-hole IP> +dnssec +trace

And could you share the results of dedicated lookups through both your CoreDNS and your Pi-hole?

dig mydomain.com @<CoreDNS IP> -p <CoreDNS port> +dnssec
dig mydomain.com @<Pi-hole IP> +dnssec

Well, it gets the correct response to the A record from CoreDNS, but it responds to the client with SERVFAIL. So a client using Pi-Hole isn't able to resolve my domain. My understanding is that with DNSSEC, Pi-Hole gets the response from its upstream but then performs the additional step of validating the signing chain, and will respond with an error to its client if that fails? Whereas without DNSSEC, it will just send that response back to its client (and indeed, my setup works as intended if I disable DNSSEC in Pi-Hole).

Thanks for the suggestions. I won't paste the full output to avoid sharing my domain but dig mydomain.com @<Pi-hole IP> +dnssec +trace returns:

  1. NS entries and an RRSIG for the root servers as returned by the Pi-Hole (I guess these are hardcoded?)
  2. NS, DS and RRSIG for my TLD as returned by the root servers
  3. NS (the ones configured at my registrar), DS (both of those configured at my registrar) and RRSIG (just a DS entry, would I expect one here also for NS?) for my domain as returned by my TLD's nameservers
  4. An SOA and an NSEC, with both corresponding RRSIGs, for my domain as returned by one of the Cloudflare nameservers (the SOA points to the other nameserver)

Does that sound correct to you? I guess one question I have is that it's following the public path, i.e. it's not showing anything from my CoreDNS server. Could this be the issue? I'm not sure whether there's some kind of setup that I'm missing because of course, the two nameservers configured at my registrar are Cloudflare's, but presumably I'm not going to add my internal server there?

dig mydomain.com @<CoreDNS IP> -p <CoreDNS port> +dnssec:

;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31704
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;mydomain.com.				IN	A

;; ANSWER SECTION:
mydomain.com.			3600	IN	A	192.168.0.2
mydomain.com.			3600	IN	RRSIG	A 13 2 3600 <snip> <snip> mydomain.com. <sig>

;; AUTHORITY SECTION:
mydomain.com.			300	IN	NS	mydomain.com.
mydomain.com.			300	IN	RRSIG	NS 13 2 300 <snip> <snip> mydomain.com. <sig>

;; Query time: 3 msec
;; SERVER: <CoreDNS IP>#<CoreDNS port>(<CoreDNS IP>)
;; WHEN: Fri Apr 08 15:31:10 BST 2022
;; MSG SIZE  rcvd: 299

dig mydomain.com @<Pi-hole IP> +dnssec returns:

;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 1031
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;mydomain.com.				IN	A

;; Query time: 4 msec
;; SERVER: <Pi-Hole IP>#53(<Pi-Hole IP>)
;; WHEN: Fri Apr 08 15:36:59 BST 2022
;; MSG SIZE  rcvd: 35

Perhaps worth noting, dig mydomain.com @<CoreDNS IP> -p <CoreDNS port> +dnssec +trace returns the results for the root servers but then times out with connection timed out; no servers could be reached. But as shown above, if I remove +trace, I get the result.

If you'd prefer, you may upload the output to Pi-hole's tricorder, where only trusted members of the Pi-hole team will be able to access it, and it will auto-delete after 48 hours.
Direct output to a file e.g. results.log, redact if required and then run:

cat results.log | pihole tricorder

Then provide us with the token as shown once the upload completes.

Could be the case.
While the domain itself is resolved through CoreDNS, the DNS queries for DNSSEC validation will walk the chain of the authoritative DNS servers, which would be the public ones. This would true if your <my.domain> is really the same as the domain you own <mydomain.tld>, and you were not delegating to your private zone handled by your CoreDNS as a subdomain <internal.mydomain.tld> (which is a bit difficult to tell if you are obfuscating actual names).

The following digs may reveal additional details for where your validation stops:
Retrieve and store the DNS root servers DNSKEYs:

dig . DNSKEY | grep -Ev '^($|;)' > root.dnskeys

Use those DNSKEYs to validate your domain:

dig +sigchase +trusted-key=./root.dnskeys <my.domain>

EDIT:
What's the reason for employing DNSSEC for your private range DNS records that are solely looked up by local clients?

If you can do without DNSSEC, removing DNSSEC from your zone in CoreDNS would allow resolution.

Another option would be to have Pi-hole's embedded dnsmasq forward requests for <mydomain.com> specifically to the IP of your CoreDNS using server=/<my.domain>/<CoreDNS-IP>, which would disable DNSSEC validation at first.

But if you'd then insist on DNSSEC, you could try to provide a trust-anchor option for <my.domain>, which may allow your Pi-hole to successfully validate DNSSEC for your local zone in CoreDNS (note: I have never done this myself).

You'd have to apply both of those options in a custom dnsmasq configuration file (e.g. /etc/dnsmasq.d/42-local-dnssec.conf) - see dnsmasq's documentation for further details.

Interestingly, if I try to dig +sigchase, pointing it directly to CoreDNS, it fails to validate. Here's the bottom portion of the output:

;; WE HAVE MATERIAL, WE NOW DO VALIDATION
;; VERIFYING A RRset for mydomain.com. with DNSKEY:<id>: success
;; OK We found DNSKEY (or more) to validate the RRset
;; Now, we are going to validate this DNSKEY by the DS
;; OK a DS valids a DNSKEY in the RRset
;; Now verify that this DNSKEY validates the DNSKEY RRset
;; VERIFYING DNSKEY RRset for mydomain.com. with DNSKEY:<id>: success

;; OK this DNSKEY (validated by the DS) validates the RRset of the DNSKEYs, thus the DNSKEY validates the RRset
;; Now, we want to validate the DS : recursive call

;; DNSKEYset that signs the RRset to chase:
mydomain.com. 300 IN DNSKEY <snip>
;; RRSIG of the DNSKEYset that signs the RRset to chase:
mydomain.com. 300 IN RRSIG DNSKEY <snip>
;; DSset of the DNSKEYset
mydomain.com. 3600 IN DS <snip>
;; RRSIG of the DSset of the DNSKEYset
mydomain.com. 3600 IN RRSIG DS <snip>

;; WE HAVE MATERIAL, WE NOW DO VALIDATION
;; VERIFYING DS RRset for mydomain.com. with DNSKEY:<id>: RRSIG failed to verify
;; No DNSKEY is valid to check the RRSIG of the RRset: FAILED

This is with the DS record added in CoreDNS. Otherwise, it just tells me that there's no DS record and stops there.

Yes, you're right that I don't need DNSSEC for the internal DNS, but initially I set up the zone in CoreDNS without DNSSEC and Pi-Hole would log BOGUS (NSEC(3) missing) for all queries for my domain. Do you know of any way around that? Otherwise, either of the dnsmasq config options you mentioned sound like they should work, but it seems simplest to see whether I can just remove DNSSEC from the internal zone and get Pi-Hole to be happy with that.

I guess we are seeing DNSSEC doing its job: :wink:
It will neither allow your CoreDNS to spoof unsigned nor improperly signed DNS records for a public domain.

You may be able to get this to work with DNSSEC if your DNS provider's public DNS server would delegate a subdomain of your domain to your local coreDNS server (as hinted at in my previous post) - but this doesn't seem to be what you want to achieve: You want to shadow your public domain's public IP address by private range IP addresses for queries from your private network.

If you'd want to configure that by means of split zones, you'd need to do so in your DNS provider's public DNS server. I don't know if your domain's DNS provider would allow you to do so, and even if they would, they may not allow you to configure private range IP adddresses.

You still have a couple of options to tackle this within your private network:

If your sole upstream CoreDNS would do DNSSEC validation, there would be no harm in disabling DNSSEC in Pi-hole, apart from losing a bit of DNSSEC statistics (which may be still be available form CoreDNS - I am not familiar with CoreDNS).

If you want to keep DNSSEC enabled in Pi-hole, I'd recommend to go with the server option, to skip DNSSEC for your public domain only.

Alternatively, you may consider to provide the respective Local DNS records within Pi-hole. Just like your CoreDNS, Pi-hole would then serve these records itself directly.

I guess we are seeing DNSSEC doing its job: :wink:
It will neither allow your CoreDNS to spoof unsigned nor improperly signed DNS records for a public domain.

Yeah, maybe I'm just trying to abuse the system lol. It's not a particularly easy system to understand nor troubleshoot I guess, and I've found little information about setting up this particular use case (which is probably understandable). People have seemed to indicate it's possible but maybe I just misunderstand :person_shrugging:

You may be able to get this to work with DNSSEC if your DNS provider's public DNS server would delegate a subdomain of your domain to your local coreDNS server

Ah, I missed this bit:

While the domain itself is resolved through CoreDNS, the DNS queries for DNSSEC validation will walk the chain of the authoritative DNS servers, which would be the public ones. This would true if your <my.domain> is really the same as the domain you own <mydomain.tld>, and you were not delegating to your private zone handled by your CoreDNS as a subdomain <internal.mydomain.tld> (which is a bit difficult to tell if you are obfuscating actual names).

Nope, everywhere I've put mydomain.com it's just that, not dealing with any subdomains or delegation. I guess you're right though, having a delegated subdomain might be an option, but I'm not sure whether that's the best solution for me.

If your sole upstream CoreDNS would do DNSSEC validation

Not sure CoreDNS can do DNSSEC validation when acting as a forwarder. Maybe there are better choices out there, I picked it because it seemed relatively lightweight, easy to run in Docker, easy to configure and can act as the authoritative server for a zone while forwarding other queries using one of the encrypted DNS protocols. Maybe there are better solutions for my use case though!

Alternatively, you may consider to provide the respective Local DNS records within Pi-hole. Just like your CoreDNS, Pi-hole would then serve these records itself directly.

I tried this initially but wanted to have wildcards, which Pi-Hole's Local DNS feature doesn't support AFAIK.

If you want to keep DNSSEC enabled in Pi-hole, I'd recommend to go with the server option, to skip DNSSEC for your public domain only.

So yeah, this is what I've just done and it's working. I think this will sort me out for now, hadn't thought of this / realised it was an option. Thanks a lot for your help!

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.