Cd.cz Czech National Railway Company domain name not resolving in pi-hole unbound

Peterkal2112 · March 14, 2024, 4:02pm

The issue I am facing:
can not open one domain of Czech National Railway Company cd.cz

Details about my system:

VM of Debian 12 Bookworm x64 running pi-hole with unbound as recursive DNS, 1 main DNS VIP address for 2 pi-hole via keepalived with same pi-hole running on physical rpi2 as a backup failover using this setup synced via gravity sync (when one is offline, the other one will work). OPNSense as router with unbound OFF on opnsense (pi-hole and unbound is running only on pi-hole VM / rpi2) used this setup.

This everything working like it should, same issues is on physical rpi pi-hole and VM one...

uname -a

Linux pihole 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux

pi-hole showing me

OK (answered by localhost#5335)
BOGUS (refused upstream) SERVFAIL (0.5ms)

tried to get access that site with DNSSEC on and off, same result...

dig cd.cz

; <<>> DiG 9.18.19-1~deb12u1-Debian <<>> cd.cz
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 27244
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;cd.cz. IN A

;; Query time: 31 msec
;; SERVER: 192.168.1.1#53(192.168.1.1) (UDP)
;; WHEN: Thu Mar 14 16:44:03 CET 2024
;; MSG SIZE rcvd: 34

timedatectl status

           Local time: Thu 2024-03-14 17:01:09 CET
       Universal time: Thu 2024-03-14 16:01:09 UTC
             RTC time: Thu 2024-03-14 16:01:09
            Time zone: Europe/Prague (CET, +0100)

System clock synchronized: yes
NTP service: active
RTC in local TZ: no

Running on protectli VP2420
OPNSense in VM, 2 NIC are PCIe Passthroygh to OPNSense VM. (as DHCP server)
Pi-hole in VM
hypervisor: proxmox.

pi-hole VM:

neofetch

root@pihole
-----------
OS: Debian GNU/Linux 12 (bookworm) x86_64
Host: KVM/QEMU (Standard PC (i440FX + PIIX, 1996) pc-i440fx-8.1)
Kernel: 6.1.0-18-amd64
Uptime: 18 mins
Packages: 1474 (dpkg)
Shell: bash 5.2.15
Resolution: 1280x800
Terminal: /dev/pts/0
CPU: Intel Celeron J6412 (4) @ 1.996GHz
GPU: 00:02.0 Vendor 1234 Device 1111
Memory: 288MiB / 1966MiB

Edge browser showing me!

DNS_PROBE_FINISHED_NXDOMAIN

Opera Browser showing me:

ERR_NAME_NOT_RESOLVED

In Firefox it is working from third party nextdns.io provider but from internal pi-hole it is not resolving.

Hmm. We’re having trouble finding that site.

We can’t connect to the server at cd.cz.

If you entered the right address, you can:
Try again later
Check your network connection
Check that Firefox has permission to access the web (you might be connected but behind a firewall)

in cd.cz android mobile app not working.

On Mobile data it is working.

when I check tail pihole.log and F5 refreshing that site, I have the same result:

Mar 14 17:13:29: query[A] cd.cz from 192.168.1.1
Mar 14 17:13:29: forwarded cd.cz to 127.0.0.1#5335
Mar 14 17:13:29: validation cd.cz is BOGUS
Mar 14 17:13:29: reply error is SERVFAIL
Mar 14 17:13:29: query[HTTPS] cd.cz from 192.168.1.1
Mar 14 17:13:29: forwarded cd.cz to 127.0.0.1#5335
Mar 14 17:13:29: validation cd.cz is BOGUS
Mar 14 17:13:29: reply error is SERVFAIL
Mar 14 17:13:29: query[A] cd.cz from 192.168.1.1
Mar 14 17:13:29: forwarded cd.cz to 127.0.0.1#5335
Mar 14 17:13:29: validation cd.cz is BOGUS
Mar 14 17:13:29: reply error is SERVFAIL
Mar 14 17:13:29: query[HTTPS] cd.cz from 192.168.1.1
Mar 14 17:13:29: forwarded cd.cz to 127.0.0.1#5335
Mar 14 17:13:29: validation cd.cz is BOGUS
Mar 14 17:13:29: reply error is SERVFAIL

I am adding diagnosis file debug log token:
https://tricorder.pi-hole.net/hLsmJnwK/

My pi-hole VM is rebooting every hour in 45th minute just to be sure the dns resolving are working (after reboot it is always working)

First one is pi-hole VM (the main) restarting every hour in 45th minute

Second one is pi-hole on rpi2 (backup failover) resolving everything after every 45th minute of every hour while the main VM pi-hole is restarted and up and running, everything works great...

What I have changed since installing Pi-hole:

Nothing, never worked...

Bucking_Horn · March 14, 2024, 5:02pm

It is unbound that is returning BOGUS for cd.cz, so your issue is with unbound's DNSSEC validation.

DNSSEC configuration of authoritative DNS servers for cd.cz looks ok (currently), suggesting the issues lies somewhere between unbound and the authoritative DNS servers.

A common cause for failing DNSSEC validation would be wrong time and time zone information on the client. This would affect all DNS resolution, though - as you report cd.cz as failing, we can rule this one out.

If dns-utils or bind9-dnsutils are installed, the Domain Entity Lookup and Validation tool delv may provide more insights into why DNSSEC validation fails.

Run from your VM hosting unbound/Pi-hole, what's the result of the following command:

delv @127.0.0.1 -p 5335 +rtrace cd.cz

EDIT: Your output should look similar to mine:

;; fetch: cd.cz/A
;; fetch: cd.cz/DNSKEY
;; fetch: cd.cz/DS
;; fetch: cz/DNSKEY
;; fetch: cz/DS
;; fetch: ./DNSKEY
; fully validated
cd.cz.			400	IN	A	82.117.140.29
cd.cz.			400	IN	RRSIG	A 13 2 400 20240323225228 20240309221634 64022 cd.cz. HBhZ7E0AS7McYc0NJMJbXug6/nkEETSjiUtJQ9aZl0lNSAZq4tcEZjB9 kngGYd+tUnUi+oQHjbqVWP/Hc0fPdQ==

Peterkal2112 · March 14, 2024, 5:28pm

delv @127.0.0.1 -p 5335 +rtrace cd.cz

;; fetch: cd.cz/A
;; resolution failed: SERVFAIL

Bucking_Horn · March 15, 2024, 6:38am

So its not failing on DNSSEC validation itself, but on fetching the A record.

Let's see if we can find a hint why that would fail.
Please share the output of:

dig +trace cd.cz

Peterkal2112 · March 15, 2024, 7:34am

dig +trace cd.cz

; <<>>DiG 9.18.19-1~deb12u1-Debian <<>>+trace cd.cz
;; global options: +cmd
.                       83504   IN      NS      a.root-servers.net.
.                       83504   IN      NS      b.root-servers.net.
.                       83504   IN      NS      c.root-servers.net.
.                       83504   IN      NS      d.root-servers.net.
.                       83504   IN      NS      e.root-servers.net.
.                       83504   IN      NS      f.root-servers.net.
.                       83504   IN      NS      g.root-servers.net.
.                       83504   IN      NS      h.root-servers.net.
.                       83504   IN      NS      i.root-servers.net.
.                       83504   IN      NS      j.root-servers.net.
.                       83504   IN      NS      k.root-servers.net.
.                       83504   IN      NS      l.root-servers.net.
.                       83504   IN      NS      m.root-servers.net.
.                       83504   IN      RRSIG   NS 8 0 518400 20240328050000 20240315040000 30903 . 5hkhdm2QI7IiVVPpn4QQrWu+n1S9LzyptO3/EwwTHtCWRmWgWoTWBaGa XP7Awra7enj05HLcOAM0lfpj9KI3yjLk9XKjqNXG8Wnep2Mj0z21SC33 bT6UwEBXKDhq0iYzcy6eUrEJJPowgSbC1JdLhtkHEknDAfdJCBoRqFPs sRLNMmKjCkK1HvYu8FFtPf7KsWAHEN8DI3729f8sXlB/KTAqlO+ooBZV wszpUVA5CdMlmTJFsoh9qhwhNMfTxj1Vn7AgxBDTaQcwNr0eOFFmDJOx E2DnlAY93TjIElyHEklyXP3085TS1lCK0WPjIKs4ilgz8mjGUKviMq2T B9RESQ==
;; Received 537 bytes from 192.168.1.1#53(192.168.1.1) in 3 ms

;; UDP setup with 2001:503:c27::2:30#53(2001:503:c27::2:30) for cd.cz failed: network unreachable.
;; UDP setup with 2001:503:c27::2:30#53(2001:503:c27::2:30) for cd.cz failed: network unreachable.
;; UDP setup with 2001:503:c27::2:30#53(2001:503:c27::2:30) for cd.cz failed: network unreachable.
cz.                     172800  IN      NS      a.ns.nic.cz.
cz.                     172800  IN      NS      b.ns.nic.cz.
cz.                     172800  IN      NS      c.ns.nic.cz.
cz.                     172800  IN      NS      d.ns.nic.cz.
cz.                     86400   IN      DS      20237 13 2 CFF0F3ECDBC529C1F0031BA1840BFB835853B9209ED1E508FFF48451 D7B778E2
cz.                     86400   IN      RRSIG   DS 8 1 86400 20240328050000 20240315040000 30903 . sJcDsPCyeqXXqXwfb7e4n1o99pYb+UoJYCsw1yJQvoODmBYq8xJzfNqU dxKOWnOxTjus92Qv5Y/xoB0wmOfe1DFJ5cBZmdmJmtJ5pEjomuAds3Ec GrEMqdFtdfFm0QMsrrAbZrnpPF1iZ/YsDVdqk95c03CIeV+Qdzk1/yDz LFIQIdbnknvTjkfe2u1Fiqstn3KGKTQOyuLPzn44kNIK9BNbCx0U2cYO P6jrUYPqE2PdLR/q8nUZjtuJwoY4XMG2LKWU+mRVI8w2Ov4cFc1IqZps 1EfFOIPw+3MVtkRtxpXEg20ztIM+qgSV7UWJVpUybU8u2XkoLMp6RJNb VChbig==
;; Received 616 bytes from 170.247.170.2#53(b.root-servers.net) in 23 ms

;; UDP setup with 2001:678:10::1#53(2001:678:10::1) for cd.cz failed: network unreachable.
;; UDP setup with 2001:678:1::1#53(2001:678:1::1) for cd.cz failed: network unreachable.
cd.cz.                  3600    IN      NS      ns2.cdt.cz.
cd.cz.                  3600    IN      NS      ns1.cdt.cz.
cd.cz.                  3600    IN      DS      64022 13 2 7B121FC9398097BB52C234D016D919EC55BBA86FC53E65AC7810D8D0 EF0AB7E2
cd.cz.                  3600    IN      RRSIG   DS 13 2 3600 20240319222134 20240305205134 54801 cz. sjLkOpQEE7GsSd8w72K3ewwxn6z7WMVpEYd93ld5TqiA8C+WTe1jxXhu Em24VfBs5MRF7hPqMiVsZ0fO0fsIWA==
couldn't get address for 'ns2.cdt.cz': failure
couldn't get address for 'ns1.cdt.cz': failure
dig: couldn't get address for 'ns2.cdt.cz': no more

Bucking_Horn · March 15, 2024, 8:47am

Your dig seems to have an issue with contacting a root server via IPv6 (IPv6 address is one of j.root-servers.net.):

;; UDP setup with 2001:503:c27::2:30#53(2001:503:c27::2:30) 
for cd.cz failed: network unreachable.

What happens if you force dig to use strictly IPv4?

dig -4 +trace cd.cz

Peterkal2112 · March 15, 2024, 10:31am

dig -4 +trace cd.cz

; <<>>DiG 9.18.19-1~deb12u1-Debian <<>>-4 +trace cd.cz
;; global options: +cmd
.                       83734   IN      NS      i.root-servers.net.
.                       83734   IN      NS      l.root-servers.net.
.                       83734   IN      NS      d.root-servers.net.
.                       83734   IN      NS      a.root-servers.net.
.                       83734   IN      NS      g.root-servers.net.
.                       83734   IN      NS      e.root-servers.net.
.                       83734   IN      NS      h.root-servers.net.
.                       83734   IN      NS      c.root-servers.net.
.                       83734   IN      NS      j.root-servers.net.
.                       83734   IN      NS      k.root-servers.net.
.                       83734   IN      NS      b.root-servers.net.
.                       83734   IN      NS      m.root-servers.net.
.                       83734   IN      NS      f.root-servers.net.
.                       83734   IN      RRSIG   NS 8 0 518400 20240328050000 20240315040000 30903 . 5hkhdm2QI7IiVVPpn4QQrWu+n1S9LzyptO3/EwwTHtCWRmWgWoTWBaGa XP7Awra7enj05HLcOAM0lfpj9KI3yjLk9XKjqNXG8Wnep2Mj0z21SC33 bT6UwEBXKDhq0iYzcy6eUrEJJPowgSbC1JdLhtkHEknDAfdJCBoRqFPs sRLNMmKjCkK1HvYu8FFtPf7KsWAHEN8DI3729f8sXlB/KTAqlO+ooBZV wszpUVA5CdMlmTJFsoh9qhwhNMfTxj1Vn7AgxBDTaQcwNr0eOFFmDJOx E2DnlAY93TjIElyHEklyXP3085TS1lCK0WPjIKs4ilgz8mjGUKviMq2T B9RESQ==
;; Received 537 bytes from 192.168.1.1#53(192.168.1.1) in 3 ms

cz.                     172800  IN      NS      a.ns.nic.cz.
cz.                     172800  IN      NS      b.ns.nic.cz.
cz.                     172800  IN      NS      c.ns.nic.cz.
cz.                     172800  IN      NS      d.ns.nic.cz.
cz.                     86400   IN      DS      20237 13 2 CFF0F3ECDBC529C1F0031BA1840BFB835853B9209ED1E508FFF48451 D7B778E2
cz.                     86400   IN      RRSIG   DS 8 1 86400 20240328050000 20240315040000 30903 . sJcDsPCyeqXXqXwfb7e4n1o99pYb+UoJYCsw1yJQvoODmBYq8xJzfNqU dxKOWnOxTjus92Qv5Y/xoB0wmOfe1DFJ5cBZmdmJmtJ5pEjomuAds3Ec GrEMqdFtdfFm0QMsrrAbZrnpPF1iZ/YsDVdqk95c03CIeV+Qdzk1/yDz LFIQIdbnknvTjkfe2u1Fiqstn3KGKTQOyuLPzn44kNIK9BNbCx0U2cYO P6jrUYPqE2PdLR/q8nUZjtuJwoY4XMG2LKWU+mRVI8w2Ov4cFc1IqZps 1EfFOIPw+3MVtkRtxpXEg20ztIM+qgSV7UWJVpUybU8u2XkoLMp6RJNb VChbig==
;; Received 616 bytes from 192.5.5.241#53(f.root-servers.net) in 3 ms

cd.cz.                  3600    IN      NS      ns1.cdt.cz.
cd.cz.                  3600    IN      NS      ns2.cdt.cz.
cd.cz.                  3600    IN      DS      64022 13 2 7B121FC9398097BB52C234D016D919EC55BBA86FC53E65AC7810D8D0 EF0AB7E2
cd.cz.                  3600    IN      RRSIG   DS 13 2 3600 20240319222134 20240305205134 54801 cz. sjLkOpQEE7GsSd8w72K3ewwxn6z7WMVpEYd93ld5TqiA8C+WTe1jxXhu Em24VfBs5MRF7hPqMiVsZ0fO0fsIWA==
couldn't get address for 'ns1.cdt.cz': failure
couldn't get address for 'ns2.cdt.cz': failure
dig: couldn't get address for 'ns1.cdt.cz': no more

Bucking_Horn · March 15, 2024, 2:39pm

So no network unreachable with IPv4 only, but it still fails to retrieve the IPs of the authoritative DNS servers ns1.cdt.cz. and ns2.cdt.cz..

Do those domains resolve by themselves?

delv @127.0.0.1 -p 5335 +rtrace ns1.cdt.cz

dig ns1.cdt.cz

Peterkal2112 · March 15, 2024, 2:57pm

delv @127.0.0.1 -p 5335 +rtrace ns1.cdt.cz

;; fetch: ns1.cdt.cz/A
;; resolution failed: SERVFAIL

dig ns1.cdt.cz

; <<>> DiG 9.18.19-1~deb12u1-Debian <<>> ns1.cdt.cz
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 62113
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;ns1.cdt.cz. IN A

;; Query time: 3 msec
;; SERVER: 192.168.1.1#53(192.168.1.1) (UDP)
;; WHEN: Fri Mar 15 15:57:11 CET 2024
;; MSG SIZE rcvd: 39

Bucking_Horn · March 15, 2024, 3:19pm

ns1.cdt.cz indeed does not resolve from your machine.
The answer should have been 81.19.33.2.

Let's see if we would've got a reply for cd.cz if we queried that IP directly:

dig -4 +norecurse cd.cz @81.19.33.2

Mine returns:

; <<>> DiG 9.18.24-1-Raspbian <<>> -4 +norecurse cd.cz @81.19.33.2
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34444
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 18f1063a7691671f0100000065f466fd9583b80bc072ea79 (good)
; EDE: 18 (Prohibited)
;; QUESTION SECTION:
;cd.cz.				IN	A

;; ANSWER SECTION:
cd.cz.			400	IN	A	82.117.140.29

;; AUTHORITY SECTION:
cd.cz.			86400	IN	NS	ns2.cdt.cz.
cd.cz.			86400	IN	NS	ns1.cdt.cz.

;; Query time: 19 msec
;; SERVER: 81.19.33.2#53(81.19.33.2) (UDP)
;; WHEN: Fri Mar 15 16:19:25 CET 2024
;; MSG SIZE  rcvd: 126

While that comes with an EDE code 18 (Prohibited), it still returns the A record with status NOERROR.

Peterkal2112 · March 15, 2024, 3:20pm

dig -4 +norecurse cd.cz @81.19.33.2

; <<>> DiG 9.18.19-1~deb12u1-Debian <<>> -4 +norecurse cd.cz @81.19.33.2
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 37024
;; flags: qr ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; EDE: 20 (Not Authoritative): (ABC4)
;; QUESTION SECTION:
;cd.cz. IN A

;; Query time: 0 msec
;; SERVER: 81.19.33.2#53(81.19.33.2) (UDP)
;; WHEN: Fri Mar 15 16:20:17 CET 2024
;; MSG SIZE rcvd: 44

Bucking_Horn · March 15, 2024, 3:54pm

For some reason, querying the authoritative DNS server for cd.cz was REFUSED, and rather quickly:

A response in zero time would suggest that answer has been generated by the same device, or by a device very near to it. Is there something in your network that could block access?

Apart from that observation, as I received an unexpected EDE code as well, that could hint at a configuration issue on the authoritative server.
It may limit access to certain known good IP addresses, or refuse access for known bad ones, and perhaps your current public IP landed on a bad list, or it may not handle things like EDNS correctly.
Note that this is purely guesswork on my behalf.

Short of contacting the DNS maintainers for cd.cz, there seems nothing you could do to allow recursion to succeed for that domain.

If this persists even if you received a new public IP (e.g. after a router restart), then you could try to have Pi-hole by-pass unbound for that specific domain.

Try creating a custom dnsmasq configuration, e.g. at /etc/dnsmasq.d/42-bypass-cz-railway.conf with the following contents:

# allow *.cd.cz to resolve, 
# as unbound's recursion currently fails (03/2024)
server=/cd.cz/9.9.9.9

then run a dnsmasq syntax check:

pihole-FTL dnsmasq-test

If ok, restart Pi-hole:

pihole restartdns

Peterkal2112 · March 15, 2024, 5:58pm

there should be nothing in my network that will block that (only pi-hole doing bloking of dns)

I have ipv6 off..

I have my public IP static so it is not changing...

This provided "solution", I am thank you for that, with all respect i do not consider that as solution.

I can provide more information if needed....

Bucking_Horn · March 20, 2024, 11:10am

Your dig results have established that your issue is with the authoritative DNS servers for cd.cz.
Note that neither Pi-hole nor unbound were involved in that.

dig +trace would walk the chain of DNS servers from the root servers to the final authoritative DNS server for cd.cz, just as unbound as a recursive resolver does.

And the final dig that you ran was a direct request for the A record for cd.cz to one of those authoritative DNS servers (81.19.33.2), and that server REFUSED your request.

Neither my unbound nor public DNS resolvers seem to have issues resolving that domain, and it is remarkable that 81.19.33.2 chooses to send different replies to me and to you.

That would suggest that the issue lies somewhere upstream of your client, between you and the authoritative DNS servers for cd.cz.

As explained, short of contacting the DNS maintainers for cd.cz and
inquire with them why their DNS servers would do so, we have no indication that you could do anything to change that on your end, aside from my suggested workaround to by-pass unbound for that domain.

Your actual failure happens earlier during recursion, when trying to acquire an IP address of the authoritative DNS servers for cd.cz..

You could try to step into that resolution chain by the same means we've used so far, or enable unbound's logging to see if it would shed some additional insights.
I'd probably leave verbosity as is and start by adding the following options to your pi-hole.conf:

    val-log-level: 2
    log-servfail: yes
    log-time-ascii: yes

But I am afraid as we already know that the ultimate DNS request for the A record to the authoritative DNS server also fails for you specifically, the end result will be the same.

Peterkal2112 · March 20, 2024, 11:41am

Thank you..
I checked that domain in Domain DNS Health Checker - Check DNS, Blacklist, Email, MX Health (dnschecker.org)
and there are some issues with that domain name...

So the problem is not in my Pi-Hole, it is not in unbound, it is not in my network...
But it is on the side of root DNS serve ???
Or the problem is in my ISP network ??
Or the problem is somwhere else ?

Bucking_Horn · March 20, 2024, 11:54am

That may contribute to your issue, but if it was just that, then you, me and every one else with a similar setup would experience the same - but that's not the case.

That would suggest the issue is specific to something in your DNS resolution chain, caused by either something in your network or on the route from your ISP to the authoritative DNS servers.

As another thought, perhaps unbound support may be more versed in tracing down issues like that.
It may be worth approaching them with your observation.
While they may also be unable to fix this, they probably have better chances to pinpoint a cause.

system · April 10, 2024, 11:54am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.