Pi-hole v5.13 (467,000 domains in blocklist)
FTL v5.18.2
Web Interface v5.16
Unbound version: 1.13.1
Stubby version: 0.3.0
Debian 11
I'm making a new pi-hole lxc and I was testing some queries, and it appears that non-cached queries are extremely slow, or timeout all together. I'm doing all my testing using nslookup and dig from the pi-hole container itself.
Queries direct to 1.1.1.1 are extremely fast and come back immediately, so I have an issue somewhere in the pihole/unbound/stubby chain that is slowing down the new query process. If dig is timing out, then there is an unacceptable level of latency in the system even if the query eventually returns.
The way I have this setup is pihole configured to use unbound (query addr is 127.0.0.1:5335), and then unbound has a forward-addr configured for stubby (127.0.0.1:8053). Then stubby is configured to do DNS over TLS to 1.0.0.1 and 1.1.1.1.
/etc/unbound/unbound.conf.d/stubby.conf
server:
do-not-query-localhost: no
forward-zone:
name: "."
forward-addr: 127.0.0.1@8053
stubby.yml
#
# This is a yaml version of the stubby configuration file (it replaces the
# json based stubby.conf file used in earlier versions of getdns/stubby).
#
# For more information see
# https://dnsprivacy.org/wiki/display/DP/Configuring+Stubby
#
# This format does not fully support all yaml features - the restrictions are:
# - the outer-most data structure must be a yaml mapping
# - mapping keys must be yaml scalars
# - plain scalars will be converted to json unchanged
# - non-plain scalars (quoted, double-quoted, wrapped) will be interpreted
# as json strings, i.e. double quoted.
# - yaml tags are not supported
# - IPv6 addresses ending in :: are not yet supported (use ::0)
#
# Note that we plan to introduce a more compact format for defining upstreams
# in future: https://github.com/getdnsapi/stubby/issues/79
# Logging is currently configured at runtime using command line arguments. See
# > stubby -h
# for details.
# Specifies whether to run as a recursive or stub resolver
# For stubby this MUST be set to GETDNS_RESOLUTION_STUB
resolution_type: GETDNS_RESOLUTION_STUB
# Ordered list composed of one or more transport protocols:
# GETDNS_TRANSPORT_UDP, GETDNS_TRANSPORT_TCP or GETDNS_TRANSPORT_TLS
# If only one transport value is specified it will be the only transport used.
# Should it not be available basic resolution will fail.
# Fallback transport options are specified by including multiple values in the
# list. Strict mode (see below) should use only GETDNS_TRANSPORT_TLS.
dns_transport_list:
- GETDNS_TRANSPORT_TLS
# Selects Strict or Opportunistic Usage profile as described in
# https://datatracker.ietf.org/doc/draft-ietf-dprive-dtls-and-tls-profiles/
# Strict mode requires that authentication information for the upstreams is
# specified below. Opportunistic may fallback to clear text DNS if UDP or TCP
# is included in the transport list above.
# For Strict use GETDNS_AUTHENTICATION_REQUIRED
# For Opportunistic use GETDNS_AUTHENTICATION_NONE
tls_authentication: GETDNS_AUTHENTICATION_REQUIRED
# EDNS0 option to pad the size of the DNS query to the given blocksize
# 128 is currently recommended by
# https://tools.ietf.org/html/draft-ietf-dprive-padding-policy-03
tls_query_padding_blocksize: 128
# EDNS0 option for ECS client privacy as described in Section 7.1.2 of
# https://tools.ietf.org/html/rfc7871
edns_client_subnet_private : 1
# EDNS0 option for keepalive idle timeout in ms as specified in
# https://tools.ietf.org/html/rfc7828
# This keeps idle TLS connections open to avoid the overhead of opening a new
# connection for every query.
idle_timeout: 10000
# Set the listen addresses for the stubby DAEMON. This specifies localhost IPv4
# and IPv6. It will listen on port 53 by default. Use <IP_address>@<port> to
# specify a different port
listen_addresses:
- 127.0.0.1@8053
# Instructs stubby to distribute queries across all available name servers.
# Set to 0 to treat the upstreams below as an ordered list and use a single
# upstream until it becomes unavailable, then use the next one.
round_robin_upstreams: 1
# Require DNSSEC validation. For releases earlier than 1.2 a trust anchor must
# be configured configured manually. This can be done with unbound-anchor.
dnssec_return_status: GETDNS_EXTENSION_TRUE
# Specify the location of the installed trust anchor file (leave commented out
# for zero configuration DNSSEC)
# dnssec_trust_anchors: "/etc/unbound/getdns-root.key"
dnssec_trust_anchors: "/var/lib/unbound/root.key"
# Specify the list of upstream recursive name servers to send queries to
# In Strict mode upstreams need either a tls_auth_name or a tls_pubkey_pinset
# so the upstream can be authenticated.
# The list below includes all the available test servers but only has the subset
# operated the stubby/getdns developers enabled. You can enable any of the
# others you want to use by uncommenting the relevant section. See:
# https://dnsprivacy.org/wiki/display/DP/DNS+Privacy+Test+Servers
# If you don't have IPv6 then comment then out those upstreams.
# In Opportunistic mode they only require an IP address in address_data.
# The information for an upstream can include the following:
# - address_data: IPv4 or IPv6 address of the upstream
# port: Port for UDP/TCP (default is 53)
# tls_auth_name: Authentication domain name checked against the server
# certificate
# tls_pubkey_pinset: An SPKI pinset verified against the keys in the server
# certificate
# - digest: Only "sha256" is currently supported
# value: Base64 encoded value of the sha256 fingerprint of the public
# key
# tls_port: Port for TLS (default is 853)
upstream_recursive_servers:
# IPv4 addresses
# The 1.1.1.1 Cloudflare Servers
- address_data: 1.1.1.1
tls_auth_name: "cloudflare-dns.com"
- address_data: 1.0.0.1
tls_auth_name: "cloudflare-dns.com"
root@DNS:/# tail -n20 /var/log/pihole.log
Nov 11 16:20:26 dnsmasq[1420]: query[A] facebook.com from 172.20.5.2
Nov 11 16:20:26 dnsmasq[1420]: forwarded facebook.com to 127.0.0.1#5335
Nov 11 16:20:31 dnsmasq[1420]: query[A] facebook.com from 172.20.5.2
Nov 11 16:20:31 dnsmasq[1420]: forwarded facebook.com to 127.0.0.1#5335
Nov 11 16:20:36 dnsmasq[1420]: query[A] facebook.com from 172.20.5.2
Nov 11 16:20:36 dnsmasq[1420]: forwarded facebook.com to 127.0.0.1#5335
Nov 11 16:20:46 dnsmasq[1420]: query[A] facebook.com from 172.20.5.2
Nov 11 16:20:46 dnsmasq[1420]: forwarded facebook.com to 127.0.0.1#5335
Nov 11 16:20:51 dnsmasq[1420]: query[A] facebook.com from 172.20.5.2
Nov 11 16:20:51 dnsmasq[1420]: forwarded facebook.com to 127.0.0.1#5335
Nov 11 16:20:56 dnsmasq[1420]: query[A] facebook.com from 172.20.5.2
Nov 11 16:20:56 dnsmasq[1420]: forwarded facebook.com to 127.0.0.1#5335
Nov 11 16:21:03 dnsmasq[1420]: query[A] facebook.com from 172.20.5.2
Nov 11 16:21:03 dnsmasq[1420]: forwarded facebook.com to 127.0.0.1#5335
Nov 11 16:21:08 dnsmasq[1420]: query[A] facebook.com from 172.20.5.2
Nov 11 16:21:08 dnsmasq[1420]: forwarded facebook.com to 127.0.0.1#5335
Nov 11 16:21:11 dnsmasq[1420]: validation result is INSECURE
Nov 11 16:21:11 dnsmasq[1420]: reply facebook.com is 157.240.241.35
Nov 11 16:22:12 dnsmasq[1420]: query[A] facebook.com from 172.20.5.2
Nov 11 16:22:12 dnsmasq[1420]: cached facebook.com is 157.240.241.35
root@DNS:/# tail -n50 /var/log/unbound.log
Nov 11 21:20:56 unbound[1869:1] info: query response was THROWAWAY
Nov 11 21:20:56 unbound[1869:1] info: 127.0.0.1 facebook.com. A IN
Nov 11 21:20:56 unbound[1869:1] info: resolving facebook.com. A IN
Nov 11 21:21:01 unbound[1869:1] info: response for facebook.com. DS IN
Nov 11 21:21:01 unbound[1869:1] info: reply from <.> 127.0.0.1#8053
Nov 11 21:21:01 unbound[1869:1] info: query response was THROWAWAY
Nov 11 21:21:01 unbound[1869:1] info: DS response was error, thus bogus
Nov 11 21:21:01 unbound[1869:1] info: DS response was error, thus bogus
Nov 11 21:21:01 unbound[1869:1] info: DS response was error, thus bogus
Nov 11 21:21:01 unbound[1869:1] info: DS response was error, thus bogus
Nov 11 21:21:01 unbound[1869:1] info: DS response was error, thus bogus
Nov 11 21:21:01 unbound[1869:1] info: DS response was error, thus bogus
Nov 11 21:21:01 unbound[1869:1] info: resolving facebook.com. DS IN
Nov 11 21:21:03 unbound[1869:1] info: 127.0.0.1 facebook.com. A IN
Nov 11 21:21:03 unbound[1869:1] info: resolving facebook.com. A IN
Nov 11 21:21:06 unbound[1869:1] info: response for facebook.com. DS IN
Nov 11 21:21:06 unbound[1869:1] info: reply from <.> 127.0.0.1#8053
Nov 11 21:21:06 unbound[1869:1] info: query response was THROWAWAY
Nov 11 21:21:08 unbound[1869:1] info: 127.0.0.1 facebook.com. A IN
Nov 11 21:21:08 unbound[1869:1] info: resolving facebook.com. A IN
Nov 11 21:21:11 unbound[1869:1] info: response for facebook.com. DS IN
Nov 11 21:21:11 unbound[1869:1] info: reply from <.> 127.0.0.1#8053
Nov 11 21:21:11 unbound[1869:1] info: query response was THROWAWAY
Nov 11 21:21:11 unbound[1869:1] info: response for facebook.com. DS IN
Nov 11 21:21:11 unbound[1869:1] info: reply from <.> 127.0.0.1#8053
Nov 11 21:21:11 unbound[1869:1] info: query response was nodata ANSWER
Nov 11 21:21:11 unbound[1869:1] info: NSEC3s for the referral proved no DS.
Nov 11 21:21:11 unbound[1869:1] info: NSEC3s for the referral proved no DS.
Nov 11 21:21:11 unbound[1869:1] info: NSEC3s for the referral proved no DS.
Nov 11 21:21:11 unbound[1869:1] info: NSEC3s for the referral proved no DS.
Nov 11 21:21:11 unbound[1869:1] info: NSEC3s for the referral proved no DS.
Nov 11 21:21:11 unbound[1869:1] info: NSEC3s for the referral proved no DS.
Nov 11 21:21:11 unbound[1869:1] info: NSEC3s for the referral proved no DS.
Nov 11 21:21:11 unbound[1869:1] info: NSEC3s for the referral proved no DS.
Nov 11 21:21:11 unbound[1869:1] info: Verified that unsigned response is INSECURE
Nov 11 21:21:11 unbound[1869:1] info: 127.0.0.1 facebook.com. A IN NOERROR 45.143035 0 57
Nov 11 21:21:11 unbound[1869:1] info: Verified that unsigned response is INSECURE
Nov 11 21:21:11 unbound[1869:1] info: 127.0.0.1 facebook.com. A IN NOERROR 35.147836 0 57
Nov 11 21:21:11 unbound[1869:1] info: Verified that unsigned response is INSECURE
Nov 11 21:21:11 unbound[1869:1] info: 127.0.0.1 facebook.com. A IN NOERROR 19.683813 0 57
Nov 11 21:21:11 unbound[1869:1] info: Verified that unsigned response is INSECURE
Nov 11 21:21:11 unbound[1869:1] info: 127.0.0.1 facebook.com. A IN NOERROR 24.684877 0 57
Nov 11 21:21:11 unbound[1869:1] info: Verified that unsigned response is INSECURE
Nov 11 21:21:11 unbound[1869:1] info: 127.0.0.1 facebook.com. A IN NOERROR 8.101218 0 57
Nov 11 21:21:11 unbound[1869:1] info: Verified that unsigned response is INSECURE
Nov 11 21:21:11 unbound[1869:1] info: 127.0.0.1 facebook.com. A IN NOERROR 14.682845 0 57
Nov 11 21:21:11 unbound[1869:1] info: Verified that unsigned response is INSECURE
Nov 11 21:21:11 unbound[1869:1] info: 127.0.0.1 facebook.com. A IN NOERROR 40.148779 0 57
Nov 11 21:21:11 unbound[1869:1] info: Verified that unsigned response is INSECURE
Nov 11 21:21:11 unbound[1869:1] info: 127.0.0.1 facebook.com. A IN NOERROR 3.100717 0 57
Those logs are when I tried using dig @172.20.5.2 facebook.com a couple times in a row, all of which timed out ;; connection timed out; no servers could be reached
After waiting a full minute and trying the dig again, it came back immediately. I assumed the original query finally made it back to pihole and got cached though.
What are some other ways to troubleshoot this and find where exactly is causing this slow down?
Any off-the-cuff ideas for what could be wrong?