Every 2 minutes 0.debian.pool.ntp.org queries by both Raspberry Pi's in my network

[tomislief]

Expected Behaviour:

[Few queries to an NTP server.]

Actual Behaviour:

[Every 2 minutes, both Raspberry Pi's in my network are querying 0.debian.pool.ntp.org queries. ]

The issue started occurring when I enabled IPv6 on my router.
I have installed Unbound and I am using the Pihole's DHCP. Both also have IPv6 enabled.
I have checked the the time and date on both Raspberry Pi's, these are correct.
Pinging 0.debian.pool.ntp.org works fine.

I did not have ~2300 queries/day to the NTP server when IPv6 was disabled (not sure how many I had), is this normal?
It seems to me that no information is returned as I see NODATA as reply in the log.

Help and/or insights are much appreciated!

Last 8 queries for 0.debian.pool.ntp.org:

|2022-06-08 11:50:13|AAAA|0.debian.pool.ntp.org|localhost|OK (cache)
INSECURE|NODATA (0.0ms)|Blacklist|
|2022-06-08 11:50:13|A|0.debian.pool.ntp.org|localhost|OK, answered by localhost#5335
INSECURE|IP (0.7ms)|Blacklist|
|2022-06-08 11:50:07|AAAA|0.debian.pool.ntp.org|PrinterServer.lan|OK, answered by localhost#5335
INSECURE|NODATA (90.9ms)|Blacklist|
|2022-06-08 11:50:07|A|0.debian.pool.ntp.org|PrinterServer.lan|OK, answered by localhost#5335
INSECURE|IP (1.3ms)|Blacklist|
|2022-06-08 11:47:43|AAAA|0.debian.pool.ntp.org|localhost|OK (cache)
INSECURE|NODATA (0.0ms)|Blacklist|
|2022-06-08 11:47:43|A|0.debian.pool.ntp.org|localhost|OK, answered by localhost#5335
INSECURE|IP (0.9ms)|Blacklist|
|2022-06-08 11:47:37|AAAA|0.debian.pool.ntp.org|PrinterServer.lan|OK (cache)
INSECURE|NODATA (0.1ms)|Blacklist|
|2022-06-08 11:47:37|A|0.debian.pool.ntp.org|PrinterServer.lan|OK, answered by localhost#5335
INSECURE|IP (1.4ms)|Blacklist|

Debug Token:

https://tricorder.pi-hole.net/PpjeMm0T/

[korhant]

I'm using NTP Daemon on my Raspberry Pi, in the /etc/ntp.conf file you can set min and max poll intervals from time server. My ntp.conf file has a line:
server time.cloudflare.com iburst minpoll 12 maxpoll 12
min-max poll time could be an integer from 4 to 17. 4 means '2 to the power of 4 seconds (16 sec)' and 17 means '2 to the power of 17 seconds (roughly one and a half day)' I use Cloudflare's NTP services.

[tomislief]

Limiting the poll interval to the time server is a solution but I am not sure if it would solve the underlaying problem. Actually, I do not know if the current amount of queries is an indication for a problem.

Pinging 0.debian.pool.ntp.org works fine btw.

[deHakkelaar]

@tomislief

|2022-06-08 11:50:13|AAAA|0.debian.pool.ntp.org|localhost|OK (cache)
INSECURE|NODATA (0.0ms)|Blacklist|

It appears from above log entry this domain is on the "Blacklist".
Check your blacklist!

Also excessive NTP querying might be related to a recent openresolv package change that comes with Raspbian/Pi-OS Bullseye and maybe also Debian and derivatives:

DNSMASQ_WARN for Max Concurrent Queries (max is 150) when using VPN or when using Pi-Hole as local network DNS with 35+ devices

I just followed the instructions and it now works, its no longer pinging the debian.pool.ntp servers. Thank you so much, I'm doing a student showcase for my college about this and was in a panic to get it fixed.

You would be affected if you have below similar lines in unbound's configuration:

@Norlig

pi@raspberrypi:/etc/dnsmasq.d $ sudo grep -v '#\|^$' -R /etc/unbound/unbound.conf*
[..]
/etc/unbound/unbound.conf.d/resolvconf_resolvers.conf:forward-zone:
/etc/unbound/unbound.conf.d/resolvconf_resolvers.conf:  name: "."
/etc/unbound/unbound.conf.d/resolvconf_resolvers.conf:  forward-addr: 192.168.10.1

[tomislief]

I copy-pasted the query log from the admin interface, "blacklist" the label of the button
The queries are not blacklisted, sorry for the confusion.

I checked /etc/unbound/unbound.conf and it contained no similar lines.
/etc/resolvconf.conf however contains this line: unbound_conf=/etc/unbound/unbound.conf.d/resolvconf_resolvers.conf
I have commented out this line, not deleted the unwanted unbound configuration file (although instructed) and restarted unbound. Unfortunately no improvement.

I am not using VPN btw.
The issue started when I enabled IPv6 on my router, unbound and pihole. No issues when using IPv4.
Therefore I am not sure that the issue could be related to the opensesolv package that you mentioned.

I tried disabling IPv6 on unbound and pihole, rebooted the raspberry, no change.
The time server queries stopped when I disabled IPv6 on my router.

[deHakkelaar]

@tomislief

I checked /etc/unbound/unbound.conf and it contained no similar lines.
/etc/resolvconf.conf however contains this line: unbound_conf=/etc/unbound/unbound.conf.d/resolvconf_resolvers.conf
I have commented out this line, not deleted the unwanted unbound configuration file (although instructed) and restarted unbound . Unfortunately no improvement.

Check if that file exists:

stat /etc/unbound/unbound.conf.d/resolvconf_resolvers.conf

If exists, check its content:

cat /etc/unbound/unbound.conf.d/resolvconf_resolvers.conf

If exists, you should remove/delete it because it configures unbound like a regular caching forwarding DNS server instead of a recursive one.
And can cause other issues like a DNS loop depending on what was in /etc/resolv.conf.

[tomislief]

/etc/unbound/unbound.conf.d/resolvconf_resolvers.conf exists
It contains this:

# Generated by resolvconf

forward-zone:
        name: "lan"
        forward-addr: 2a02:8109:8680:2ad8:a29c:eb:72e:4ff3

forward-zone:
        name: "."
        forward-addr: 208.67.222.222
        forward-addr: 208.67.220.220
        forward-addr: 2a02:8109:8680:2ad8:a29c:eb:72e:4ff3
        forward-addr: 2a02:8109:8680:2ad8:c.....:a51c

etc/resolv.conf contains this:

# Generated by resolvconf
domain lan
nameserver 127.0.0.1

unbound-resolvconf.service appears to be running
etc/dhcpcd.conf somehow still contains static addresses to OpenDNS servers that I used before using unbound:

static domain_name_servers=208.67.222.222 208.67.220.220

I guess that my pihole is still using OpenDNS servers?
Should some file not point to 127.0.0.1#5335 so the port is added too?
Should I disable unbound-resolvconf.service?
Should I proceed with removing /etc/unbound/unbound.conf.d/resolvconf_resolvers.conf?
If I do this, should I comment out this line unbound_conf=/etc/unbound/unbound.conf.d/resolvconf_resolvers.conf in /etc/resolvconf.conf?

I am sorry for the amount of questions, I am just trying to avoid a situation in which nothing is working anymore and I have no clue what to do

[jfb]

Here is the permanent fix for your unbound configuration change (you have only done part of this):

1. Edit file /etc/resolvconf.conf and comment out the last line which should read:

unbound_conf=/etc/unbound/unbound.conf.d/resolvconf_resolvers.conf

2. Delete the unwanted unbound configuration file:

sudo rm /etc/unbound/unbound.conf.d/resolvconf_resolvers.conf

3. Restart unbound:

sudo service unbound restart

[deHakkelaar]

@tomislief

etc/resolv.conf contains this:

# Generated by resolvconf
domain lan
nameserver 127.0.0.1

Plus you'll notice that if you disable and stop that unbound-resolvconf.service unit with below:

sudo systemctl disable --now unbound-resolvconf.service

And restart the dhcpcd.service unit:

sudo systemctl restart dhcpcd.service

The /etc/resolv.conf file will be populated with the DNS nameserver(s) you defined in the /etc/dhcpcd.conf file instead of the loopback IP 127.0.0.1:

pi@ph5b:~ $ systemctl status unbound-resolvconf.service
● unbound-resolvconf.service - Unbound DNS server via resolvconf
     Loaded: loaded (/lib/systemd/system/unbound-resolvconf.service; disabled; vendor preset: enabled)
     Active: inactive (dead)

pi@ph5b:~ $ cat /etc/dhcpcd.conf
[..]
  static domain_name_servers=10.0.0.1

pi@ph5b:~ $ cat /etc/resolv.conf
[..]
nameserver 10.0.0.1

EDIT: Those two packages, unbound and openresolv, try hard to break unbound's recursion and mess up resolv.conf

[tomislief]

Many thanks for your help!

I have done both (comment out the line, deleted the unwanted file, restarted unbound + disabled the service).
Last night at around 3am the pihole reduced itself to querying the time server about every 30 minutes. No idea why! The other raspberry pi in my network (printerserver) did not change its frequency of once every 2 minutes.
The last ~50 minutes I have not seen another time server query from my pihole, the other raspberry pi keeps querying every 2 minutes.
So my problem is partially resolved.

I would like to use unbound and not the OpenDNS servers that are now listed in /etc/dhcpcd.conf and /etc/resolv.conf. Do I just change the addresses below into 127.0.0.1#5335 in /etc/dhcpcd.conf, however this was called a "loopback IP"?

# fallback to static profile on eth0
#interface eth0
#fallback static_eth0
interface eth0
        static ip_address=192.168.1.150/24
        static routers=192.168.1.1
        static domain_name_servers=208.67.222.222 208.67.220.220

[deHakkelaar]

@tomislief

Do I just change the addresses below into 127.0.0.1#5335 in /etc/dhcpcd.conf , however this was called a "loopback IP"?

I would advice against that.
These domain_name_servers that end up in resolv.conf are used by local processes running on the Pi-hole host and are independent from what you've configured as upstream DNS servers in the Pi-hole settings.
Meaning if you loopback queries for them back to Pi-hole, you wouldn't have DNS when you mess up Pi-hole again tweaking etc
This is disadvantageous if you lose DNS while troubleshooting.
I have my router IP configured.
But you could also opt to enter your ISP DNS servers or whatever other external DNS provider you prefer.

Also you cant define a port (#5335 or whatever) in dhcpcd.conf and is also not valid in resolv.conf.

[tomislief]

I am starting to understand these systems now (slightly)

Two things puzzle me though. First, why I am starting to have this issue when I enable IPv6? All was good when I was using IPv4.
Second, why does my other raspberry pi (cups printerserver) in my network still have the issue? It has no unbound installed. Is there anything that I can do about this?

I would love to know more about this!

[deHakkelaar]

@tomislief

First, why I am starting to have this issue when I enable IPv6? All was good when I was using IPv4.

It might have to do with the IPv6 GUA prefix changing by your ISP upstream which comes from your router via "router advertisement" or short RA:

IPv6 | Global addressing

Which in turn triggers a series of events which does not happen with IPv4.
Below posting hints to RA ... and the thread is full of related info:

WARNING: Raspbian October 2021 release bullseye + unbound

It's only an issue if openresolv is installed instead of classic/slim resolvconf, and it caused other issues already like restarting Unbound every time an RA with changed info is received. Basically it doesn't seem to work well at all with setups where a local resolver is meant to be used by the network instead of by the local host only. So instead of working around all the issues that openresolv causes, what about replacing it with resolvconf when installed as part of the Pi-hole installer?

EDIT:

@tomislief

Second, why does my other raspberry pi (cups printerserver) in my network still have the issue? It has no unbound installed. Is there anything that I can do about this?

Have you tried a good old trusty reboot?

[tomislief]

Thanks for the information, I will check it out!

I tried restarting all systems, no improvement..

[tomislief]

With journalctl -u systemd-timesyncd.service I checked the pihole and the other raspberry pi in my network, both still seem to successfully sync the time ~2 minutes for some time already. It seems that because I enabled IPv6 on my router and pihole, the NTP sync queries became visible in the pihole query log.
Somehow currently the pihole time sync is not included in the pihole query log anymore after the actions mentioned in one of the previous posts.

I will look into limiting the sync process as suggested earlier in this thread.

[deHakkelaar]

Some pointers concerning systemd-timesyncd.service that comes default with Pi-OS/Raspbian.
You already figured out to list the systemd logs:

journalctl --full --no-pager -u systemd-timesyncd.service

Below one shows time/date and if NTP is active and syncing:

pi@ph5b:~ $ timedatectl
               Local time: Fri 2022-06-10 19:35:14 CEST
           Universal time: Fri 2022-06-10 17:35:14 UTC
                 RTC time: n/a
                Time zone: Europe/Amsterdam (CEST, +0200)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Below one shows details about the syncing:

pi@ph5b:~ $ timedatectl timesync-status
       Server: 185.216.161.42 (0.debian.pool.ntp.org)
Poll interval: 34min 8s (min: 32s; max 34min 8s)
         Leap: normal
      Version: 4
      Stratum: 1
    Reference: GPS
    Precision: 2us (-19)
Root distance: 45us (max: 5s)
       Offset: -2.118ms
        Delay: 17.115ms
       Jitter: 2.938ms
 Packet count: 2654
    Frequency: -17.507ppm

Am not sure but from above it looks like systemd-timesyncd is suppose to poll every 34 minutes when successful and goes down to 32 seconds when it fails syncing.
Maybe someone else can confirm or debunk?

Or if you want to monitor:

timedatectl --monitor timesync-status

If install ntpdate:

pi@ph5b:~ $ sudo apt install ntpdate
[..]
After this operation, 234 kB of additional disk space will be used.

You can manually test connectivity with NTP server IP's or DNS pool names:

pi@ph5b:~ $ ntpdate -q 0.debian.pool.ntp.org
server 162.159.200.123, stratum 3, offset -0.004728, delay 0.05746
server 45.85.15.35, stratum 2, offset -0.002805, delay 0.03673
server 5.79.108.34, stratum 2, offset -0.003996, delay 0.03769
server 188.165.11.86, stratum 2, offset -0.004380, delay 0.04564
10 Jun 19:46:18 ntpdate[6372]: adjust time server 45.85.15.35 offset -0.002805 sec

Above is also a good test to see if the 0.debian.pool.ntp.org name resolves to IP's properly on the host itself (IP's in the pool are a bit dynamic):

pi@ph5b:~ $ dig +short 0.debian.pool.ntp.org a
188.165.11.86
162.159.200.123
45.85.15.35
5.79.108.34

[deHakkelaar]

@deHakkelaar

Am not sure but from above it looks like systemd-timesyncd is suppose to poll every 34 minutes when successful and goes down to 32 seconds when it fails syncing.
Maybe someone else can confirm or debunk?

Yeah I waited a good 15 minutes and --monitor now shows differrent stats as before (EDIT: and Packet count went up by one):

pi@ph5b:~ $ timedatectl --monitor timesync-status
       Server: 185.216.161.42 (0.debian.pool.ntp.org)
Poll interval: 34min 8s (min: 32s; max 34min 8s)
         Leap: normal
      Version: 4
      Stratum: 1
    Reference: GPS
    Precision: 2us (-19)
Root distance: 45us (max: 5s)
       Offset: -3.340ms
        Delay: 22.399ms
       Jitter: 3.365ms
 Packet count: 2655
    Frequency: -18.322ppm

[tomislief]

The details about the sync helped!
My Raspberry Pi's each have a poll interval of 2min 8s, sometimes it switches to 1min 4s.
Both Raspberry's seems to resolve the IP addresses properly.

My poll interval's are frequent but the time is correct so I will leave it at that.

What I do not understand is that the NTP queries of the Raspberry on which the Pihole is installed are not included in the pihole query log and the NTP queries of the other raspberry pi in my network are included.

[deHakkelaar]

@tomislief

My Raspberry Pi's each have a poll interval of 2min 8s, sometimes it switches to 1min 4s.

Check if below OOTB defaults have been changed:

pi@ph5b:~ $ cat /etc/systemd/timesyncd.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.

[Time]
#NTP=
#FallbackNTP=0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org
#RootDistanceMaxSec=5
#PollIntervalMinSec=32
#PollIntervalMaxSec=2048

The hashed out lines with directives are the defaults (note above two last lines!):

pi@ph5b:~ $ man timesyncd.conf
[..]
       PollIntervalMinSec=, PollIntervalMaxSec=
           The minimum and maximum poll intervals for NTP messages. Each
           setting takes a time value (in seconds).  PollIntervalMinSec=
           must not be smaller than 16 seconds.  PollIntervalMaxSec=
           must be larger than PollIntervalMinSec=.  PollIntervalMinSec=
           defaults to 32 seconds, and PollIntervalMaxSec= defaults to
           2048 seconds.

If thats not it, I'm outa ideas and know too little about what influences the pol frequency.
I would suggest to post a support question on the Raspbian/Pi-hole Raspberry Pi forums as this does not appear to be a Pi-hole issue.
EDIT: And if get to a solution, could you feedback here pls?

@tomislief

What I do not understand is that the NTP queries of the Raspberry on which the Pihole is installed are not included in the pihole query log and the NTP queries of the other raspberry pi in my network are included.

Thats because local processes including systemd-timesyncd use the DNS nameserver(s) defined in resolv.conf:

@deHakkelaar

These domain_name_servers that end up in resolv.conf are used by local processes running on the Pi-hole host and are independent from what you've configured as upstream DNS servers in the Pi-hole settings.

Check /etc/resolv.conf on respective hosts!

[tomislief]

The default settings on both raspberry's have not changed. I will not investigate this any further. The poll interval is frequent but the times are correct so essentially there is no problem.

It makes sense that the pihole is not using itself
/etc/resolv.conf contains an IPv6 DNS address of my internet service provider. This address is passed on to the other raspberry pi as well. I can comment it out, it however is uncommented after a reboot. Can I somehow block the address elsewhere?