getOverTimeID is too large warning

General
1

I opened a thread several weeks ago about the dreaded "getOverTimeID is too large" warning message. Someone mentioned resolving this by running fake-hwclock, which my Debian 9 install on my PogoPlug didn't include by default, so I installed that. It definitely fixes the issue of the box starting up with time set to zero - 1/1/1970 @ 00:00.

But I booted up a PogoPlug running pihole today that hadn't been booted in a couple of weeks, so fake-hwclock set the date at startup to about 2 weeks ago, and that was enough to cause pihole to complain with the getOverTimeID is too large warning.

Does anybody know what the threshold is for it to consider the time delta "too large"? I'm trying to find a permanent fix to this error. I did create a perl script that can be inserted in the pihole startup script that will wait for NTP to set the time before starting pihole-FTL, but pihole software updates can overlay that local mod, so it's not really ideal.

Suggestions? I'm curious what the significance of this warning message is anyway, i.e. why does it care?

2

This is one of the instances that can only be solved reliably by hardware.

You could consider acquiring an RTC, if that would be compatible with your device. However, I am unfamiliar with any PogoPlug devices, so I wouldn't know whether yours already contains a battery-backuped RTC or not (in which case fake-hwclock would make little sense).

fake-hwclock does two things:
It reads the current time from the system clock and writes it to a file on certain occasions (by default, that's every hour and on controlled shutdowns), and it reads the time from that same file at system startup and sets your system clock accordingly.

Later during system operation, a time-sync service (i.e. another piece of s/w entirely) may then acquire actual time information via an NTP server and update your system clock, provided your device has internet connectivity.

While fake-hwclock is certainly better as having no time reference at all, it still would do little more than assure that your time always moves forward as expected (as long as NTP services are accessible).

Specifically, your time will always be off after prolongued periods of shutdown.
Depending on your system configuration, your system's time-sync service may then refuse to adjust the system clock from NTP information if it considers the gap to be too large.
In such a case, you would have to manuallly adjust time information on your device.

Apart from Pi-hole's UI not being able to produce accurate results, a whole bunch of cryptographic algorithms rely on a common time-frame on all involved peers.
For example, if you were to use DNSSEC, record signature validation would fail, and thus DNS itself, resulting in Pi-hole's refusal to resolve any domain, which in turn would look like an internet outage for all clients.

So if you would want to use DNSSEC, and your PogoPlug does not feature an RTC already, I'd recommend to check whether your PogoPlug would support to connect a battery-backuped RTC module and buy one of those, e.g. a DS3231 isn't very expensive.

1 Like
3

Are your NTP servers IP addresses or are you using FQDNs?

If they are IP addresses and the issue is merely that pihole-FTL is starting before NTP has time to correct, then you can use the delay startup setting for FTL.

That does require that the NTP service is able to update the clock accurately. If the delta is too large then NTP will not update without manual intervention.

4

Yeah, the problem is pihole-FTL starts a few seconds before NTP has set the system time. I did try the DELAY_STARTUP option, but that didn't work. Here are my log entries showing it did delay starting the resolver until NTP had set the time properly

[1969-12-31 18:01:01.477 1642M] Sleeping for 3 seconds as requested by configuration ..
[2021-03-27 06:24:06.702 1642M] Done sleeping, continuing startup of resolver...

but I still got the errors. DL6ER said it was because most of the startup is done before the delay is processed. There was a suggestion of moving the delay earlier, immediately after processing options file, but I don't think it went any further than just a suggestion. See this thread: getOverTimeID is too large warning - #8 by jfb

5

How long a delay did you try? Just the 3 seconds? Does 10 or more work?

6

Thanks for your reply, Bucking_Horn. PogoPlug is a small ARM architecture device with no RTC, and no way to add one that I know of. These devices were originally intended for cloud backup, etc, but the company has been out of business for I'm guessing about 5 years, so some people (like me) are repurposing them for other things.

I'm not having any problem with NTP setting the time - even when the device initially starts up with time set to 1/1/1970. The only problem is when pihole-FTL starts up before NTP has had a chance to finish setting the time. The DELAY_STARTUP would do just what I need if it waited until AFTER the delay to set the start_time variable it's using to determine that "getOverTimeID is too large".

7

I didn't try a longer delay time since it looked like the system time was set properly at the end of the 3-second delay. I assumed a longer delay wouldn't make a difference since DL6ER said the delay didn't work because of where it was being processed in the pi-hole code.

8

Wouldn't this only be a problem if NTP never updated the system time? i.e. shouldn't DNSSEC be fine as long as system time is accurate when the DNS request is made? Or Is it just a warning that the time may be off, based on when it thinks pihole-FTL started up?

9

I was answering your inquiry about the significance of the message you observed.
You are right in assuming that actual repercussions may trigger only on certain conditions (though it is irrelevant by what means your system time has been set - if it's off, then certain methods may fail as explained).

If acquiring the answer through a public NTP server takes too much time., you may consider using your router as an NTP source, provided your router supports it (and ideally has RTC backup :wink: ).

10

DELAY_STARTUP was originally intended for cases where the network interface was not up before FTL started.

We had the discussion of moving the delay to stop everything until the delay times out but I feel that is inefficient. I think that we should be doing housekeeping during the delay and loading up the database in the background.

I think this case here is a far edge case that shouldn't cause us to change the whole behavior of the delay. It's nice to be able to use old hardware and I used to have a few Pogos running linux chrooted but they shouldn't be running a vital network service like DNS if they aren't a reliable platform.

11

Wouldn't this be an issue for any device that doesn't have a hardware clock, including the Raspberry Pi? Wondering why more people running pi-hole on their Raspberry Pi aren't reporting this.

12

You can add an RTC to a RPi for $5USD.

And most of our users wouldn't have a DNS server that was off for a couple of weeks at a time.

13

Good point... :wink:

14

So, I took a look at the code to understand the reason for the warning message, and I see it is in overTime.c, when OverTimeID is > the number of OVERTIME_SLOTS - 1. Can anyone explain what overTime.c is used for? Does this have something to do with loading the database, or possibly related to the graphs and stats on the dashboard?

Inquiring minds want to know.... ;-D

15

See Delay earlier if requested by DL6ER · Pull Request #1090 · pi-hole/FTL · GitHub still awaiting final review + approve or dismissal.

Yes, this. FTL aims at both efficiency and as small as possible memory footprint. Hence, the overTime slots are static and relative to the current system time. Each slot corresponds to one bar in this graph:

Screenshot from 2021-05-09 08-22-37
Screenshot from 2021-05-09 08-22-371000×253 20 KB

When loading queries from the database, we bin them into these slots. The last slot is "right now", the first one is "24 hours ago". So if your system clock says "last week" and we import a query that is today (or, in fact, any more recent than two weeks ago), it'd have to be binned into a slot that is in the future (remember that our system clock tells us that it is still two weeks ago). However, FTL knows that a query in the future is not a good idea so it complains. We should improve this warning message. Maybe ensure it is only printed once.

This all has no effect on how your DNS server works, it just means that some of the statistics computed from the imported queries will be incorrect.

1 Like
16

@rsantag Would you mind trying

pihole checkout ftl tweak/incorrect_hwclock_warning

and check if this gives you a better (as in more understandable) warning message about the queries in the future? There are plans to go through all warnings and improve their usefulness for non-developers but this hasn't started, yet.

17

Thanks very much for the explanation. That helps a lot.

I'll give the tweak a try and let you know.

18

Yes, definitely clearer description

[2021-05-09 10:36:59.030 1653/T1657] Compiled 0 whitelist and 0 blacklist regex filters for 0 clients in 6.3 msec
[2021-05-09 10:38:06.613 1653M] WARN: Found database entries in the future (1620574500, last: 0). Your over-time statistics may be incorrect
[2021-05-09 10:38:06.647 1653M] New upstream server: 192.168.1.254:53 (0/1024)
[2021-05-09 10:38:54.364 1653M] New upstream server: 8.8.4.4:53 (1/1024)

I also noticed I only get the message once rather than for every query. That will also make a huge difference, so that pihole-FTL.log doesn't get huge (especially if you don't have logrotate installed)

19
20

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.