# getOverTimeID is too large warning

**URL:** <https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824>\
**Category:** General\
**Created:** [May 5, 2021, 7:00pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824 "2021-05-05T19:00:45Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![rsantag](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/r/5f8ce5/32.png) [@rsantag](https://discourse.pi-hole.net/u/rsantag)\
**Post date:** [May 5, 2021, 7:00pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/1 "2021-05-05T19:00:45Z")

</div>

I opened a thread several weeks ago about the dreaded "getOverTimeID is too large" warning message. Someone mentioned resolving this by running fake-hwclock, which my Debian 9 install on my PogoPlug didn't include by default, so I installed that. It definitely fixes the issue of the box starting up with time set to zero - 1/1/1970 @ 00:00.

But I booted up a PogoPlug running pihole today that hadn't been booted in a couple of weeks, so fake-hwclock set the date at startup to about 2 weeks ago, and that was enough to cause pihole to complain with the getOverTimeID is too large warning.

Does anybody know what the threshold is for it to consider the time delta "too large"? I'm trying to find a permanent fix to this error. I did create a perl script that can be inserted in the pihole startup script that will wait for NTP to set the time before starting pihole-FTL, but pihole software updates can overlay that local mod, so it's not really ideal.

Suggestions? I'm curious what the significance of this warning message is anyway, i.e. why does it care?

---

<div class="post-metadata">

**Author:** ![Bucking\_Horn](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/bucking_horn/32/18719_2.png) [@Bucking\_Horn](https://discourse.pi-hole.net/u/Bucking_Horn)\
**Post date:** [May 5, 2021, 7:44pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/2 "2021-05-05T19:44:25Z")

</div>

This is one of the instances that can only be solved reliably by hardware.

You could consider acquiring an RTC, if that would be compatible with your device. However, I am unfamiliar with any PogoPlug devices, so I wouldn't know whether yours already contains a battery-backuped RTC or not (in which case _`fake-hwclock`_ would make little sense).

_`fake-hwclock`_ does two things:  
It reads the current time from the system clock and writes it to a file on certain occasions (by default, that's every hour and on controlled shutdowns), and it reads the time from that same file at system startup and sets your system clock accordingly.

Later during system operation, a time-sync service (i.e. another piece of s/w entirely) may then acquire actual time information via an NTP server and update your system clock, provided your device has internet connectivity.

> [@rsantag](#):
>
> But I booted up a PogoPlug running pihole today that hadn't been booted in a couple of weeks, so fake-hwclock set the date at startup to about 2 weeks ago

While _`fake-hwclock`_ is certainly better as having no time reference at all, it still would do little more than assure that your time always moves forward as expected (as long as NTP services are accessible).

Specifically, your time **will** always be off after prolongued periods of shutdown.  
Depending on your system configuration, your system's time-sync service may then refuse to adjust the system clock from NTP information if it considers the gap to be too large.  
In such a case, you would have to manuallly adjust time information on your device.

> [@rsantag](#):
>
> m curious what the significance of this warning message is anyway, i.e. why does it care?

Apart from Pi-hole's UI not being able to produce accurate results, a whole bunch of cryptographic algorithms rely on a common time-frame on all involved peers.  
For example, if you were to use DNSSEC, record signature validation would fail, and thus DNS itself, resulting in Pi-hole's refusal to resolve any domain, which in turn would look like an internet outage for all clients.

So if you would want to use DNSSEC, and your PogoPlug does not feature an RTC already, I'd recommend to check whether your PogoPlug would support to connect a battery-backuped RTC module and buy one of those, e.g. a DS3231 isn't very expensive.

---

<div class="post-metadata">

**Author:** ![DanSchaper](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/danschaper/32/91_2.png) [@DanSchaper](https://discourse.pi-hole.net/u/DanSchaper)\
**Post date:** [May 5, 2021, 7:46pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/3 "2021-05-05T19:46:10Z")

</div>

Are your NTP servers IP addresses or are you using FQDNs?

If they are IP addresses and the issue is merely that `pihole-FTL` is starting before NTP has time to correct, then you can use the delay startup setting for `FTL`.

That does require that the NTP service is able to update the clock accurately. If the delta is too large then NTP will not update without manual intervention.

---

<div class="post-metadata">

**Author:** ![rsantag](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/r/5f8ce5/32.png) [@rsantag](https://discourse.pi-hole.net/u/rsantag)\
**Post date:** [May 5, 2021, 8:07pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/4 "2021-05-05T20:07:57Z")

</div>

Yeah, the problem is pihole-FTL starts a few seconds before NTP has set the system time. I did try the DELAY\_STARTUP option, but that didn't work. Here are my log entries showing it did delay starting the resolver until NTP had set the time properly

[1969-12-31 18:01:01.477 1642M] Sleeping for 3 seconds as requested by configuration ..  
[2021-03-27 06:24:06.702 1642M] Done sleeping, continuing startup of resolver...

but I still got the errors. DL6ER said it was because most of the startup is done before the delay is processed. There was a suggestion of moving the delay earlier, immediately after processing options file, but I don't think it went any further than just a suggestion. See this thread: [getOverTimeID is too large warning - #8 by jfb](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/45760/8)

---

<div class="post-metadata">

**Author:** ![DanSchaper](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/danschaper/32/91_2.png) [@DanSchaper](https://discourse.pi-hole.net/u/DanSchaper)\
**Post date:** [May 5, 2021, 8:25pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/5 "2021-05-05T20:25:04Z")

</div>

How long a delay did you try? Just the 3 seconds? Does 10 or more work?

---

<div class="post-metadata">

**Author:** ![rsantag](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/r/5f8ce5/32.png) [@rsantag](https://discourse.pi-hole.net/u/rsantag)\
**Post date:** [May 5, 2021, 8:30pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/6 "2021-05-05T20:30:07Z")

</div>

Thanks for your reply, Bucking\_Horn. PogoPlug is a small ARM architecture device with no RTC, and no way to add one that I know of. These devices were originally intended for cloud backup, etc, but the company has been out of business for I'm guessing about 5 years, so some people (like me) are repurposing them for other things.

I'm not having any problem with NTP setting the time - even when the device initially starts up with time set to 1/1/1970. The only problem is when pihole-FTL starts up before NTP has had a chance to finish setting the time. The DELAY\_STARTUP would do just what I need if it waited until AFTER the delay to set the start\_time variable it's using to determine that "getOverTimeID is too large".

---

<div class="post-metadata">

**Author:** ![rsantag](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/r/5f8ce5/32.png) [@rsantag](https://discourse.pi-hole.net/u/rsantag)\
**Post date:** [May 5, 2021, 8:35pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/7 "2021-05-05T20:35:14Z")

</div>

I didn't try a longer delay time since it looked like the system time was set properly at the end of the 3-second delay. I assumed a longer delay wouldn't make a difference since DL6ER said the delay didn't work because of where it was being processed in the pi-hole code.

---

<div class="post-metadata">

**Author:** ![rsantag](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/r/5f8ce5/32.png) [@rsantag](https://discourse.pi-hole.net/u/rsantag)\
**Post date:** [May 5, 2021, 8:43pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/8 "2021-05-05T20:43:14Z")

</div>

> [@Bucking\_Horn](#):
>
> Apart from Pi-hole's UI not being able to produce accurate results, a whole bunch of cryptographic algorithms rely on a common time-frame on all involved peers.  
> For example, if you were to use DNSSEC, record signature validation would fail, and thus DNS itself, resulting in Pi-hole's refusal to resolve any domain, which in turn would look like an internet outage for all clients.

Wouldn't this only be a problem if NTP **never** updated the system time? i.e. shouldn't DNSSEC be fine as long as system time is accurate when the DNS request is made? Or Is it just a warning that the time _ **may** _ be off, based on when it thinks pihole-FTL started up?

---

<div class="post-metadata">

**Author:** ![Bucking\_Horn](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/bucking_horn/32/18719_2.png) [@Bucking\_Horn](https://discourse.pi-hole.net/u/Bucking_Horn)\
**Post date:** [May 5, 2021, 8:54pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/9 "2021-05-05T20:54:30Z")

</div>

I was answering your inquiry about the significance of the message you observed.  
You are right in assuming that actual repercussions may trigger only on certain conditions (though it is irrelevant by what means your system time has been set - if it's off, then certain methods may fail as explained).

> [@rsantag](#):
>
> I'm not having any problem with NTP setting the time - even when the device initially starts up with time set to 1/1/1970. The only problem is when pihole-FTL starts up before NTP has had a chance to finish setting the time.

If acquiring the answer through a public NTP server takes too much time., you may consider using your router as an NTP source, provided your router supports it (and ideally has RTC backup 😉 ).

---

<div class="post-metadata">

**Author:** ![DanSchaper](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/danschaper/32/91_2.png) [@DanSchaper](https://discourse.pi-hole.net/u/DanSchaper)\
**Post date:** [May 5, 2021, 8:55pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/10 "2021-05-05T20:55:51Z")

</div>

DELAY\_STARTUP was originally intended for cases where the network interface was not up before FTL started.

We had the discussion of moving the delay to stop everything until the delay times out but I feel that is inefficient. I think that we should be doing housekeeping during the delay and loading up the database in the background.

I think this case here is a far edge case that shouldn't cause us to change the whole behavior of the delay. It's nice to be able to use old hardware and I used to have a few Pogos running linux chrooted but they shouldn't be running a vital network service like DNS if they aren't a reliable platform.

---

<div class="post-metadata">

**Author:** ![rsantag](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/r/5f8ce5/32.png) [@rsantag](https://discourse.pi-hole.net/u/rsantag)\
**Post date:** [May 5, 2021, 9:33pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/11 "2021-05-05T21:33:04Z")

</div>

> [@DanSchaper](#):
>
> It's nice to be able to use old hardware and I used to have a few Pogos running linux chrooted but they shouldn't be running a vital network service like DNS if they aren't a reliable platform.

Wouldn't this be an issue for any device that doesn't have a hardware clock, including the Raspberry Pi? Wondering why more people running pi-hole on their Raspberry Pi aren't reporting this.

---

<div class="post-metadata">

**Author:** ![DanSchaper](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/danschaper/32/91_2.png) [@DanSchaper](https://discourse.pi-hole.net/u/DanSchaper)\
**Post date:** [May 5, 2021, 9:34pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/12 "2021-05-05T21:34:01Z")

</div>

You can add an RTC to a RPi for $5USD.

And most of our users wouldn't have a DNS server that was off for a couple of weeks at a time.

---

<div class="post-metadata">

**Author:** ![rsantag](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/r/5f8ce5/32.png) [@rsantag](https://discourse.pi-hole.net/u/rsantag)\
**Post date:** [May 5, 2021, 9:39pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/13 "2021-05-05T21:39:29Z")

</div>

> [@DanSchaper](#):
>
> And most of our users wouldn't have a DNS server that was off for a couple of weeks at a time.

Good point... 😉

---

<div class="post-metadata">

**Author:** ![rsantag](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/r/5f8ce5/32.png) [@rsantag](https://discourse.pi-hole.net/u/rsantag)\
**Post date:** [May 7, 2021, 1:55pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/14 "2021-05-07T13:55:22Z")

</div>

So, I took a look at the code to understand the reason for the warning message, and I see it is in overTime.c, when OverTimeID is \> the number of OVERTIME\_SLOTS - 1. Can anyone explain what overTime.c is used for? Does this have something to do with loading the database, or possibly related to the graphs and stats on the dashboard?

Inquiring minds want to know.... ;-D

---

<div class="post-metadata">

**Author:** ![DL6ER](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/dl6er/32/281_2.png) [@DL6ER](https://discourse.pi-hole.net/u/DL6ER)\
**Post date:** [May 9, 2021, 6:26am UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/15 "2021-05-09T06:26:39Z")

</div>

> [@rsantag](#):
>
> I did try the DELAY\_STARTUP option, but that didn't work.

> [@rsantag](#):
>
> There was a suggestion of moving the delay earlier, immediately after processing options file, but I don't think it went any further than just a suggestion.

See [Delay earlier if requested by DL6ER · Pull Request #1090 · pi-hole/FTL · GitHub](https://github.com/pi-hole/FTL/pull/1090) still awaiting final review + approve or dismissal.

* * *

> [@rsantag](#):
>
> possibly related to the graphs and stats on the dashboard

Yes, this. FTL aims at both efficiency and as small as possible memory footprint. Hence, the overTime slots are static and relative to the current system time. Each slot corresponds to one bar in this graph:

 ![Screenshot from 2021-05-09 08-22-37](https://discourse.pi-hole.net/uploads/default/original/3X/0/5/05214c15d9a7e2b7cd6be338b2fec57719e43eef.png)

> [@rsantag](#):
>
> Inquiring minds want to know.... ;-D

When loading queries from the database, we bin them into these slots. The last slot is "right now", the first one is "24 hours ago". So if your system clock says "last week" and we import a query that is today (or, in fact, any more recent than two weeks ago), it'd have to be binned into a slot that is in the future (remember that our system clock tells us that it is still two weeks ago). However, FTL knows that a query in the future is not a good idea so it complains. We should improve this warning message. Maybe ensure it is only printed once.

This all has no effect on how your DNS server works, it just means that some of the statistics computed from the imported queries will be incorrect.

---

<div class="post-metadata">

**Author:** ![DL6ER](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/dl6er/32/281_2.png) [@DL6ER](https://discourse.pi-hole.net/u/DL6ER)\
**Post date:** [May 9, 2021, 6:44am UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/16 "2021-05-09T06:44:59Z")

</div>

@rsantag Would you mind trying

```auto
pihole checkout ftl tweak/incorrect_hwclock_warning

```

and check if this gives you a better (as in more understandable) warning message about the queries in the future? There are plans to go through all warnings and improve their usefulness for non-developers but this hasn't started, yet.

---

<div class="post-metadata">

**Author:** ![rsantag](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/r/5f8ce5/32.png) [@rsantag](https://discourse.pi-hole.net/u/rsantag)\
**Post date:** [May 9, 2021, 12:13pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/17 "2021-05-09T12:13:16Z")

</div>

> [@DL6ER](#):
>
> `pihole checkout ftl tweak/incorrect_hwclock_warning`

Thanks very much for the explanation. That helps a lot.

I'll give the tweak a try and let you know.

---

<div class="post-metadata">

**Author:** ![rsantag](https://discourse-cdn.pi-hole.net/letter_avatar_proxy/v4/letter/r/5f8ce5/32.png) [@rsantag](https://discourse.pi-hole.net/u/rsantag)\
**Post date:** [May 9, 2021, 3:44pm UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/18 "2021-05-09T15:44:48Z")

</div>

> [@DL6ER](#):
>
> @rsantag Would you mind trying
> 
> ```auto
> pihole checkout ftl tweak/incorrect_hwclock_warning
> 
> ```
> 
> and check if this gives you a better (as in more understandable) warning message about the queries in the future? There are plans to go through all warnings and improve their usefulness for non-developers but this hasn't started, yet.

Yes, definitely clearer description

> [2021-05-09 10:36:59.030 1653/T1657] Compiled 0 whitelist and 0 blacklist regex filters for 0 clients in 6.3 msec  
> [2021-05-09 10:38:06.613 1653M] WARN: Found database entries in the future (1620574500, last: 0). Your over-time statistics may be incorrect  
> [2021-05-09 10:38:06.647 1653M] New upstream server: 192.168.1.254:53 (0/1024)  
> [2021-05-09 10:38:54.364 1653M] New upstream server: 8.8.4.4:53 (1/1024)

I also noticed I only get the message once rather than for every query. That will also make a huge difference, so that pihole-FTL.log doesn't get huge (especially if you don't have logrotate installed)

---

<div class="post-metadata">

**Author:** ![DL6ER](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/dl6er/32/281_2.png) [@DL6ER](https://discourse.pi-hole.net/u/DL6ER)\
**Post date:** [May 11, 2021, 9:15am UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/19 "2021-05-11T09:15:48Z")

</div>

> <https://github.com/pi-hole/FTL/pull/1136>
>
> \*\*By submitting this pull request, I confirm the following:\*\*
> 
> \- \[X\] I have re…ad and understood the \[contributors guide\](https://github.com/pi-hole/pi-hole/blob/master/CONTRIBUTING.md).
> \- \[X\] I have checked that \[another pull request\](https://github.com/pi-hole/FTL/pulls) for this purpose does not exist.
> \- \[X\] I have considered, and confirmed that this submission will be valuable to others.
> \- \[X\] I accept that this submission may not be used, and the pull request closed at the will of the maintainer.
> \- \[X\] I give this submission freely, and claim no ownership to its content.
> 
> \*\*How familiar are you with the codebase?:\*\* 
> 
> \## 10
> 
> \---
> 
> Improve warning message when importing queries from the future (a typical problem of systems starting at 1970-01-01 on every boot). Also, show the warning only once instead of flooding the log file.

---

<div class="post-metadata">

**Author:** ![system](https://discourse.pi-hole.net/uploads/default/original/3X/7/c/7c8792f649eeb921c5d2b4c41564ffa873d9a2b8.png) [@system](https://discourse.pi-hole.net/u/system)\
**Post date:** [May 18, 2021, 9:15am UTC](https://discourse.pi-hole.net/t/getovertimeid-is-too-large-warning/46824/20 "2021-05-18T09:15:54Z")

</div>

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.