Pi suddenly stopped working

akash · June 4, 2019, 5:54pm

One fine day, my PiHole enabled RPi suddenly stopped working.

Here are the details

PiHole on 192.168.0.200
It works as a DHCP server giving out IP addresses to other devices on the network
My router's DLink DIR 825AC is configured to "DHCP Relay" and the ip address matches that of the RPi
My router's only DNS server is 192.168.0.200

I didn't change any settings - and it has been working fine (just like this) for a long time.

But suddenly, I couldn't access anything on the Internet. The only fix was to change my routers DNS to 1.1.1.1 (i.e. move away from using the RPi as a DNS lookup server)

I tried to generate a debug token. But that wouldn't upload. So here it is --> https://pastebin.com/xJ6KsTY4

I also tried to update to the latest, but that didn't work either:

  [✓] Detected ARM-hf architecture (armv6 or lower) Using ARM binary
  [i] Checking for existing FTL binary...
  [✗] Downloading and Installing FTL
  Error: Unable to get latest release location from GitHub
  [✗] FTL Engine not installed

  Unable to complete update, please contact Pi-hole Support

I'd really appreciate some help.

With this exception, the software has been excellent and very very stable. I'm full of appreciation for all the devs behind this. Thank you.

jfb · June 4, 2019, 6:42pm

You have DNSSEC enabled and some of your domain queries are coming back as BOGUS, which may indicate a problem with DNSSEC authentication.

Check that the time on your Pi is correct for your location: date

If not correct, you can reset your time with this command: timedatectl set-time '2015-11-20 16:14:50'

where you would substitute the correct date and time in that format

If you find that the time is off and you reset it, then restart FTL with sudo service pihole-FTL restart

akash · June 5, 2019, 11:52am

Yes, the time is off.

It is being measured in UTC, and it is off by around 3.5hrs.

I tried to update it, but there is an error:

rpi@raspberrypi:~ $ timedatectl set-time '2019-06-05 11:46:20'
Failed to set time: Automatic time synchronization is enabled

Anything else that I could try?

Also, a couple of questions

Why doesn't just resetting the pi solve the problem?
Is this also the reason that pihole -up is not working?
Is this the reason why I'm getting an error uploading a log?
Why might have this happened? I didn't change a thing

Thanks!

jfb · June 5, 2019, 1:36pm

http://www.itsprings.co.in/linux/Failed-to-set-timeAutomatic-time-synchronization-i

The Pi does not have a built-in clock, so it depends on external time servers for accurate time.

Probably yes, with DNSSEC enabled. If the time is off, the authentication mechanism won't work and the DNS replies can be rejected as BOGUS.

If your Pi lost power suddenly, this can cause time problems.

akash · June 8, 2019, 6:57am

Fixed, thank you!

The time out-of-sync was the issue
To fix it I had to disable auto setting of date time with timedatectl set-ntp false and then manually set the UTC time
After this I restarted the sync with timedatectl set-ntp yes - but I could never figure out why the NTP sync wasn't working. The timedatectl said NTP-synchronized: No
Anyway, it wasn't perfect but it allowed pihole -up to function. And after that, everything seems fine. And NTP is synchronized.

deHakkelaar · June 8, 2019, 11:30am

Probably chicken egg dilemma.
The ntpd daemon is configured with domain names:

pi@noads:~ $ cat /etc/ntp.conf
[..]
pool 0.debian.pool.ntp.org iburst
pool 1.debian.pool.ntp.org iburst
pool 2.debian.pool.ntp.org iburst
pool 3.debian.pool.ntp.org iburst
[..]

And pihole-FTL is only able to resolve them to IP, via DNSSEC, if time set by ntpd is correct.
A possible solution would have been to configure ntpd with IP addresses instead of those pools making ntpd independent of DNS resolution.

DanSchaper · June 8, 2019, 6:43pm

github.com/pi-hole/pi-hole

Bogus Responses with DNSSEC enabled

opened 10:06AM - 02 Mar 19 UTC

closed 12:19AM - 23 Aug 20 UTC

DenyDarko

triage: Issue

**In raising this issue, I confirm the following:** `{please fill the checkboxes…, e.g: [X]}` - [X] I have read and understood the [contributors guide](https://github.com/pi-hole/pi-hole/blob/master/CONTRIBUTING.md). - [X] The issue I am reporting can be *replicated*. - [X] The issue I am reporting isn't a duplicate (see [FAQs](https://github.com/pi-hole/pi-hole/wiki/FAQs), [closed issues](https://github.com/pi-hole/pi-hole/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), and [open issues](https://github.com/pi-hole/pi-hole/issues)). **How familiar are you with the the source code relevant to this issue?:** `{3}` --- **Expected behaviour:** `{Functional DNS Resolving with DNSSEC enabled}` **Actual behaviour:** `{Bogus responses, I've tested on DNS Watch, Cloudflare and Quad9}` **Steps to reproduce:** `{It's appearing out of the blue at random times}` **Debug token provided by [uploading `pihole -d` log](https://discourse.pi-hole.net/t/the-pihole-command-with-examples/738#debug):** `{https://tricorder.pi-hole.net/nm8biwv0m1!}` **Troubleshooting undertaken, and/or other relevant information:** `{Disabling DNSSEC and re-enabling it after a whilefixes it. I've seen some Stalled Closed issues, no actual fix ever proposed though :( }` ![12121](https://user-images.githubusercontent.com/3320229/53680378-6e764c00-3ce3-11e9-8d33-54751a9ba075.png)

deHakkelaar · June 8, 2019, 9:28pm

pi@noads:~ $ cat /etc/ntp.conf
[..]
pool 0.debian.pool.ntp.org iburst
pool 1.debian.pool.ntp.org iburst
pool 2.debian.pool.ntp.org iburst
pool 3.debian.pool.ntp.org iburst

pi@noads:~ $ host 0.debian.pool.ntp.org
0.debian.pool.ntp.org has address 109.236.82.30
0.debian.pool.ntp.org has address 95.211.212.5
0.debian.pool.ntp.org has address 80.100.130.235
0.debian.pool.ntp.org has address 149.210.142.45

pi@noads:~ $ sudo ntpdate -u 109.236.82.30
 8 Jun 23:19:20 ntpdate[9447]: adjust time server 109.236.82.30 offset -0.001288 sec

No DNS needed to sync and plenty of ntp servers to choose from if query for the other default configured ntp pools

DanSchaper · June 8, 2019, 10:44pm

host uses DNS to find the IP address.

deHakkelaar · June 8, 2019, 11:38pm

Thats not what I meant.
Use host to lookup as many IP addresses as you desire and put them in the ntp.conf file on a server line (hash out the default pool ones oc).

EDIT: eg.

server 109.236.82.30
server 95.211.212.5
server 80.100.130.235
server 149.210.142.45

deHakkelaar · June 24, 2019, 7:09pm

Sorry, I noticed current distro's with systemd use below one for time sync instead of old fashion ntpd:

$ man systemd-timesyncd.service
[..]
DESCRIPTION
       systemd-timesyncd is a system service that may be used to
       synchronize the local system clock with a remote Network Time
       Protocol server. It also saves the local time to disk every time
       the clock has been synchronized and uses this to possibly
       advance the system realtime clock on subsequent reboots to
       ensure it monotonically advances even if the system lacks a
       battery-buffered RTC chip.

If want to know if systemd-timesyncd is active:

$ sudo service systemd-timesyncd status
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled
  Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: active (running) since Mon 2019-06-03 07:50:04 CEST; 3 weeks 0

If want to check if its configured to sync:

$ timedatectl
[..]
 Network time on: yes
NTP synchronized: yes
 RTC in local TZ: no

NTP servers need to be configured here on the "NTP=" line:

$ cat /etc/systemd/timesyncd.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.

[Time]
#NTP=
#FallbackNTP=0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org

Eg.

NTP=109.236.82.30 95.211.212.5 80.100.130.235 149.210.142.45

EDIT: Ow and for the record, if you still have ntpd installed (apt policy ntp), below one is holding systemd-timesyncd back:

pi@noads:~ $ cat /lib/systemd/system/systemd-timesyncd.service.d/disable-with-time-daemon.conf
[Unit]
# don't run timesyncd if we have another NTP daemon installed
ConditionFileIsExecutable=!/usr/sbin/ntpd
ConditionFileIsExecutable=!/usr/sbin/openntpd
ConditionFileIsExecutable=!/usr/sbin/chronyd
ConditionFileIsExecutable=!/usr/sbin/VBoxService

To switch:

sudo service ntp stop

sudo apt remove ntp

sudo service systemd-timesyncd restart

sudo service systemd-timesyncd status

timedatectl

jpgpi250 · July 3, 2019, 6:38am

Thank you for this, I didn't know this service existed, however I do NOT agree with your statement old fashion 'ntpd'. The man page of systemd-timesyncd itself says: This minimalistic service will set the system clock, which might be sufficient for most users, the more robust implementation of the NTP deamon might still prove to be of value, and makes life consistent for people managing more than a single Linux system, systems that don't always have the systemd-timesyncd service.
The systemd-timesyncd service is an SNTP client, only implementing the client part of NTP. If you want to use your pihole as the source to go for time (NTP server), this will NOT be possible.
The systemd-timesyncd service, according to the manual, writes to tour SD card, every time the clock has been synchronized, NOT one of my favorites to read, I rather spend the 8$ for a new SD card (mentioned in some other topic) on something else, but that's a personal point of view.

Thank you again for pointing out the existence of this, running out of the box, but unconfigured service

If somebody decides to go for the NTP daemon instead of the systemd-timesyncd service, here is how to clean up, avoiding the service to run but do nothing (as you explained earlier)

sudo service systemd-timesyncd stop
sudo systemctl disable systemd-timesyncd

deHakkelaar · July 3, 2019, 10:53am

Yeah I do not agree with allot of the SBC distros getting bloated with software you might not want.
I would prefer a really minimalistic install and if I need time sync or some network manager/mangler, I'll choose to install which one.
But they try to make it a bit non Linux user friendly which is fine with me but still I would prefer a really minimalistic install.

You dont have to disable the systemd-timesyncd daemon.
It wont start if any of below NTP daemons is installed:

pi@noads:~ $ cat /lib/systemd/system/systemd-timesyncd.service.d/disable-with-time-daemon.conf
[Unit]
# don't run timesyncd if we have another NTP daemon installed
ConditionFileIsExecutable=!/usr/sbin/ntpd
ConditionFileIsExecutable=!/usr/sbin/openntpd
ConditionFileIsExecutable=!/usr/sbin/chronyd
ConditionFileIsExecutable=!/usr/sbin/VBoxService

system · July 24, 2019, 10:53am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.