DNS service is not running

zodiac1978 · October 2, 2023, 9:46pm

My pi-hole was running fine for years on my Raspi 2. Now it stopped working and I have no idea why. Maybe the SD card is broken?

The "DNS service is not running" is showing in the web GUI and the adblocking does not work. I looked for the error, but there are sooo many threads here, on GitHub and Reddit and nothing works.

On the settings page I see this error message:
There was a problem applying your settings.
Debugging information:
PHP error (2): fsockopen(): unable to connect to 127.0.0.1:4711 (Connection refused) in /var/www/html/admin/scripts/pi-hole/php/FTL.php:47

"pihole restartdns" returns
[✗] Failed to retrieve unit state: Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Failed to restart pihole-FTL.service: Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
See system logs and 'systemctl status pihole-FTL.service' for details.

Using this prompt, I get:
Failed to get properties: Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)

Updating the pi-hole with pihole -up does not work, repairing does not work ...

*** [ DIAGNOSING ]: Operating system
[i] dig return code: 10
[i] dig response: dig: couldn't get address for 'ns1.pi-hole.net': failure
[✗] Distro: Raspbian
[✗] Error: Raspbian is not a supported distro (Prerequisites - Pi-hole documentation)

It looks like my distro is outdated, but the DNS is not working, so I can't update the pi itself.

*** [ DIAGNOSING ]: Name resolution (IPv4) using a random blocked domain and a known ad-serving domain
[✗] Failed to resolve www.nubipgdacht.com on lo (127.0.0.1)
[✗] Failed to resolve www.nubipgdacht.com on eth0 (192.168.178.57)
[✓] doubleclick.com is 142.250.181.206 via a remote, public DNS server (8.8.8.8)

*** [ DIAGNOSING ]: Name resolution (IPv6) using a random blocked domain and a known ad-serving domain
[✗] Failed to resolve livescience.us.intellitxt.com on lo (::1)
[✗] Failed to resolve livescience.us.intellitxt.com on eth0 (2a01:c22:8c4e:2f00:d4f0:d656:6118:481d)
[✗] Failed to resolve livescience.us.intellitxt.com on eth0 (fe80::86a5:f85f:d11d:7e3d)
[✓] doubleclick.com is 2a00:1450:4005:802::200e via a remote, public DNS server (2001:4860:4860::8888)

No I see there seems to be a crash:
[2023-10-02 21:57:13.121 1118M] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2023-10-02 21:57:13.121 1118M] ----------------------------> FTL crashed! <----------------------------
[2023-10-02 21:57:13.122 1118M] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2023-10-02 21:57:13.122 1118M] Please report a bug at GitHub · Where software is built
[2023-10-02 21:57:13.123 1118M] and include in your report already the following details:
[2023-10-02 21:57:13.123 1118M] FTL has been running for 1692812 seconds
[2023-10-02 21:57:13.123 1118M] FTL branch: master
[2023-10-02 21:57:13.123 1118M] FTL version: v5.20
[2023-10-02 21:57:13.124 1118M] FTL commit: cba74934
[2023-10-02 21:57:13.124 1118M] FTL date: 2022-12-21 21:06:00 +0000
[2023-10-02 21:57:13.125 1118M] FTL user: started as pihole, ended as pihole
[2023-10-02 21:57:13.126 1118M] Compiled for armv6hf (compiled on CI) using arm-linux-gnueabihf-gcc (GCC) 10.3.0
[2023-10-02 21:57:13.126 1118M] Process details: MID: 1118
[2023-10-02 21:57:13.126 1118M] PID: 1118
[2023-10-02 21:57:13.127 1118M] TID: 1118
[2023-10-02 21:57:13.127 1118M] Name: pihole-FTL
[2023-10-02 21:57:13.129 1118M] Received signal: Speicherzugriffsfehler
[2023-10-02 21:57:13.130 1118M] at address: 0xe1d430b2
[2023-10-02 21:57:13.130 1118M] with code: SEGV_MAPERR (Address not mapped to object)

Saving debug.log token not possible either:

curl failed, contact Pi-hole support for assistance.
Error message: curl: (6) Could not resolve host: tricorder.pi-hole.net

The /etc/pihole/pihole-FTL.db seems to be corrupt, so I renamed it. But t doesn't created a new DB.

I'm out of ideas ... what is wrong with my pi-hole? How do I repair it? There is nothing custom here, just pi-hole and and a Raspi 2 which was running fine for years ...

zodiac1978 · October 2, 2023, 9:56pm

After restarting the DNS is running for some very short time (1-2 minutes max) and then it shows "DNS service not running" again ...

rdwebdesign · October 2, 2023, 10:45pm

This seems an issue with your Raspberry Pi DNS.

This will temporarily reset the nameserver on the Pi to bypass Pi-Hole DNS.

sudo nano /etc/resolv.conf

Edit the nameserver line to nameserver 9.9.9.9 or your preferred third party DNS service, save and exit

Run

pihole -d

and upload the debug log.

Bucking_Horn · October 3, 2023, 9:52am

You did encounter a veritable crash.

While this could be caused by a defective sd card, you'd probably had seen previous indications for that.

Could you please provide the full crash log, by following the instructions from the log file?
They would look similar to:

[2023-10-02 21:57:13.130 1118M] Please also include some lines from above the !!!!!!!!! header.
[2023-10-02 21:57:13.130 1118M]  Thank you for helping us to improve our FTL engine!
[2023-10-02 21:57:13.130 1118M]  FTL terminated!

Also, let's have a look at your sd card.

Run from your Pi-hole machine, what's the output of

df -h

zodiac1978 · October 3, 2023, 11:00am

Thanks for helping!

First the output of df -h

Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root        15G     11G  3,4G   76% /
devtmpfs        183M       0  183M    0% /dev
tmpfs           216M    4,4M  211M    3% /dev/shm
tmpfs           216M    3,2M  212M    2% /run
tmpfs           5,0M    4,0K  5,0M    1% /run/lock
tmpfs           216M       0  216M    0% /sys/fs/cgroup
/dev/mmcblk0p1   44M     25M   19M   58% /boot
tmpfs            43M    4,0K   43M    1% /run/user/1000

This is my debug token, after I set the nameserver from 127.0.0.1 to 9.9.9.9:
https://tricorder.pi-hole.net/wWSDfWvL/

Not sure if the DNS was still okay, so I made another debug:
https://tricorder.pi-hole.net/gmws3vIn/

Web GUI is showing: DNS service not running

Looks like the db is locked and corrupted again.

Could you please provide the full crash log, by following the instructions from the log file?

Not sure what you mean by this. Which instructions?
This? "Please report a bug at GitHub · Where software is built"

Should I move this to GitHub instead or additionally?

Thanks again!

All the best
Torsten

Bucking_Horn · October 3, 2023, 11:08am

The ones I've quoted above, as appearing in your logs.
They would mark the end of the crash information.
We'd appreciate if you could extract that entire section, plus a few lines from above the crash report.

Let's have a look at your sd card's file system health:

sudo e2fsck -fvn /dev/mmcblk0p1

zodiac1978 · October 3, 2023, 11:23am

I thought this would be in the debug log.

Here is the part above the !!!!!!!!! line:

-----tail of FTL.log------
   [2023-10-03 12:39:13.208 1061M] getNameFromIP("192.168.178.35") - SQL error prepare: database is locked
   [2023-10-03 12:39:14.210 1061M] SQLite3 message: database is locked in "SELECT interface FROM network JOIN network_addresses ON network_addresses.network_id = network.id WHERE network_addresses.ip = ? AND interface != 'N/A' AND interface IS NOT NULL;" (5)
   [2023-10-03 12:39:14.210 1061M] getIfaceFromIP("192.168.178.35") - SQL error prepare: database is locked
   [2023-10-03 12:39:15.219 1061M] SQLite3 message: database is locked in "SELECT hwaddr FROM network WHERE id = (SELECT network_id FROM network_addresses WHERE ip = ? GROUP BY ip HAVING max(lastSeen));" (5)
   [2023-10-03 12:39:15.220 1061M] getMACfromIP("192.168.178.35") - SQL error prepare: database is locked
   [2023-10-03 12:39:16.221 1061M] SQLite3 message: database is locked in "SELECT name FROM network_addresses WHERE name IS NOT NULL AND ip = ?;" (5)
   [2023-10-03 12:39:16.222 1061M] getNameFromIP("192.168.178.35") - SQL error prepare: database is locked
   [2023-10-03 12:39:17.224 1061M] SQLite3 message: database is locked in "SELECT interface FROM network JOIN network_addresses ON network_addresses.network_id = network.id WHERE network_addresses.ip = ? AND interface != 'N/A' AND interface IS NOT NULL;" (5)
   [2023-10-03 12:39:17.224 1061M] getIfaceFromIP("192.168.178.35") - SQL error prepare: database is locked
   [2023-10-03 12:39:23.298 1061/T1077] Notice: Database size is 624.00 MB, deleted 542950 rows
   [2023-10-03 12:39:24.239 1061/T1077] SQLite3 message: database corruption at line 85570 of [89c459e766] (11)
   [2023-10-03 12:39:24.240 1061/T1077] SQLite3 message: statement aborts at 13: [UPDATE network_addresses SET name = ?1, nameUpdated = (cast(strftime('%s', 'now') as int)) WHERE ip = ?2] database disk image is malformed (11)
   [2023-10-03 12:39:24.241 1061/T1077] update_netDB_name(192.168.178.80, "192-168-178-80.fritz.box"): Failed to step (error 11): database disk image is malformed
   [2023-10-03 12:39:24.241 1061/T1077] WARN: Database /etc/pihole/pihole-FTL.db is damaged and cannot be used.
   [2023-10-03 12:39:24.242 1061/T1077] Database error in ARP cache processing loop
   [2023-10-03 12:40:27.547 1061M] Resizing "FTL-dns-cache" from 4096 to (512 * 16) == 8192 (/dev/shm: 4.6MB used, 225.5MB total, FTL uses 4.6MB)
   [2023-10-03 12:40:27.549 1061M] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

And the output from sudo e2fsck -fvn /dev/mmcblk0p1

is here:

e2fsck 1.44.5 (15-Dec-2018)
Warnung! /dev/mmcblk0p1 ist eingehängt.
ext2fs_open2: Ungültige magische Zahl im Superblock
e2fsck: Superblock ungültig, Datensicherungs-Blöcke werden versucht ...
e2fsck: Ungültige magische Zahl im Superblock beim Versuch, /dev/mmcblk0p1 zu öffnen

Der Superblock ist unlesbar bzw. beschreibt kein gültiges ext2/ext3/ext4-
Dateisystem. Wenn das Gerät gültig ist und ein ext2/ext3/ext4-
Dateisystem (kein swap oder ufs usw.) enthält, dann ist der Superblock
beschädigt, und Sie könnten versuchen, e2fsck mit einem anderen Superblock
zu starten:
    e2fsck -b 8193 <Gerät>
 oder
    e2fsck -b 32768 <Gerät>

/dev/mmcblk0p1 hat ein vfat-Dateisystem mit Namen „boot“

I hope the German output is understandable ... looks like there is indeed something broken ...

Bucking_Horn · October 3, 2023, 11:57am

Kein Problem (es gäbe sogar eine deutschsprachige Kategorie hier im Forum).

The debug log only contains a short excerpt of /var/log/pihole/FTL.log, so it would be by chance if the full event would appear in it.

Thanks for those lines.
They show quite a few file system related issues, which may indeed indicate a worn sd card.

Regardless of further analysis, you should probably be prepared to do a clean install of Pi-hole, either after formatting the same card, or on a fresh one.

To that end, and before you try anything else: If you haven't done so already, you should download a teleporter Backup from your Pi-hole via Settings | Teleporter.

Note that there is a chance that gravity.db may also already be affected by file system corruption, so creating the backup now may fail.

And apologies - I've asked you to analyse the wrong block device.
For RPi OS, that would likely be mmcblk0p2.

After(!) you've done the backup, you could possibly try again with

sudo e2fsck -fvn /dev/mmcblk0p2

This will force (f) a verbose (v) dry run for potential file system repairs without actually applying them, by virtue of the n option.

zodiac1978 · October 3, 2023, 12:37pm

This is the output of tail /var/log/pihole/FTL.log:

[2023-10-03 12:40:27.553 1061M] FTL date: 2022-12-21 21:06:00 +0000
[2023-10-03 12:40:27.554 1061M] FTL user: started as pihole, ended as pihole
[2023-10-03 12:40:27.554 1061M] Compiled for armv6hf (compiled on CI) using arm-linux-gnueabihf-gcc (GCC) 10.3.0
[2023-10-03 12:40:27.556 1061M] Process details: MID: 1061
[2023-10-03 12:40:27.556 1061M]                  PID: 1061
[2023-10-03 12:40:27.556 1061M]                  TID: 1061
[2023-10-03 12:40:27.557 1061M]                  Name: pihole-FTL
[2023-10-03 12:40:27.560 1061M] Received signal: Speicherzugriffsfehler
[2023-10-03 12:40:27.560 1061M]      at address: 0xe5963004
[2023-10-03 12:40:27.561 1061M]      with code:  SEGV_MAPERR (Address not mapped to object)

Backup with Teleporter worked. But I think I only need the 5 whitelist entries if I start from scratch.

Now the output from sudo e2fsck -fvn /dev/mmcblk0p2

e2fsck 1.44.5 (15-Dec-2018)
Warnung! /dev/mmcblk0p2 ist eingehängt.
Warnung: Journal-Wiederherstellung wird übersprungen, da sich das Dateisystem
im Nur-Lesen-Modus befindet.
Durchgang 1: Inodes, Blöcke und Größen werden geprüft
dtime für gelöschten Inode 1979 ist Null.  Reparieren? nein

Inodes wurden gefunden, die Teil einer defekten verketteten Liste von
verwaisten Inodes waren.  Reparieren? nein

Inode 47766 war Teil der Liste verwaister Inodes.  IGNORIERT.
Inode 47771 war Teil der Liste verwaister Inodes.  IGNORIERT.
Inode 128871, i_Blocks ist 104, sollte 96 sein.  Reparieren? nein

Inode 134813, i_Blocks ist 224, sollte 168 sein.  Reparieren? nein

Inode 135477, i_Blocks ist 120, sollte 112 sein.  Reparieren? nein


Zusätzliche Läufe werden durchgeführt, um die von mehr als einem Inode
beanspruchten Blöcke zu klären ...
Durchgang 1B: Suche nach mehrfach beanspruchten Blöcken
Mehrfach beanspruchte(r) Block/Blöcke in Inode 641281: 2655511
Durchgang 1C: Verzeichnisse werden nach Inodes mit mehrfach belegten Blöcken durchsucht.
Durchgang 1D: Mehrfach belegte Blöcke werden abgeglichen.
(Es gibt 1 Inodes, die mehrfach belegte Blöcke enthalten.)

Datei /var/log/exim4/mainlog.1 (Inode #641281, Änderungszeit Tue Sep 12 23:48:55 2023) 
  hat 1 mehrfach belegte(n) Block/Blöcke, gemeinsam genutzt mit 0 Datei(en):
Mehrfach referenzierte Blöcke werden geklont? nein

Datei löschen? nein

Durchgang 2: Verzeichnisstruktur wird geprüft
Eintrag „.ICE-unix“ in /tmp (18) hat einen falschen Dateityp (war 2, sollte 1 sein).
Reparieren? nein

Eintrag „.Test-unix“ in /tmp (18) hat einen falschen Dateityp (war 2, sollte 1 sein).
Reparieren? nein

Eintrag „systemd-private-1ac8342e96af4fb0999c014087bbd2ce-systemd-timesyncd.service-opjjK7“ in /tmp (18) hat einen falschen Dateityp (war 2, sollte 1 sein).
Reparieren? nein

Eintrag „pulse-PKdhtXMmr18n“ in /tmp (18) hat gelöschten/unbenutzten Inode 128848.  Bereinigen? nein

Eintrag „.vncserver-license“ in /tmp (18) hat einen falschen Dateityp (war 2, sollte 1 sein).
Reparieren? nein

Eintrag „.vnc-vncservice“ in /tmp (18) hat einen falschen Dateityp (war 2, sollte 1 sein).
Reparieren? nein

Eintrag „.X0-lock“ in /tmp (18) hat einen falschen Dateityp (war 1, sollte 6 sein).
Reparieren? nein

Eintrag „ssh-KeHlsAOwBBLT“ in /tmp (18) hat einen falschen Dateityp (war 2, sollte 1 sein).
Reparieren? nein

Eintrag „ssh-K2TsDM3vhrVc“ in /tmp (18) hat einen falschen Dateityp (war 2, sollte 1 sein).
Reparieren? nein

Verzeichnis-Inode 48282, Block Nr.0, Offset 0: Verzeichnis defekt
Retten? nein

e2fsck: abgebrochen

rootfs: ********** WARNUNG: Noch Fehler im Dateisystem  **********

If I need st start from scratch, is my Raspi 2 still capable of this or do I need to update my hardware too?

Bucking_Horn · October 3, 2023, 12:49pm

I'd recomend to start afresh, especially if it just would be Pi-hole running on that RPi.

You may attempt to fix those file system issues, but I'd recommend to start from a fresh image straight away, preferably from a fresh sd card.
You may get away with formatting the current one if you do not have a spare one ready at the moment, but I wouldn't postpone getting a new one for too long.

And you can safely continue to use your Pi 2.
It is fully sufficient for running Pi-hole's DNS server in your typical home network. It may just be a bit slow in serving its web interface at times.

zodiac1978 · October 3, 2023, 1:02pm

Will do this.

Nevertheless, thank you for helping and providing such fast and knowledgeable replies!

system · October 24, 2023, 1:02pm

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.