Network table stays empty after restoring from a backup copy

Please follow the below template, it will help us to help you!

If you are Experiencing issues with a Pi-hole install that has non-standard elements (e.g you are using nginx instead of lighttpd, or there is some other aspect of your install that is customised) - please use the Community Help category.

Expected Behaviour:

Network list would fill up after flushing it

Raspberry OS Buster
Raspberry Pi4B+

I tried to change the port 5335 to 5353 because I thought I had mistyped it during install (pihole+unbound). Everything went awry from there. I was able to restore a full OS copy to the SSD from an SD card I had copied the SSD contents to a few days ago. DNS etc. works again, but the network list stays empty. There was also a message at some point about the pihole-FTL.db being locked, but that may have been caused by my making a new copy of the SSD using the Raspberry PI's SD card copier.
I also restored the pihole Teleporter profile(?) that I made yesterday.

Actual Behaviour:

Network table doesnt fill up.
output from command:
pi@RPI-HOLE:~ $ pihole -f
[i] Flushing /var/log/pihole.log ...Error: database is locked
[✓] Restarting DNS server
[✓] Flushed /var/log/pihole.log
[✓] Deleted queries from database
pi@RPI-HOLE:~ $

Debug Token:

https://tricorder.pi-hole.net/yodvl1r6i8

Are there any errors in

/var/log/pihole-FTL.log

Except of the empty network table, pihole is working as intended?

Hi,
Thanks for helping out. I don't care if deleting the log etc. history is the easiest solution. I do like to keep the gravity db and white- and blacklist.

There are errors in the log. THis is the last part with the errors. They appeared at an earlier time, too, but similar/same.
I suppose I need a new empty database.

[2020-07-04 11:42:11.403 10725] New upstream server: ::1 (0/1024)
[2020-07-04 11:42:11.536 10725] New upstream server: 127.0.0.1 (1/1024)
[2020-07-04 11:42:35.414 10725] Reloading DNS cache
[2020-07-04 11:42:35.415 10725] Blocking status is enabled
[2020-07-04 11:42:35.415 10725] SQLite3 message: file unlinked while open: /etc/pihole/gravity.db (28)
[2020-07-04 11:42:35.423 10725] INFO: No regex whitelist entries found
[2020-07-04 11:42:35.427 10725] Compiled 0 whitelist and 11 blacklist regex filters in 4.8 msec
[2020-07-04 11:43:00.007 10725] SQLite3 message: database corruption at line 65570 of [18db032d05] (11)
[2020-07-04 11:43:00.007 10725] SQLite3 message: statement aborts at 3: [SELECT MAX(ID) FROM queries] database disk image is malformed (11)
[2020-07-04 11:43:00.007 10725] Encountered step error in get_max_query_ID(): database disk image is malformed
[2020-07-04 11:43:00.007 10725] Encountered error while trying to close database: database is locked
[2020-07-04 11:43:00.007 10725] SQLite3 message: database corruption at line 65570 of [18db032d05] (11)
[2020-07-04 11:43:00.007 10725] SQLite3 message: statement aborts at 3: [INSERT INTO queries VALUES (NULL,?,?,?,?,?,?)] database disk image is malformed (11)
[2020-07-04 11:43:00.007 10725] Encountered error while trying to store queries in long-term database: database disk image is malformed
[2020-07-04 11:43:00.007 10725] dbquery("END TRANSACTION") called but database is not available!
[2020-07-04 11:43:00.007 10725] END TRANSACTION failed when trying to store queries to long-term database
[2020-07-04 11:43:00.008 10725] dbquery("BEGIN TRANSACTION IMMEDIATE") called but database is not available!
[2020-07-04 11:43:00.009 10725] ERROR: Storing devices in network table ("BEGIN TRANSACTION IMMEDIATE") failed
[2020-07-04 11:46:51.419 10725] Reloading DNS cache
[2020-07-04 11:46:51.419 10725] Blocking status is enabled
[2020-07-04 11:46:51.420 10725] SQLite3 message: file unlinked while open: /etc/pihole/gravity.db (28)
[2020-07-04 11:46:51.436 10725] INFO: No regex whitelist entries found
[2020-07-04 11:46:51.439 10725] Compiled 0 whitelist and 11 blacklist regex filters in 3.7 msec
[2020-07-04 11:57:45.375 10725] Resizing "/FTL-strings" from 4096 to 8192
[2020-07-04 12:08:33.527 10725] Resizing "/FTL-dns-cache" from 4096 to 8192
[2020-07-04 12:32:00.061 10725] Resizing "/FTL-strings" from 8192 to 12288
[2020-07-04 13:00:10.369 10725] Resizing "/FTL-dns-cache" from 8192 to 12288
[2020-07-04 13:08:37.009 10725] Resizing "/FTL-strings" from 12288 to 16384
pi@RPI-HOLE:~ $

Trying to change the port from 5335 to 5353 was a real dumb thing to do. I was misinformed and figured I had all the places where it appears covered. Wow, what a mess it caused. Lost internet, had to work a lot on the router to get it back. No peace to do it.
PiHole with unbound and PiVPN and Wireguard are all working OK again. Even the query log. But not the network list.

Oh and one really nasty thing was that the PI's clock was terribly off after restoring the profile (not sure exactly when). Raspberry PI OS doesnt' just set it from NTP. It needs to be close to correct time and date already. Doing that manually, once you notice the problem, is not trivial.

You can try to "repair" the database as described here:

Or you just move/delete it

sudo service pihole-FTL stop

sudo mv /etc/pihole/pihole-FTL.db /etc/pihole/pihole-FTL-old.db

sudo service pihole-FTL start

This will just delete your long-term database and network table, group management, adlists, etc are not affected because they are stored in /etc/pihole/gravity.db


Note: the Pihole documentation for setting up unbound has changed from recommending port 5353 to 5335 as the former could be used by an mdns client which might cause issues. If it was running well for you, there is no reason to change it.

I had 5335 and working fine. Then in many posts on the net I read 5353. Thinking wrongly that it was somehow essential or from a sense I want to get it right, you never know when this mistake will bite me .... I set out to change it ... and soon after I found out about the issue you mentioned ...

I did as outlined in the message you linked to. Not knowing at first how to get out of the sqlite mode caused some trouble. In the end I got it done. However in PiHole it made no difference. No long term history and it doesn't pick up device names, just IP's.

The end of the sql dump looks like this and it explains why that didn't restore anything.
...
INSERT INTO queries VALUES(1414492,1593353052,2,2,'normandy.cdn.mozilla.net','192.168.10.3','127.0.0.1');
/****** CORRUPTION ERROR *******/
CREATE TABLE ftl ( id INTEGER PRIMARY KEY NOT NULL, value BLOB NOT NULL );
INSERT INTO ftl VALUES(0,5);
INSERT INTO ftl VALUES(1,1593353033);
INSERT INTO ftl VALUES(2,1590728441);
CREATE TABLE counters ( id INTEGER PRIMARY KEY NOT NULL, value INTEGER NOT NULL );
INSERT INTO counters VALUES(0,1414476);
INSERT INTO counters VALUES(1,239472);
CREATE TABLE network_addresses ( network_id INTEGER NOT NULL, ip TEXT NOT NULL, lastSeen INTEGER NOT NULL DEFAULT (cast(strftime('%s', 'now') as int)), UNIQUE(network_id,ip), FOREIGN KEY(network_id) REFERENCES network(id));
CREATE TABLE IF NOT EXISTS "network" ( id INTEGER PRIMARY KEY NOT NULL, hwaddr TEXT UNIQUE NOT NULL, interface TEXT NOT NULL, name TEXT, firstSeen INTEGER NOT NULL, lastQuery INTEGER NOT NULL, numQueries INTEGER NOT NULL, macVendor TEXT);
DELETE FROM sqlite_sequence;
INSERT INTO sqlite_sequence VALUES('queries',1414476);
CREATE INDEX idx_queries_timestamps ON queries (timestamp);
ROLLBACK; -- due to errors

I will admit I have had to pull the power supply around the time of the attempt to change port. When I insert an SD Card reader in the USB port of the PI, it sometimes locks up totally. The simpler reader doesn't do it as often. In such case there is not much to do except pull the power. The SD card reader is how I make a backup copy of the SSD to SD card. The latest copy did save my ass ...

Maybe it's something else that prevents the PiHole from picking up the device names. Some local device names are recognised ok currently in a terminal session of my laptop. It could very well be that that is because of the Local DNS entries in PiHole I made for a few devices. Other devices still don't have names. They used to get name in the query log. Now they do not.

I had to make a lot of temporary changes to get internet back when the PiHole was in disarray. I am somehow well versed in DD-WRT on my R7800, but who knows I forgot something.
Conditional forwarding on my Pihole never worked nicely. It added the local domain to most queries in the log.
But I'll try that again, too.

I had PiHole GUI clean the network table. There is still a grave error. I think I need to remove the data base like you suggested and get a new one.

Error

e[K [✗] Failed to truncate network_addresses table
Database location: /etc/pihole/pihole-FTL.db
Output: Error: no such table: network_addresses

Allright! Deleting the /pihole-FTL.db database solved it. Now the network table picks up devices again. I have not yet seen names, just IP's.
A minute later it gives names for some devices. Even for one on VPN as mi-a3.pivpn. Just like it used to do.

Thanks a lot. I think this is solved. The rest, if there are hostnames missing, is other LAN and router stuff.

Glad it worked. Please mark as "solved".

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.