If I restart the pihole-FTL service, it starts working again (for 1-2 hours), but the stats and graphs on the AdminUI are gone.
I have another Pihole in this network, that one works well even after the upgrade.
I have tried repairing it with the pihole -r command and refreshing the Gravity database, restarting the Raspberry, but none of them worked.
What can cause an issue after an upgrade like this?
On the Pi-hole that is showing this error, please run
pihole checkout ftl development
And see if it's more stable.
NOTE: For anyone coming to this post from Reddit or a search engine: Don't run this command. This is only for this specific user and this specific case.
Also another pihole in a separate network has some issues as well after the upgrade.
Basically the gravity.db seems malformed, it won't even update the db with the pihole -g command.
Debug token for this one: https://tricorder.pi-hole.net/8q0su2mezg
I don't know if this is related to the issue posted in the OP, if seems not I will gladly create a separate thread for it.
You seem to have multiple DHCP servers on the segment and multiple address subnets. Can you explain what the network configuration is?
* Received 300 bytes from eth0.10:10.0.0.2
Offered IP address: 10.0.0.3
Server IP address: 10.0.0.2
Relay-agent IP address: N/A
BOOTP server: (empty)
BOOTP file: (empty)
DHCP options:
Message type: DHCPOFFER (2)
server-identifier: 10.0.0.2
lease-time: 86400 ( 1d )
--- end of options ---
* Received 300 bytes from eth0:10.0.1.2
Offered IP address: 10.0.1.3
Server IP address: 10.0.1.2
Relay-agent IP address: N/A
BOOTP server: (empty)
BOOTP file: (empty)
DHCP options:
Message type: DHCPOFFER (2)
server-identifier: 10.0.1.2
lease-time: 86400 ( 1d )
--- end of options ---
* Received 300 bytes from eth0.20:10.0.2.2
Offered IP address: 10.0.2.3
Server IP address: 10.0.2.2
Relay-agent IP address: N/A
BOOTP server: (empty)
BOOTP file: (empty)
DHCP options:
Message type: DHCPOFFER (2)
server-identifier: 10.0.2.2
lease-time: 43200 ( 12h )
--- end of options ---
There's also some odd lines in FTLs log that suggests a third party script or modification made.
[2021-04-20 11:35:21.771 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:23.766 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:23.767 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:25.765 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:25.765 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:26.538 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:26.538 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:27.767 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:27.768 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:29.769 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:29.769 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:31.769 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:31.769 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:32.769 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:32.769 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:32.790 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:32.790 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:33.770 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:33.771 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:35.758 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:35.758 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:37.761 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:37.762 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:38.764 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:38.764 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:39.764 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:39.764 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:41.758 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:41.758 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:43.765 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:43.765 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:43.770 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:43.772 19190/T19191] IPv4 telnet error: Success (0)
[2021-04-20 11:35:43.773 19190/T19191] Client denied (at max capacity of 255): 260
[2021-04-20 11:35:43.773 19190/T19191] IPv4 telnet error: Success (0)
It's possible that you may be seeing rate limiting that could be caused by your network configuration. A more detailed explanation of how you have set up Pi-hole and the clients would help determine the best response.
Yes I think I have for this device. But that means that I have to reflash the whole SD card.
I just asked if maybe there are other methods available for downgrading.
No third party scripts used (only gravity-sync, but that does not modify anything as far as I know - it just pulls the database from here - anyway, I also suspected that this can cause issues so I disabled it since I have this problem).
Yes I have multiple interfaces (sub-interfaces) and multiple DHCP servers as well. I have multiple networks (Guest, IoT, etc..) separated by VLANs.
The Pi-hole Raspberry Pi is also used as a mDNS repeater so I need access to all networks on this device. That's why this Raspberry has multiple interfaces. Also this way, I have a DNS server for each network and I don't have to forward DNS packets to my main network.
This was configured long time ago and never had any problems with it before. Maybe this can cause issues in this new version?
Anyway I have a much simpler network, with one Pihole, one network, one DHCP server (Pihole as DHCP server) and I have problems with this version there as well:
*** [ DIAGNOSING ]: Networking
[✓] IPv4 address(es) bound to the eth0 interface:
192.168.3.2/24 does not match the IP found in /etc/pihole/setupVars.conf (https://discourse.pi-hole.net/t/use-ipv6-ula-addresses-for-pi-hole/2127)
[✓] IPv6 address(es) bound to the eth0 interface:
2001:<REDACT>::2 does not match the IP found in /etc/pihole/setupVars.conf (https://discourse.pi-hole.net/t/use-ipv6-ula-addresses-for-pi-hole/2127)
2001:<REDACT>:982f does not match the IP found in /etc/pihole/setupVars.conf (https://discourse.pi-hole.net/t/use-ipv6-ula-addresses-for-pi-hole/2127)
fe80::b76b:de83:e2b1:aacc does not match the IP found in /etc/pihole/setupVars.conf (https://discourse.pi-hole.net/t/use-ipv6-ula-addresses-for-pi-hole/2127)
^ Please note that you may have more than one IP address listed.
As long as one of them is green, and it matches what is in /etc/pihole/setupVars.conf, there is no need for concern.
The link to the FAQ is for an issue that sometimes occurs when the IPv6 address changes, which is why we check for it.
*** [ DIAGNOSING ]: Name resolution (IPv4) using a random blocked domain and a known ad-serving domain
[✗] Failed to resolve via localhost (127.0.0.1)
[✗] Failed to resolve via Pi-hole (10.0.0.50)
[✓] doubleclick.com is 172.217.18.78 via a remote, public DNS server (8.8.8.8)
And the database issue that we identified and would have been addressed with development. But now that is in the master release so run pihole -up and you'll also need to delete the existing database at /etc/pihole/pihole-FTL.db.
[2021-04-20 19:09:44.078 21839/T21843] SQLite3 message: database corruption at line 67162 of [5d4c65779d] (11)
[2021-04-20 19:09:44.087 21839/T21843] SQLite3 message: statement aborts at 6: [SELECT COUNT(DISTINCT domain) FROM vw_regex_blacklist] database disk image is malformed (11)
[2021-04-20 19:09:44.087 21839/T21843] gravityDB_count(SELECT COUNT(DISTINCT domain) FROM vw_regex_blacklist) - SQL error step database disk image is malformed
[2021-04-20 19:09:44.089 21839/T21843] WARN: Database query failed, assuming there are no blacklist regex entries
[2021-04-20 19:09:44.122 21839/T21843] SQLite3 message: database corruption at line 67162 of [5d4c65779d] (11)
[2021-04-20 19:09:44.123 21839/T21843] SQLite3 message: statement aborts at 6: [SELECT COUNT(DISTINCT domain) FROM vw_regex_whitelist] database disk image is malformed (11)
[2021-04-20 19:09:44.123 21839/T21843] gravityDB_count(SELECT COUNT(DISTINCT domain) FROM vw_regex_whitelist) - SQL error step database disk image is malformed
[2021-04-20 19:09:44.124 21839/T21843] WARN: Database query failed, assuming there are no whitelist regex entries
[2021-04-20 19:09:44.125 21839/T21843] Compiled 0 whitelist and 0 blacklist regex filters for 8 clients in 49.0 msec
It has been identified as the source for a lot of issues with Pi-hole. Like locking or corrupting the database. Sometimes it even stops FTL and never restarts it leading to about a dozen reports all over the place that Pi-hole "crashed". Investigations then reveal that the logs say that FTL was shut down externally and just never got restarted. Probably gravity sync is buggy, crashes midway and leaves Pi-hole in an undefined state.
I have seen many reports that the issues didn't happen again once the cron job for gravity sync was disabled.
Got it, I also thought about this. But as I said, this was the first thing which I removed and turned off completely.
No sync is going on right now (I can also confirm this because the script is not running, removed from crontab), but I still has this issue. And since then I have tried removing the database, repairing pihole, etc...
Ok, so this one, I have stopped pihole-FTL service, fixed the IP address in the setupVars.conf (I have read somewhere that this one can be also removed, but now I left it there) removed the pihole-FTL.db and ran pihole -up.
pihole -up reported:
[i] No source list found, or it is empty
[i] Building tree...
[✗] Unable to build gravity tree in /etc/pihole/gravity.db_temp
Error: no such table: main.gravity
Error: database disk image is malformed
Error: database disk image is malformed
[i] Number of gravity domains: ( unique domains)
Error: near ")": syntax error
Error: database disk image is malformed
[i] Number of exact blacklisted domains:
Error: database disk image is malformed
[i] Number of regex blacklist filters:
Error: database disk image is malformed
[i] Number of exact whitelisted domains:
Error: database disk image is malformed
Current Pi-hole version is v5.3.1.
Current AdminLTE version is v5.5.
Current FTL version is development vDev-74a6b07.
Once the database is damaged then it's damaged. Stopping the sync wouldn't repair the existing damage and thus the database needs to be deleted completely and started over again.
Yes, I know this.
But it seemeed at some time that the database is correct, no errors like this were reported. pihole -g ran fine and I was able to use pihole normally for a few hours.
Anyway, I will move my piholes back to the master branch, delete the database on both of them, gravity-sync is turned off completely everywhere and I will use it for a few days with the included blocklist only.
I have removed all databases, ran pihole -g -r, etc, but still getting the same after a few hours.
No third party scripts are running. https://tricorder.pi-hole.net/oaxx8hgajp
