I've had this happen a handful of times before but it's been a while. However this morning woke up to find my PiHole stopped (disabled) again. I can get to the admin interface and click 'Enable' but it appears to try & fail, going back to 'Disabled' shortly after.
This time I managed to remember to do a debug log. Found here: https://tricorder.pi-hole.net/3FOp0dRL/
The previous times this has happened, restarting the Docker container fixed the issue for a while, but obviously hasn't been a permanent solution.
Content of pihole-FTL.log when I tail it from the UI has this chunk repeating every-so-often:
[2022-03-29 15:40:01.298 493/T502] ERROR: SQL query "END TRANSACTION" failed: database is locked
[2022-03-29 15:40:01.298 493/T502] END TRANSACTION failed when trying to store queries to long-term database
[2022-03-29 15:40:01.298 493/T502] Keeping queries in memory for later new attempt
[2022-03-29 15:40:02.535 493/T502] ERROR: SQL query "END TRANSACTION" failed: database is locked
[2022-03-29 15:40:02.535 493/T502] WARNING: Storing devices in network table failed: database is locked
So seems like something is locking the DB and either forgetting to unlock or crashing before it can.
Thanks for any help!
It would seem that Pi-hole receives an excessive amount of API requests (showing up in the debug log even more often than the lines you've quoted):
*** [ DIAGNOSING ]: contents of /var/log
-rw-r--r-- 1 pihole pihole 15M Mar 29 15:30 /var/log/pihole-FTL.log
-----tail of pihole-FTL.log------
[2022-03-29 15:29:26.917 493/T499] Client denied (at max capacity of 255): 315
[2022-03-29 15:29:26.917 493/T499] IPv4 telnet error: Success (0)
[2022-03-29 15:29:27.935 493/T499] Client denied (at max capacity of 255): 315
[2022-03-29 15:29:27.935 493/T499] IPv4 telnet error: Success (0)
[2022-03-29 15:29:28.947 493/T499] Client denied (at max capacity of 255): 315
[2022-03-29 15:29:28.947 493/T499] IPv4 telnet error: Success (0)
[2022-03-29 15:29:29.961 493/T499] Client denied (at max capacity of 255): 315
[2022-03-29 15:29:29.962 493/T499] IPv4 telnet error: Success (0)
[2022-03-29 15:29:30.981 493/T499] Client denied (at max capacity of 255): 315
[2022-03-29 15:29:30.981 493/T499] IPv4 telnet error: Success (0)
[2022-03-29 15:29:32.034 493/T499] Client denied (at max capacity of 255): 315
[2022-03-29 15:29:32.034 493/T499] IPv4 telnet error: Success (0)
[2022-03-29 15:29:33.054 493/T499] Client denied (at max capacity of 255): 315
[2022-03-29 15:29:33.054 493/T499] IPv4 telnet error: Success (0)
[2022-03-29 15:29:34.067 493/T499] Client denied (at max capacity of 255): 315
[2022-03-29 15:29:34.067 493/T499] IPv4 telnet error: Success (0)
[2022-03-29 15:29:35.080 493/T499] Client denied (at max capacity of 255): 315
[2022-03-29 15:29:35.080 493/T499] IPv4 telnet error: Success (0)
[2022-03-29 15:29:36.093 493/T499] Client denied (at max capacity of 255): 315
[2022-03-29 15:29:36.093 493/T499] IPv4 telnet error: Success (0)
[2022-03-29 15:29:36.490 493/T499] Client denied (at max capacity of 255): 315
[2022-03-29 15:29:36.490 493/T499] IPv4 telnet error: Success (0)
[2022-03-29 15:29:36.515 493/T499] Client denied (at max capacity of 255): 315
[2022-03-29 15:29:36.515 493/T499] IPv4 telnet error: Success (0)
[2022-03-29 15:29:37.267 493/T499] Client denied (at max capacity of 255): 315
[2022-03-29 15:29:37.267 493/T499] IPv4 telnet error: Success (0)
[2022-03-29 15:29:37.344 493/T499] Client denied (at max capacity of 255): 315
[2022-03-29 15:29:37.344 493/T499] IPv4 telnet error: Success (0)
[2022-03-29 15:29:49.467 493/T499] Client denied (at max capacity of 255): 315
[2022-03-29 15:29:49.467 493/T499] IPv4 telnet error: Success (0)
Do you run any third-party scripts using Pi-hole's API?
Nope, none at all. This just runs on my Synology and I only check on it when things start going weird/unexpected on my client PCs (e.g. ads start showing, DNS resolution delays/failures, etc.)
Let's check Pi-hole's databases by running the following commands on your Pi-hole machine:
sqlite3 pihole-FTL.db "PRAGMA integrity_check"
sqlite3 gravity.db "PRAGMA integrity_check"
Depending on your Docker volume mounts for your Pi-hole container, you would have to change into the correct directory before running above commands, or to adjust for file locations on your system.
That looks like you ran that command from the root directory.
If that's the case:
I can't be more specific since I do not know where you did mount your Docker volumes for your Pi-hole container.
Gotchya. Found them and re-ran:
root@PiHole:/# sqlite3 ./etc/pihole/gravity.db "PRAGMA integrity_check"
ok
root@PiHole:/# sqlite3 ./etc/pihole/pihole-FTL.db "PRAGMA integrity_check"
ok
That's good, as it shows that integrity of both of Pi-hole's databases is intact.
At the same time, it doesn't provide us with any leads as to why your Pi-hole would fail shortly after starting it.
Let's see if your logs actually contains crash information:
zgrep -n 'FTL crashed!' /var/log/pihole-FTL.log*
EDIT:
Your observation also bears similarities with DNS resolvers stops randomly.
Since that report is some two years old, I don't know whether it's likely you would be affected by the same underlying Docker issue.
Still, you could try to look for a message like
docker: Error response from daemon: cgroups: cannot found cgroup mount destination: unknown.
in your Docker logs.
You should be able to access them with
docker logs <name of your Pi-hole container>
But I am afraid not aware if that would work or how you'd do that instead on a Synology system.
root@PiHole:/# zgrep -n 'FTL crashed!' /var/log/pihole-FTL.log*
root@PiHole:/#
I don't find the daemon error in my logs but I do see a whole hell of a lot of these
