Please follow the below template, it will help us to help you!
If you are Experiencing issues with a Pi-hole install that has non-standard elements (e.g you are using nginx instead of lighttpd, or there is some other aspect of your install that is customised) - please use the Community Help category.
Expected Behaviour:
Pi-Hole container keeps running
Actual Behaviour:
I have a new 3 node Docker Swarm setup, with Dell 3050s MFF machines. All nodes FS are synched with GlusterFS, each node of the cluster is running a dedicated service with its own separated folder instance of Pi Hole due to redundancy and I run nebula-sync to sync manually the settings between them.
And since Ive built this setup Im having a weird situation where the container randomly receives a [i] Container stop requested... and starts shutting down. Then since my policy is to restart the container, it reboots and do the same thing again right away [i] Container stop requested... shutdown, rince and repeat, does that about 10 times, until exactly like it started it stops and the container stabilizes again and keeps on.
Then the system works for a random amount of time, it already stayed up for like 30 minutes, and sometimes several hours. This already happened on every node and this only happens with the pihole services. I have about 8 other services on each node and none of them are behaving like that. Also both docker swarm and the phyisical monitoring of the nodes doesn't show anything out of ordinary.
The Pi Hole logs don't show anything going wrong before [i] Container stop requested..., it just decides to shutdown.
Possibly unrelated to your issue, but as you are running your Pi-hole as a Docker container which only has IPv4 connectivity, you could consider to configure Pi-hole's embedded webserver to only listen on IPv4 ports, e.g. by setting FTLCONF_webserver_port to "80,443os".
Would there be any revealing lines in /var/log/pihole/FTL.log immediately preceding the times when your container shuts down?
Would the times when the container shuts down correlate with nebula-sync activity?
Not really, it just say its shutting down, i'll get a full log for you once it happens again. I've also disabled nebula-sync exactly because i tought it was interfering and the problem kept happening even without it.
It strangely seems to mostly happens around 11:00 AM for some reason. I dont have any cron at that time on any of my systems.
An update. Its not an automation i've added, tbh I didn't add any automations yet hahaha, and on docker logs both of the swarm and the service itself with journal, nothing reported in the hours the container shutdown.
But I've discovered that this problem is caused by either Docker Swarm or GlusterFS. I cannot say which one because I just rolled back to my old setup, which just run the nodes independently of each other and the issue just vanished, i even restore the same volumes that pi hole was installed with all my configs and data. I'm suspicious of Docker Swarm since it may sense that something is wrong with the container and kill it to reboot. I dont really know because there's nothing on the logs indicating the reason why something is asking the container to shutdown.
With all three nodes running independently of each other but still with the same underlying OS (debian), docker instalation and hardware, the problem vanished. I miss some cool stuff like automatic failover and the overlay network, but i prefer a less flexible but stable setup than the other way around.