Please follow the below template, it will help us to help you!
Expected Behaviour:
pihole services DNS reqests
Actual Behaviour:
First users connecting got no DNS requests answered.
Hard booted the Pihole server to restore service.
I have no expectation of locating the issue, but noticed this in the pihole-FTL log.
[2020-01-10 09:34:06.293 679] Resizing "/FTL-strings" from 65536 to 69632
[2020-01-10 09:34:06.293 679] FATAL: realloc_shm(): Failed to open shared memory
object "/FTL-strings": Permission denied
[2020-01-10 12:09:27.060 684] Using log file /var/log/pihole-FTL.log
[2020-01-10 12:09:27.062 684] ########## FTL started! ##########
That silence from 09:34 until 12:09 when I reloaded the pihole server looks ominous.
Maybe...
If you do you run that on an RPi, you'll regularly see time jumps in your logs. RPis are missing a proper RTC, the fake hw-clock will load a last known timestamp from a file on boot and use that until time is synced from the network again.
Likely, you were suffering an outage from somewhere shortly after 9:34 until you rebooted at 12:09. Without further data, you can't be sure about this. Your observed time gap may or may not be the result of such a time-sync update.
As to the error, a bit of a wide shot, but since I've seen you experimenting with dnsmasq and pihole-FTL in other topics on this forum:
If you'd run pihole-FTL under a different user previously, and if that run somehow not terminated correctly, it could leave shared memory locks that Pi-hole's user is not able to remove.
Getting rid of those locks by running the following command on your Pi-hole machine could help you fixing this:
Local network has a Pi based tier 1 NTP service. Namepi is synced to it. Holds time to within 100 nanoseconds or so.
Namepi has been up and running for some time, probably ten days or so, with no issues up til now.
namepi is dedicated to providing name service, so other than housekeeping, there should be nothing competing with pihole-FTL, other than the other two name servers upstream of pihole-FTL.
As far as I know any scheduled activities such as backup run between 02:00 and 05:00.
pihole-FTL usually runs as user pihole. Can't prove it was before the reboot, but it certainly is now. pihole user was set up by the install script I guess, no idea its properties.
Doubt any of the original shm config config survived the hard boot it got just after 12
I'm pretty sure that the issue was pihole-FTL locking up, thinking back, before the reboot the admin display did not have any entries in the boxes across the top and the graphs did not display.
Unless that permission issue in the log means something to somebody, I guess this will stay a mystery.
Apologies - my time-sync remark was aimed at explaining the time gap you observe, not trying to suggest a cause for your outage.
(I am going to rephrase that sentence above).
Maybe a developer could add his insights if a mere failed FTL-String resize could bring Pi-hole to a complete halt.
This could be a one time glitch due to a power problem or other interruption, or it could be an early sign that you are having system or hardware problems.
All those are possibilities. Sort of doubtful it's a power glitch in the incoming supply, there are at least six other Pi's on the same feed.
PSU is a standard Pi foundation unit, as are all the other pi's.
Only thing that comes to mind is that that Pi is using a 16g microSD. Possibly space issue?
If it happens again, is there any data I can capture before I reload it? Any shm diagnostics I can pre load so I can get more data if I see another incident.
I have remote syslog on this server, was looking through my syslog server and came across syslog entries covering this crunch. Not sure if they are of any use in tracking this, but possibly, so posted here:
The initial log is for a DHCP refresh. Think the next logs are chrontab driven pihole-FTL activiies.
Jan 10 09:26:59 namepi systemd-timesyncd[315]: Network configuration changed, trying to establish connection.
Jan 10 09:26:59 namepi systemd-timesyncd[315]: Synchronized to time server 192.168.101.120:123 (192.168.101.120).
Jan 10 09:30:01 namepi CRON[18667]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 10 09:30:01 namepi CRON[18671]: (root) CMD ( PATH="$PATH:/usr/local/bin/" pihole updatechecker local)
Jan 10 09:30:02 namepi CRON[18667]: pam_unix(cron:session): session closed for user root
Jan 10 09:34:06 namepi systemd-logind[377]: Removed session c2.
Jan 10 09:34:06 namepi systemd[1]: Stopping User Manager for UID 999...
Jan 10 09:34:06 namepi systemd[668]: Stopped target Default.
Jan 10 09:34:06 namepi systemd[668]: Stopped target Basic System.
Jan 10 09:34:06 namepi systemd[668]: Stopped target Paths.
Jan 10 09:34:06 namepi systemd[668]: Stopped target Timers.
Jan 10 09:34:06 namepi systemd[668]: Stopped target Sockets.
Jan 10 09:34:06 namepi systemd[668]: Closed GnuPG cryptographic agent (access for web browsers).
Jan 10 09:34:06 namepi systemd[668]: Closed GnuPG cryptographic agent (ssh-agent emulation).
Jan 10 09:34:06 namepi systemd[668]: Closed D-Bus User Message Bus Socket.
Jan 10 09:34:06 namepi systemd[668]: Closed GnuPG cryptographic agent and passphrase cache.
Jan 10 09:34:06 namepi systemd[668]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Jan 10 09:34:06 namepi systemd[668]: Reached target Shutdown.
Jan 10 09:34:06 namepi systemd[668]: Starting Exit the Session...
Jan 10 09:34:06 namepi systemd[668]: Received SIGRTMIN+24 from PID 18691 (kill).
Jan 10 09:34:06 namepi systemd: pam_unix(systemd-user:session): session closed for user pihole
Jan 10 09:34:06 namepi systemd[1]: Stopped User Manager for UID 999.
Jan 10 09:34:06 namepi systemd[1]: Removed slice User Slice of pihole.
Jan 10 09:39:01 namepi CRON[18701]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 10 09:39:01 namepi CRON[18705]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jan 10 09:39:01 namepi CRON[18701]: pam_unix(cron:session): session closed for user root
Jan 10 09:39:25 namepi systemd[1]: Starting Clean php session files...
Jan 10 09:39:25 namepi systemd[1]: Started Clean php session files.
Jan 10 09:40:01 namepi CRON[18754]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 10 09:40:01 namepi CRON[18758]: (root) CMD ( PATH="$PATH:/usr/local/bin/" pihole updatechecker local)
Jan 10 09:40:01 namepi CRON[18754]: pam_unix(cron:session): session closed for user root
Jan 10 09:50:01 namepi CRON[18785]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 10 09:50:01 namepi CRON[18789]: (root) CMD ( PATH="$PATH:/usr/local/bin/" pihole updatechecker local)
Jan 10 09:50:01 namepi CRON[18785]: pam_unix(cron:session): session closed for user root
