That looks like a normal startup without any errors.
Did Pi-hole bind to port 53 this time?
here is the output
sudo ss -tulpn
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
udp UNCONN 0 0 0.0.0.0:55965 0.0.0.0:* users:(("avahi-daemon",pid=357,fd=14))
udp UNCONN 0 0 0.0.0.0:5353 0.0.0.0:* users:(("avahi-daemon",pid=357,fd=12))
udp UNCONN 0 0 127.0.0.1:5053 0.0.0.0:* users:(("cloudflared",pid=465,fd=5))
udp UNCONN 0 0 0.0.0.0:68 0.0.0.0:* users:(("dhcpcd",pid=428,fd=10))
udp UNCONN 0 0 *:5353 *:* users:(("avahi-daemon",pid=357,fd=13))
udp UNCONN 0 0 *:52118 *:* users:(("avahi-daemon",pid=357,fd=15))
tcp LISTEN 0 4096 127.0.0.1:5053 0.0.0.0:* users:(("cloudflared",pid=465,fd=6))
tcp LISTEN 0 5 0.0.0.0:5900 0.0.0.0:* users:(("vncserver-x11-c",pid=501,fd=11))
tcp LISTEN 0 1024 0.0.0.0:80 0.0.0.0:* users:(("lighttpd",pid=559,fd=4))
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=526,fd=3))
tcp LISTEN 0 4096 127.0.0.1:45975 0.0.0.0:* users:(("cloudflared",pid=465,fd=3))
tcp LISTEN 0 5 [::]:5900 [::]:* users:(("vncserver-x11-c",pid=501,fd=10))
tcp LISTEN 0 1024 [::]:80 [::]:* users:(("lighttpd",pid=559,fd=5))
tcp LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=526,fd=4))
I have uploaded another debug log as well.
https://tricorder.pi-hole.net/1b6pd3cacw
The Web interface says DNS not running, FTL still down
The new debug log just looks the same as your first.
It's probably worth noting that content of pihole-FTL.log has changed, while pihole.log just stayed the same.
This is coherent with Pi-hole starting up and logging normally, but not processing and logging any queries.
Let's check if pihole-FTL is running at all:
ps -e | grep "PID\|pihole"
And also check dnsmasq configuration for possible syntax errors:
pihole-FTL dnsmasq-test
~ $ ps -e | grep "PID|pihole"
PID TTY TIME CMD
pihole-FTL dnsmasq-test
dnsmasq: syntax check OK.
It seems Pi-hole is indeed not running anymore.
Your recent pihole-FTL.log shows it to have started with the following PID:
[2020-09-05 05:46:49.217 5260M] PID of FTL process: 5260
sudo lsof -p 5260 likely won't return anything, as no Pi-hole process seems to be active.
Try checking the service again:
sudo systemctl status --full --no-pager pihole-FTL.service
Also, check the current end of /var/log/pihole-FTL.log for any errors that would imply a shutdown of pihole-FTL.
And check for under-voltage situations also:
dmesg -T | grep -i voltage
Before my communication with you, I noticed that I can restart the RPI and every thing would work fine for about 10 min then DNS would stop working and FTL was down.
sudo lsof -p 5260
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
Output information may be incomplete.
$ sudo systemctl status --full --no-pager pihole-FTL.service
● pihole-FTL.service - LSB: pihole-FTL daemon
Loaded: loaded (/etc/init.d/pihole-FTL; generated)
Active: active (exited) since Sat 2020-09-05 05:46:51 CDT; 1h 25min ago
Docs: man:systemd-sysv-generator(8)
Process: 5215 ExecStart=/etc/init.d/pihole-FTL start (code=exited, status=0/SUCCESS)
Sep 05 05:45:50 kidsraspberrypi systemd[1]: Starting LSB: pihole-FTL daemon...
Sep 05 05:45:50 kidsraspberrypi pihole-FTL[5215]: Not running
Sep 05 05:45:51 kidsraspberrypi su[5233]: (to pihole) root on none
Sep 05 05:45:51 kidsraspberrypi su[5233]: pam_unix(su:session): session opened for user pihole by (uid=0)
Sep 05 05:46:49 kidsraspberrypi pihole-FTL[5215]: FTL started!
Sep 05 05:46:51 kidsraspberrypi systemd[1]: Started LSB: pihole-FTL daemon.
pi@kidsraspberrypi:~ $
This line of code does not return anything.
pi@kidsraspberrypi:~ $ dmesg -T | grep -i voltage
Try running pihole-FTL in the foreground -f with below and post the output when it stops (or not):
sudo setcap CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_NET_ADMIN,CAP_SYS_NICE+eip "$(which pihole-FTL)"; sudo su -s /bin/sh -c "/usr/bin/pihole-FTL -f"
You would need to kill it again in another SSH session if still running and when finished diagnosing with:
sudo kill $(pidof pihole-FTL)
Likely out of memory:
[2020-09-05 03:48:46.017 1422M] Resizing "/FTL-queries" from 484671488 to 484900864
[2020-09-05 03:48:46.036 1422M] Imported 8658199 queries from the long-term database
[2020-09-05 03:48:46.037 1422M] -> Total DNS queries: 8658199
[2020-09-05 03:48:46.037 1422M] -> Cached DNS queries: 47
[2020-09-05 03:48:46.037 1422M] -> Forwarded DNS queries: 8658151
[2020-09-05 03:48:46.037 1422M] -> Blocked DNS queries: 1
[2020-09-05 03:48:46.037 1422M] -> Unknown DNS queries: 0
[2020-09-05 03:48:46.037 1422M] -> Unique domains: 51
[2020-09-05 03:48:46.037 1422M] -> Unique clients: 10
[2020-09-05 03:48:46.038 1422M] -> Known forward destinations: 3
2 Likes
How do I fix the memory issue. This is a dedicated rpi running just the stock buster version and Pihole.
First, take a look at where 8.6 million DNS requests are coming from. What is the output of these commands from the Pi terminal:
echo ">top-clients >quit" | nc localhost 4711
echo ">top-domains >quit" | nc localhost 4711
echo ">top-ads >quit" | nc localhost 4711
Not sure I am doing this right.
I typed your suggestions into terminal and there appears to be no output.
pi@kidsraspberrypi:~ $ echo ">top-clients >quit" | nc localhost 4711
pi@kidsraspberrypi:~ $ echo ">top-domains >quit" | nc localhost 4711
pi@kidsraspberrypi:~ $ echo ">top-ads >quit" | nc localhost 4711
Restart FTL and run them again.
pi@kidsraspberrypi:~ $ echo ">top-ads >quit" | nc localhost 4711
0 1 metrics.icloud.com
pi@kidsraspberrypi:~ $ echo ">top-domains >quit" | nc localhost 4711
pi@kidsraspberrypi:~ $ echo ">top-ads >quit" | nc localhost 4711
What is the output of the following:
ls -lha /etc/pihole/*.db
pi@kidsraspberrypi:~ $ ls -lha /etc/pihole/*.db
-rw-rw-r-- 1 pihole pihole 165M Aug 30 14:58 /etc/pihole/gravity.db
-rw-r--r-- 1 pihole pihole 2.4M Aug 15 07:43 /etc/pihole/macvendor.db
-rw-r--r-- 1 pihole pihole 1.8G Sep 5 08:29 /etc/pihole/pihole-FTL.db
Probably you'll see the oom-killer active in the kernel ring buffer if running out:
dmesg -T
EDIT:
https://docs.memset.com/other/linux-s-oom-process-killer
This is a pretty huge long term database. When you start Pi-hole, it reads the most recent 24 hours from the database, and then loads that that into memory. With your very large number of daily queries, this is causing memory and performance issues. Let's do this to get to the bottom of the problem. First, we will move your long term database, and create a new one from scratch. Then, monitor your Pi-hole for an hour or so and re-run those three commands to see the activity in that hour.
sudo service pihole-FTL stop
sudo mv /etc/pihole/pihole-FTL.db /etc/pihole/pihole-FTL-old.db
sudo service pihole-FTL start
Yesterday I was frustrated and reinstalled rasbian and pihole and started from scratch.
This morning FTL is down again!!!
sudo service pihole-FTL stop
sudo mv /etc/pihole/pihole-FTL.db /etc/pihole/pihole-FTL-old.db
sudo service pihole-FTL start
Just did the above. Any other suggestions? I am using a POE hat. Could that have anything to do with FTL shutting down?
Here is the debug output https://tricorder.pi-hole.net/i88apirhzf
This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.