FTL daemon fails to start after reboot but works fine when sudo pihole-FTL is ran

Help
6

How does your systemd unit look like?

systemctl cat pihole-FTL.service

Above might need sudo or root powers for your distro.

EDIT: Ow found a nice one, what does below show?

sudo capsh --print

From:

pi@ph5b:~ $ man capsh
[..]
DESCRIPTION
       Linux  capability  support and use can be explored and constrained
       with this tool. This tool provides a  handy  wrapper  for  certain
       types of capability testing and environment creation. It also pro‐
       vides some debugging features useful  for  summarizing  capability
       state.

EDIT2: Ow here is mine:

pi@ph5b:~ $ sudo capsh --print
Current: =ep
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,cap_perfmon,cap_bpf,cap_checkpoint_restore
Ambient set =
Current IAB:
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=0(root) euid=0(root)
gid=0(root)
groups=0(root)
Guessed mode: UNCERTAIN (0)
7

Hi,

For the first one I get this output:

root@UniFi-CloudKey:~# systemctl cat pihole-FTL.service
# /etc/systemd/system/pihole-FTL.service
[Unit]
Description=Pi-hole FTL
# This unit is supposed to indicate when network functionality is available, but it is only
# very weakly defined what that is supposed to mean, with one exception: at shutdown, a unit
# that is ordered after network-online.target will be stopped before the network
Wants=network-online.target
After=network-online.target
# A target that should be used as synchronization point for all host/network name service lookups.
# All services for which the availability of full host/network name resolution is essential should
# be ordered after this target, but not pull it in.
Wants=nss-lookup.target
Before=nss-lookup.target

# Limit (re)start loop to 5 within 1 minute
StartLimitBurst=5
StartLimitIntervalSec=60s

[Service]
User=pihole
PermissionsStartOnly=true
AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_NET_ADMIN CAP_SYS_NICE CAP_IPC_LOCK CAP_CHOWN

ExecStartPre=/opt/pihole/pihole-FTL-prestart.sh
ExecStart=/usr/bin/pihole-FTL -f
Restart=on-failure
RestartSec=5s
ExecReload=/bin/kill -HUP $MAINPID
ExecStopPost=/opt/pihole/pihole-FTL-poststop.sh

# Use graceful shutdown with a reasonable timeout
TimeoutStopSec=10s

# Make /usr, /boot, /etc and possibly some more folders read-only...
ProtectSystem=full
# ... except /etc/pihole
# This merely retains r/w access rights, it does not add any new.
# Must still be writable on the host!
ReadWriteDirectories=/etc/pihole

[Install]
WantedBy=multi-user.target

For the second I get this:

root@UniFi-CloudKey:~# sudo capsh --print
Current: =ep
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
uid=0(root)
gid=0(root)
groups=0(root)
root@UniFi-CloudKey:~#
8

Kernel caps available seem to be the same comparing your and my capsh --print outputs.

But your systemd unit appears completely different from mine (also Buster version 10 but a Raspbian release):

pi@ph5a:~ $ lsb_release -d
Description:    Raspbian GNU/Linux 10 (buster)

pi@ph5a:~ $ systemctl cat pihole-FTL.service
# /run/systemd/generator.late/pihole-FTL.service
# Automatically generated by systemd-sysv-generator

[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/pihole-FTL
Description=LSB: pihole-FTL daemon
Before=multi-user.target
Before=multi-user.target
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
After=network-online.target
Wants=network-online.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
SuccessExitStatus=5 6
ExecStart=/etc/init.d/pihole-FTL start
ExecStop=/etc/init.d/pihole-FTL stop
ExecReload=/etc/init.d/pihole-FTL reload

Notice the ExecStart= line above with that missing /etc/init.d/ script on your system which you mentioned earlier.
Whats your systemd version?

apt policy systemd

Did you create that /etc/systemd/system/pihole-FTL.service file manually or something?
Your not suppose to as the unit should be auto generated:

pi@ph5a:~ $ systemctl cat pihole-FTL.service
# /run/systemd/generator.late/pihole-FTL.service
# Automatically generated by systemd-sysv-generator
[..]

pi@ph5b:~ $ man systemd-sysv-generator
[..]
DESCRIPTION
       systemd-sysv-generator is a generator that creates wrapper
       .service units for SysV init[1] scripts in /etc/init.d/* at boot
       and when configuration of the system manager is reloaded. This
       will allow systemd(1) to support them similarly to native units.

And bc the /etc/init.d/pihole-FTL file is also missing I have no clue how you got to this point.

9

Great question, nothing here is manually generated so this must be what it does on Debian. I can only assume it some differences with armhf setups. The output is as follows:

root@UniFi-CloudKey:~# apt policy systemd
systemd:
  Installed: 241-7~deb10u9
  Candidate: 241-7~deb10u9
  Version table:
 *** 241-7~deb10u9 500
        500 http://deb.debian.org/debian-security buster/updates/main armhf Packages
        100 /var/lib/dpkg/status
     241-7~deb10u8 500
        500 http://deb.debian.org/debian buster/main armhf Packages
root@UniFi-CloudKey:~#

Maybe I should try and rebuilld this with raspbian distro - at this point not sure it matters :slight_smile:

11

Try backup that file (if its an actual file or symlink) to your home folder ~ with below:

sudo mv /etc/systemd/system/pihole-FTL.service ~

And create that missing init.d script manually with below:

sudo nano /etc/init.d/pihole-FTL

And add below content:

Make it executable:

sudo chmod +x /etc/init.d/pihole-FTL

And run below to be sure:

sudo systemctl daemon-reload

Restart:

sudo service pihole-FTL restart

sudo systemctl restart pihole-FTL.service

And check status and journals and if that systemd unit is displayed different now:

systemctl cat pihole-FTL.service

systemctl status pihole-FTL.service

journalctl --full --no-pager -b -u pihole-FTL.service

1 Like
12

OMG, that got it working. I had to correct the following - all good :slight_smile:

sudo systemctl restart pi-hole-FTL.service

to 

sudo systemctl restart pihole-FTL.service

Log output:

root@UniFi-CloudKey:~# systemctl status pihole-FTL.service
● pihole-FTL.service - LSB: pihole-FTL daemon
   Loaded: loaded (/etc/init.d/pihole-FTL; generated)
   Active: active (exited) since Wed 2023-04-05 11:57:38 PDT; 2min 7s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 2072 ExecStart=/etc/init.d/pihole-FTL start (code=exited, status=0/SUCCESS)

Apr 05 11:57:38 UniFi-CloudKey systemd[1]: Starting LSB: pihole-FTL daemon...
Apr 05 11:57:38 UniFi-CloudKey pihole-FTL[2072]: Not running
Apr 05 11:57:38 UniFi-CloudKey pihole-FTL[2072]: FTL started!
Apr 05 11:57:38 UniFi-CloudKey systemd[1]: Started LSB: pihole-FTL daemon.
root@UniFi-CloudKey:~#

and

root@UniFi-CloudKey:~# journalctl --full --no-pager -b -u pihole-FTL.service
-- Logs begin at Tue 2023-04-04 15:09:32 PDT, end at Wed 2023-04-05 12:00:05 PDT. --
Apr 05 11:57:33 UniFi-CloudKey systemd[1]: Stopping LSB: pihole-FTL daemon...
Apr 05 11:57:37 UniFi-CloudKey pihole-FTL[2040]: ....
Apr 05 11:57:37 UniFi-CloudKey pihole-FTL[2040]: Stopped
Apr 05 11:57:38 UniFi-CloudKey systemd[1]: pihole-FTL.service: Succeeded.
Apr 05 11:57:38 UniFi-CloudKey systemd[1]: Stopped LSB: pihole-FTL daemon.
Apr 05 11:57:38 UniFi-CloudKey systemd[1]: Starting LSB: pihole-FTL daemon...
Apr 05 11:57:38 UniFi-CloudKey pihole-FTL[2072]: Not running
Apr 05 11:57:38 UniFi-CloudKey pihole-FTL[2072]: FTL started!
Apr 05 11:57:38 UniFi-CloudKey systemd[1]: Started LSB: pihole-FTL daemon.
root@UniFi-CloudKey:~#```

I've rebooted a few times so now will allow it to sit and run and see how it works out but so far so good.

Super appreciate the help and support!!!
1 Like
13

I'm only human :wink:
Fixed that in my reply.
Shall I mark that one as a solution (you can also do it)?

EDIT: Ow and you may want to keep an eye on above after you applied a Pi-hole update (pihole -up) to make sure it doesnt break again.

1 Like
14

If your good with me taking a few days to stress test etc I'll then be happy to mark as solution. Yes, will make sure I capture this as well in case of updates. I'm going to go through some apt-get updates/upgrades too.

15

Sure.
I'm curious too.

Ps. I added a section above to show what systemd does under the hood with old school SystemV init.d scripts:

16

This is not unlikely.
When we started to supply a real systemd service file, pihole-FTL.service changed.

Old:

New:

1 Like
17

Aha, I didnt updated in a while and that explains :wink:
I had my suspicion and thats why I didnt immediately suggested to remove that unit file to see if anyone would jump in.
I'll update shortly to see what has changed.

So this means that if the OP updates Pi-hole, its back to square one again.
Just to confirm my initial findings, below does mean something going wrong with the caps am I right?

18

@agsh , what do below two currently show?

ps h -o user,group -C pihole-FTL

getpcaps $(pidof -s pihole-FTL)

20

I think so. You provided a good workaround.

I think so, based on the status code.

Here is someone reporting the same issue. They go a step further by changing the user to root and commenting out the capabilities

21

That will also break once you run an update/repair/reconfigure ... I guess?
And that would be too easy :wink:

22

True :slight_smile:

1 Like
23

deHakkelaar Here ya go. (i'm new so not able to do @)

Also, so far everything has been working fine. I'm waiting for a pi-hole update to see if an update will break me again. So far apt-get update/upgrades are fine.

root@Pi-Hole:~# ps h -o user,group -C pihole-FTL
pihole   pihole
root@Pi-Hole:~# getpcaps $(pidof -s pihole-FTL)
Capabilities for `2496': = cap_chown,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_sys_nice+ep
root@Pi-Hole:~#
24

I've also copied down this post to ensure if it does break again, I have a way to get it back. I bet it something to do with the Ubiquiti cloud key and Ubiquiti's kernel implementation (just a guess). I've been tempted to upgrade to the latest debian kernel but that may stop me from recovering the cloud key if I ever want too so I've not gone that far.

25

Buster + very old PH version v5.3.1 still using init.d:

pi@ph5a:~ $ getpcaps $(pidof -s pihole-FTL)
cap_net_bind_service,cap_net_admin,cap_net_raw,cap_sys_nice+ep

After updating and now using the /etc/systemd unit instead:

pi@ph5a:~ $ getpcaps $(pidof -s pihole-FTL)
cap_chown,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_sys_nice+eip

Bullseye + not so old PH version v5.12.2 still using init.d:

pi@ph5b:~ $ getpcaps $(pidof -s pihole-FTL)
cap_chown,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_sys_nice=ep

And below is yours for Buster that we modified to use init.d:

root@Pi-Hole:~# getpcaps $(pidof -s pihole-FTL)
cap_chown,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_sys_nice+ep

I havent figured out what those eip flags at the end mean and if they matter... yet:

pi@ph5a:~ $ man getpcaps
[..]
       Each clause consists of a list of comma-separated capability names
       (or the word `all'), followed by an action-list.  An action-list
       consists of a sequence of operator flag pairs.  Legal operators
       are: `=', '+', and `-'.  Legal flags are: `e', `i', and `p'.
       These flags are case-sensitive and specify the Effective,
       Inheritable and Permitted sets respectively.
26

This post was so helpfull! I had my pi-hole not working since January Qnap LXD Container with Pihole not working after update

Thank you so much!

1 Like
27

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.