FTL daemon fails to start after reboot but works fine when sudo pihole-FTL is ran

Help
1

Please follow the below template, it will help us to help you!

If you are Experiencing issues with a Pi-hole install that has non-standard elements (e.g you are using nginx instead of lighttpd, or there is some other aspect of your install that is customised) - please use the Community Help category.

Expected Behaviour:

FTL Daemon to start after a reboot.To save manual intervention.

System is a Unifi Cloudkey Gen 1, upgraded to Debian 10 with the unifi components removed. I wanted to use this as it has POE, small factor and I have upgraded the rest of my gear.

I've rebuilt this setup a few times and I cannot get past this point.

Actual Behaviour:

FTL Daemon fails with the following. However, I can run sudo pihole-FTL and it will operate fine and you will see that FTL is running on port 53 in the logs due to this.

*** [ DIAGNOSING ]: Pi-hole-FTL full status

  • pihole-FTL.service - Pi-hole FTL
    Loaded: loaded (/etc/systemd/system/pihole-FTL.service; enabled; vendor preset: enabled)
    Active: failed (Result: exit-code) since Sun 2023-04-02 13:25:50 PDT; 6min ago
    Process: 1097 ExecStartPre=/opt/pihole/pihole-FTL-prestart.sh (code=exited, status=0/SUCCESS)
    Process: 1114 ExecStart=/usr/bin/pihole-FTL -f (code=exited, status=218/CAPABILITIES)
    Process: 1117 ExecStopPost=/opt/pihole/pihole-FTL-poststop.sh (code=exited, status=0/SUCCESS)
    Main PID: 1114 (code=exited, status=218/CAPABILITIES)

Apr 02 13:25:50 UniFi-CloudKey systemd[1]: pihole-FTL.service: Service RestartSec=5s expired, scheduling restart.
Apr 02 13:25:50 UniFi-CloudKey systemd[1]: pihole-FTL.service: Scheduled restart job, restart counter is at 5.
Apr 02 13:25:50 UniFi-CloudKey systemd[1]: Stopped Pi-hole FTL.
Apr 02 13:25:50 UniFi-CloudKey systemd[1]: pihole-FTL.service: Start request repeated too quickly.
Apr 02 13:25:50 UniFi-CloudKey systemd[1]: pihole-FTL.service: Failed with result 'exit-code'.
Apr 02 13:25:50 UniFi-CloudKey systemd[1]: Failed to start Pi-hole FTL.

Debug Token:

https://tricorder.pi-hole.net/TSznJd8v/

2

It seems to lack Linux capabilities:

pi@ph5b:~ $ man capabilities
[..]
DESCRIPTION
       For  the purpose of performing permission checks, traditional UNIX
       implementations distinguish two categories  of  processes:  privi‐
       leged  processes (whose effective user ID is 0, referred to as su‐
       peruser or root), and unprivileged processes (whose effective  UID
       is  nonzero).   Privileged  processes bypass all kernel permission
       checks, while unprivileged processes are subject to  full  permis‐
       sion  checking based on the process's credentials (usually: effec‐
       tive UID, effective GID, and supplementary group list).

       Starting with kernel 2.2, Linux divides the privileges  tradition‐
       ally associated with superuser into distinct units, known as capa‐
       bilities, which can be independently enabled and disabled.   Capa‐
       bilities are a per-thread attribute.

   Capabilities list
       The  following  list  shows the capabilities implemented on Linux,
       and the operations or behaviors that each capability permits:
[..]

Mine are:

pi@ph5b:~ $ getpcaps $(pidof -s pihole-FTL)
14556: cap_chown,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_sys_nice=ep

And Pi-hole tries to set them here:

pi@ph5b:~ $ cat /etc/init.d/pihole-FTL
[..]
    if setcap CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_NET_ADMIN,CAP_SYS_NICE,CAP_IPC_LOCK,CAP_CHOWN+eip "/usr/bin/pihole-FTL"; then

pi@ph5b:~ $ man setcap
[..]
DESCRIPTION
       In the absence of the -v (verify) option setcap sets the capabili‐
       ties  of  each  specified  filename to the capabilities specified.
[..]

Sounds like your missing some in that environment where your running Pi-hole in.
EDIT: If its some form of container/jail, its lacking permissions.

3

I do not have a /etc/init.d/pihole-FTL. I've checked what I can for the capabilities and they can be set from cmd line but not via systemctl script.

root@UniFi-CloudKey:/etc/init.d# getcap /usr/bin/pihole-FTL
/usr/bin/pihole-FTL = cap_chown,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_sys_nice+eip

Agree it seems to be missing some capability but I have no idea how to look at that.

4

How did you go about repurposing your Cloudkey?
Did you use a specific script for doing so to install Pi-hole?

5

I used this guide [Guide] Installing Pi-hole on Unifi Cloudkey v1 and have tried using both bullseye and buster distros both give me the same error. The pihole install is via curl -sSL https://install.pi-hole.net | bash.

I've also finished the OS install with a sudo apt-get full-upgrade and then disabled unneeded services.

6

How does your systemd unit look like?

systemctl cat pihole-FTL.service

Above might need sudo or root powers for your distro.

EDIT: Ow found a nice one, what does below show?

sudo capsh --print

From:

pi@ph5b:~ $ man capsh
[..]
DESCRIPTION
       Linux  capability  support and use can be explored and constrained
       with this tool. This tool provides a  handy  wrapper  for  certain
       types of capability testing and environment creation. It also pro‐
       vides some debugging features useful  for  summarizing  capability
       state.

EDIT2: Ow here is mine:

pi@ph5b:~ $ sudo capsh --print
Current: =ep
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,cap_perfmon,cap_bpf,cap_checkpoint_restore
Ambient set =
Current IAB:
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=0(root) euid=0(root)
gid=0(root)
groups=0(root)
Guessed mode: UNCERTAIN (0)
7

Hi,

For the first one I get this output:

root@UniFi-CloudKey:~# systemctl cat pihole-FTL.service
# /etc/systemd/system/pihole-FTL.service
[Unit]
Description=Pi-hole FTL
# This unit is supposed to indicate when network functionality is available, but it is only
# very weakly defined what that is supposed to mean, with one exception: at shutdown, a unit
# that is ordered after network-online.target will be stopped before the network
Wants=network-online.target
After=network-online.target
# A target that should be used as synchronization point for all host/network name service lookups.
# All services for which the availability of full host/network name resolution is essential should
# be ordered after this target, but not pull it in.
Wants=nss-lookup.target
Before=nss-lookup.target

# Limit (re)start loop to 5 within 1 minute
StartLimitBurst=5
StartLimitIntervalSec=60s

[Service]
User=pihole
PermissionsStartOnly=true
AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_NET_ADMIN CAP_SYS_NICE CAP_IPC_LOCK CAP_CHOWN

ExecStartPre=/opt/pihole/pihole-FTL-prestart.sh
ExecStart=/usr/bin/pihole-FTL -f
Restart=on-failure
RestartSec=5s
ExecReload=/bin/kill -HUP $MAINPID
ExecStopPost=/opt/pihole/pihole-FTL-poststop.sh

# Use graceful shutdown with a reasonable timeout
TimeoutStopSec=10s

# Make /usr, /boot, /etc and possibly some more folders read-only...
ProtectSystem=full
# ... except /etc/pihole
# This merely retains r/w access rights, it does not add any new.
# Must still be writable on the host!
ReadWriteDirectories=/etc/pihole

[Install]
WantedBy=multi-user.target

For the second I get this:

root@UniFi-CloudKey:~# sudo capsh --print
Current: =ep
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
uid=0(root)
gid=0(root)
groups=0(root)
root@UniFi-CloudKey:~#
8

Kernel caps available seem to be the same comparing your and my capsh --print outputs.

But your systemd unit appears completely different from mine (also Buster version 10 but a Raspbian release):

pi@ph5a:~ $ lsb_release -d
Description:    Raspbian GNU/Linux 10 (buster)

pi@ph5a:~ $ systemctl cat pihole-FTL.service
# /run/systemd/generator.late/pihole-FTL.service
# Automatically generated by systemd-sysv-generator

[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/pihole-FTL
Description=LSB: pihole-FTL daemon
Before=multi-user.target
Before=multi-user.target
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
After=network-online.target
Wants=network-online.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
SuccessExitStatus=5 6
ExecStart=/etc/init.d/pihole-FTL start
ExecStop=/etc/init.d/pihole-FTL stop
ExecReload=/etc/init.d/pihole-FTL reload

Notice the ExecStart= line above with that missing /etc/init.d/ script on your system which you mentioned earlier.
Whats your systemd version?

apt policy systemd

Did you create that /etc/systemd/system/pihole-FTL.service file manually or something?
Your not suppose to as the unit should be auto generated:

pi@ph5a:~ $ systemctl cat pihole-FTL.service
# /run/systemd/generator.late/pihole-FTL.service
# Automatically generated by systemd-sysv-generator
[..]

pi@ph5b:~ $ man systemd-sysv-generator
[..]
DESCRIPTION
       systemd-sysv-generator is a generator that creates wrapper
       .service units for SysV init[1] scripts in /etc/init.d/* at boot
       and when configuration of the system manager is reloaded. This
       will allow systemd(1) to support them similarly to native units.

And bc the /etc/init.d/pihole-FTL file is also missing I have no clue how you got to this point.

9

Great question, nothing here is manually generated so this must be what it does on Debian. I can only assume it some differences with armhf setups. The output is as follows:

root@UniFi-CloudKey:~# apt policy systemd
systemd:
  Installed: 241-7~deb10u9
  Candidate: 241-7~deb10u9
  Version table:
 *** 241-7~deb10u9 500
        500 http://deb.debian.org/debian-security buster/updates/main armhf Packages
        100 /var/lib/dpkg/status
     241-7~deb10u8 500
        500 http://deb.debian.org/debian buster/main armhf Packages
root@UniFi-CloudKey:~#

Maybe I should try and rebuilld this with raspbian distro - at this point not sure it matters :slight_smile:

11

Try backup that file (if its an actual file or symlink) to your home folder ~ with below:

sudo mv /etc/systemd/system/pihole-FTL.service ~

And create that missing init.d script manually with below:

sudo nano /etc/init.d/pihole-FTL

And add below content:

Make it executable:

sudo chmod +x /etc/init.d/pihole-FTL

And run below to be sure:

sudo systemctl daemon-reload

Restart:

sudo service pihole-FTL restart

sudo systemctl restart pihole-FTL.service

And check status and journals and if that systemd unit is displayed different now:

systemctl cat pihole-FTL.service

systemctl status pihole-FTL.service

journalctl --full --no-pager -b -u pihole-FTL.service

1 Like
12

OMG, that got it working. I had to correct the following - all good :slight_smile:

sudo systemctl restart pi-hole-FTL.service

to 

sudo systemctl restart pihole-FTL.service

Log output:

root@UniFi-CloudKey:~# systemctl status pihole-FTL.service
● pihole-FTL.service - LSB: pihole-FTL daemon
   Loaded: loaded (/etc/init.d/pihole-FTL; generated)
   Active: active (exited) since Wed 2023-04-05 11:57:38 PDT; 2min 7s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 2072 ExecStart=/etc/init.d/pihole-FTL start (code=exited, status=0/SUCCESS)

Apr 05 11:57:38 UniFi-CloudKey systemd[1]: Starting LSB: pihole-FTL daemon...
Apr 05 11:57:38 UniFi-CloudKey pihole-FTL[2072]: Not running
Apr 05 11:57:38 UniFi-CloudKey pihole-FTL[2072]: FTL started!
Apr 05 11:57:38 UniFi-CloudKey systemd[1]: Started LSB: pihole-FTL daemon.
root@UniFi-CloudKey:~#

and

root@UniFi-CloudKey:~# journalctl --full --no-pager -b -u pihole-FTL.service
-- Logs begin at Tue 2023-04-04 15:09:32 PDT, end at Wed 2023-04-05 12:00:05 PDT. --
Apr 05 11:57:33 UniFi-CloudKey systemd[1]: Stopping LSB: pihole-FTL daemon...
Apr 05 11:57:37 UniFi-CloudKey pihole-FTL[2040]: ....
Apr 05 11:57:37 UniFi-CloudKey pihole-FTL[2040]: Stopped
Apr 05 11:57:38 UniFi-CloudKey systemd[1]: pihole-FTL.service: Succeeded.
Apr 05 11:57:38 UniFi-CloudKey systemd[1]: Stopped LSB: pihole-FTL daemon.
Apr 05 11:57:38 UniFi-CloudKey systemd[1]: Starting LSB: pihole-FTL daemon...
Apr 05 11:57:38 UniFi-CloudKey pihole-FTL[2072]: Not running
Apr 05 11:57:38 UniFi-CloudKey pihole-FTL[2072]: FTL started!
Apr 05 11:57:38 UniFi-CloudKey systemd[1]: Started LSB: pihole-FTL daemon.
root@UniFi-CloudKey:~#```

I've rebooted a few times so now will allow it to sit and run and see how it works out but so far so good.

Super appreciate the help and support!!!
1 Like
13

I'm only human :wink:
Fixed that in my reply.
Shall I mark that one as a solution (you can also do it)?

EDIT: Ow and you may want to keep an eye on above after you applied a Pi-hole update (pihole -up) to make sure it doesnt break again.

1 Like
14

If your good with me taking a few days to stress test etc I'll then be happy to mark as solution. Yes, will make sure I capture this as well in case of updates. I'm going to go through some apt-get updates/upgrades too.

15

Sure.
I'm curious too.

Ps. I added a section above to show what systemd does under the hood with old school SystemV init.d scripts:

16

This is not unlikely.
When we started to supply a real systemd service file, pihole-FTL.service changed.

Old:

New:

1 Like
17

Aha, I didnt updated in a while and that explains :wink:
I had my suspicion and thats why I didnt immediately suggested to remove that unit file to see if anyone would jump in.
I'll update shortly to see what has changed.

So this means that if the OP updates Pi-hole, its back to square one again.
Just to confirm my initial findings, below does mean something going wrong with the caps am I right?

18

@agsh , what do below two currently show?

ps h -o user,group -C pihole-FTL

getpcaps $(pidof -s pihole-FTL)

20

I think so. You provided a good workaround.

I think so, based on the status code.

Here is someone reporting the same issue. They go a step further by changing the user to root and commenting out the capabilities

21

That will also break once you run an update/repair/reconfigure ... I guess?
And that would be too easy :wink:

22

True :slight_smile:

1 Like