pihole-FTL not starting since update to FTL 5.9

The issue I am facing:

Tried to update my pihole setup to the latest version, since then, I have the errror:

/usr/bin/pihole-FTL: Operation not permitted

Details about my system:

  • The pi-hole is installed in a disk image run via systemd-machine.
  • The disk image contain an up to date Debian Sid.

pihole -d : https://tricorder.pi-hole.net/N2gFyw55/

More details:

root@slb-mel-00:~# uname -a
Linux slb-mel-00 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64 GNU/Linux

root@slb-mel-00:~# ls -lh /usr/bin/pihole-FTL
-rwxr-xr-x 1 root root 15M Sep 14 17:53 /usr/bin/pihole-FTL

root@slb-mel-00:~# file /usr/bin/pihole-FTL
/usr/bin/pihole-FTL: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=8aee1c1d503ff5c1cd5ab23bb192b601557054e2, with debug_info, not stripped

root@slb-mel-00:~# getcap /usr/bin/pihole-FTL
/usr/bin/pihole-FTL cap_chown,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_sys_nice=eip

root@slb-mel-00:~# ldd /usr/bin/pihole-FTL
        linux-vdso.so.1 (0x00007ffd779ce000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f83a1bd1000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f83a1bc6000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f83a1ba5000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f83a19e0000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f83a223a000)

root@slb-mel-00:~# strace /usr/bin/pihole-FTL
execve("/usr/bin/pihole-FTL", ["/usr/bin/pihole-FTL"], 0x7fffc54cfae0 /* 16 vars */) = -1 EPERM (Operation not permitted)
strace: exec: Operation not permitted
+++ exited with 1 +++

What I have changed since installing Pi-hole:

I didn’t notice, but I did have a full disk just before starting the update, I have since increased the disk size.

Did you try

pihole -r

with Repair already?

Yes tried it several times already.
Basically everytime there is a call to pihole-FTL I have the same error message

The pihole maintenance set of scripts and pihole-FTL are separate things.

How does pihole -r | Repair fail?
Could you share the output for such a run?

Output from the repair


  [βœ“] Root user check

        .;;,.
        .ccccc:,.
         :cccclll:.      ..,,
          :ccccclll.   ;ooodc
           'ccll:;ll .oooodc
             .;cll.;;looo:.
                 .. ','.
                .',,,,,,'.
              .',,,,,,,,,,.
            .',,,,,,,,,,,,....
          ....''',,,,,,,'.......
        .........  ....  .........
        ..........      ..........
        ..........      ..........
        .........  ....  .........
          ........,,,,,,,'......
            ....',,,,,,,,,,,,.
               .',,,,,,,,,'.
                .',,,,,,'.
                  ..'''.

  [βœ“] Update local cache of available packages
  [i] Existing PHP installation detected : PHP version 7.4.21

  [βœ“] Checking apt-get for upgraded packages... up to date!

  [i] Checking for / installing Required dependencies for OS Check...
  [βœ“] Checking for grep
  [βœ“] Checking for dnsutils

  [i] PIHOLE_SKIP_OS_CHECK env variable set to true - installer will continue
  [i] Checking for / installing Required dependencies for this install script...
  [βœ“] Checking for git
  [βœ“] Checking for iproute2
  [βœ“] Checking for whiptail

  [i] SELinux not detected
  [i] Repair option selected
  [i] Performing reconfiguration, skipping download of local repos
  [βœ“] Resetting repository within /etc/.pihole...
  [βœ“] Resetting repository within /var/www/html/admin...
  [i] Checking for / installing Required dependencies for Pi-hole software...
  [βœ“] Checking for cron
  [βœ“] Checking for curl
  [βœ“] Checking for iputils-ping
  [βœ“] Checking for lsof
  [βœ“] Checking for netcat
  [βœ“] Checking for psmisc
  [βœ“] Checking for sudo
  [βœ“] Checking for unzip
  [βœ“] Checking for idn2
  [βœ“] Checking for sqlite3
  [βœ“] Checking for libcap2-bin
  [βœ“] Checking for dns-root-data
  [βœ“] Checking for libcap2
  [βœ“] Checking for lighttpd
  [βœ“] Checking for php7.4-common
  [βœ“] Checking for php7.4-cgi
  [βœ“] Checking for php7.4-sqlite3
  [βœ“] Checking for php7.4-xml
  [βœ“] Checking for php7.4-intl
  [βœ“] Checking for php7.4-json

  [βœ“] Enabling lighttpd service to start on reboot...
  [βœ“] Checking for user 'pihole'

  [i] FTL Checks...

  [βœ“] Detected x86_64 processor
  [i] Checking for existing FTL binary...
  [βœ“] Downloading and Installing FTL
  [βœ“] Installing scripts from /etc/.pihole

  [i] Installing configs from /etc/.pihole...
  [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
  [βœ“] Installed /etc/dnsmasq.d/01-pihole.conf
  [βœ“] Installed /etc/dnsmasq.d/06-rfc6761.conf

  [i] Installing blocking page...
  [βœ“] Creating directory for blocking page, and copying files
  [i] Backing up index.lighttpd.html
      No default index.lighttpd.html file found... not backing up

  [βœ“] Installing sudoer file

  [βœ“] Installing latest Cron script

  [i] Installing latest logrotate script...
        [i] Existing logrotate file found. No changes made.
  [i] Backing up /etc/dnsmasq.conf to /etc/dnsmasq.conf.old
  [βœ“] man pages installed and database updated
  [i] Testing if systemd-resolved is enabled
  [i] Systemd-resolved does not need to be restarted
  [βœ“] Restarting lighttpd service...
  [βœ“] Enabling lighttpd service to start on reboot...
  [i] Restarting services...
  [βœ“] Enabling pihole-FTL service to start on reboot...
  [βœ“] Restarting pihole-FTL service...
  [βœ“] Deleting existing list cache
  [i] Neutrino emissions detected...
  [βœ“] Pulling blocklist source list into range

  [βœ“] Preparing new gravity database
  [i] Using libz compression

  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [βœ“] Status: Retrieval successful
  [i] Analyzed 92995 domains

  [i] Target: https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt
  [βœ“] Status: Retrieval successful
  [i] Analyzed 39602 domains, 39587 domains invalid!
      Sample of invalid domains:
      - 2021-09-14t18:04:54.201z
      - ||videosprofitnetwork.com^
      - ||phhitgjxsit.com^
      - ||mujnmhmfhrubzq.com^
      - ||wovazaix.com^

  [βœ“] Storing downloaded domains in new gravity database
  [βœ“] Building tree
  [βœ“] Swapping databases
  [βœ“] The old database remains available.
  [i] Number of gravity domains: 372040 (93010 unique domains)
  [i] Number of exact blacklisted domains: 0
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 2
  [i] Number of regex whitelist filters: 8
  [βœ“] Cleaning up stray matter

  [βœ—] DNS service is NOT listening

  [i] The install log is located at: /etc/pihole/install.log
Update Complete!

  Current Pi-hole version is v5.4
  Current AdminLTE version is v5.6
  Invalid Option! Try 'pihole -v --help' for more information.

If I download the previous pihole-FTL binary (5.8.1), this one does start successfully.

This doesn't look too bad - seems a current pihole-FTL version was downloaded.

Probably unrelated, but one of your blocklists is not designed for use with Pi-hole:

You should remove that list from Pi-hole's configuration.

And just for completeness:

Debian Sid is the generic name for the current unstable Debian version.
Pi-hole doesn't support unstable OS versions.

Done.

That’s the reason why I posted in Community help, as I know it’s not a supported OS version :wink:

1 Like

Another attempt to solve the issue.
I manually downloaded the 5.9 version and overwrote the binary in /usr/bin

wget https://github.com/pi-hole/FTL/releases/download/v5.9/pihole-FTL-linux-x86_64
chmod +x chmod +x pihole-FTL-linux-x86_64
cp pihole-FTL-linux-x86_64 /usr/bin/pihole-FTL

root@slb-mel-00:~# /usr/bin/pihole-FTL
FTL started!
root@slb-mel-00:~# /usr/bin/pihole-FTL -v
v5.9

But if I try for example to restart the service, the faulty binary is back in place

root@slb-mel-00:~# /usr/bin/pihole-FTL -v
v5.9
root@slb-mel-00:~# systemctl restart pihole-FTL
root@slb-mel-00:~# /usr/bin/pihole-FTL -v
bash: /usr/bin/pihole-FTL: Operation not permitted
root@slb-mel-00:~# cp ./pihole-FTL-linux-x86_64 /usr/bin/pihole-FTL
root@slb-mel-00:~# /usr/bin/pihole-FTL
FTL started!
root@slb-mel-00:~# /usr/bin/pihole-FTL -v
v5.9

You probably won't find many users running Sid.

I think, anyway the issue is not with pihole, but with the system, and more precisely the capabilities :-/

root@slb-mel-00:~# sha256sum /usr/bin/pihole-FTL ./pihole-FTL-linux-x86_64
61abc38131249871ab27bc356d122b26420e667c18a69b459d3e571a84d1750e  /usr/bin/pihole-FTL
61abc38131249871ab27bc356d122b26420e667c18a69b459d3e571a84d1750e  ./pihole-FTL-linux-x86_64

root@slb-mel-00:~# /usr/bin/pihole-FTL
bash: /usr/bin/pihole-FTL: Operation not permitted

root@slb-mel-00:~# ./pihole-FTL-linux-x86_64
FTL started!

root@slb-mel-00:~# getcap /usr/bin/pihole-FTL
/usr/bin/pihole-FTL cap_chown,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_sys_nice=eip

root@slb-mel-00:~# getcap ./pihole-FTL-linux-x86_64

root@slb-mel-00:~#

This is the nature of unsupported, unfortunately.

1 Like

What does ls -lach /usr/bin/pihole-FTL show?

Is the directory mounted on a partition with any restrictions?

Aside of what Dan suggested, did you try to remove that capabilities from the binary? Probably on Sid one or all of those is for some reason not allowed :thinking:. A new security "feature", SELinux/AppArmor-wise or simply a bug with capabilities on current Sid.

Really, I'd suggest to at least stick with "testing" (Bookworm currently) instead of unstable/Sid, which is a highly volatile and untested development suite and also not in a consistent state when it's about dependencies and conflicts :face_with_monocle:.

1 Like

Or even consider running a stable distribution (currently Bullseye) on your production Pi-hole.

root@slb-mel-00:~# ls -lach /usr/bin/pihole-FTL
-rwxr-xr-x 1 root root 15M Sep 14 21:39 /usr/bin/pihole-FTL

root@slb-mel-00:~# mount | grep '/ '
/dev/loop0p1 on / type ext4 (rw,nodev,relatime)

Removing the capabilities does make the binary run.
Modified the init script, not to add them for now, before I migrate my setup to the stable version of Debian.

I agree.

I missed that it is a (systemd) container and hence on a loop device. Is this nodev mount option intentional? I mean usually device nodes are in /dev and hence an own mount, but probably nodev on root mount has still some impact in general or in combination with a container where actual devices are --bind mounted from the host. At least I haven't seen this being used yet.

The nodev option is there by default for the / partition.

Regarding the /dev :

root@slb-mel-00:~# mount | grep ' /dev'
tmpfs on /dev type tmpfs (rw,nosuid,size=4096k,nr_inodes=65536,mode=755,uid=125501440,gid=125501440)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,size=3272168k,nr_inodes=409600,uid=125501440,gid=125501440)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=125501445,mode=620,ptmxmode=666)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)

Below is the /dev content in the container

root@slb-mel-00:~# ls -lahc /dev
total 4.0K
drwxr-xr-x  7 root   root     440 Sep 14 17:42 .
drwxr-xr-x 19 root   root    4.0K Jan 12  2021 ..
drwxr-xr-x  2 root   root     180 Sep 14 17:42 char
lrwxrwxrwx  1 root   root       5 Sep 14 17:42 console -> pts/0
lrwxrwxrwx  1 root   root      11 Sep 14 17:42 core -> /proc/kcore
lrwxrwxrwx  1 root   root      13 Sep 14 17:42 fd -> /proc/self/fd
crw-rw-rw-  1 root   root    1, 7 Sep 14 17:42 full
lrwxrwxrwx  1 root   root      12 Sep 14 17:42 initctl -> /run/initctl
lrwxrwxrwx  1 root   root      28 Sep 14 17:42 log -> /run/systemd/journal/dev-log
drwxrwxrwt  2 nobody nogroup   40 Sep 14 17:42 mqueue
drwxr-xr-x  2 root   root      60 Sep 14 17:42 net
crw-rw-rw-  1 root   root    1, 3 Sep 14 17:42 null
lrwxrwxrwx  1 root   root       8 Sep 14 17:42 ptmx -> pts/ptmx
drwxr-xr-x  2 root   root       0 Sep 14 17:42 pts
crw-rw-rw-  1 root   root    1, 8 Sep 14 17:42 random
drwxrwxrwt  2 root   root     260 Sep 15 20:57 shm
lrwxrwxrwx  1 root   root      15 Sep 14 17:42 stderr -> /proc/self/fd/2
lrwxrwxrwx  1 root   root      15 Sep 14 17:42 stdin -> /proc/self/fd/0
lrwxrwxrwx  1 root   root      15 Sep 14 17:42 stdout -> /proc/self/fd/1
crw-rw-rw-  1 root   root    5, 0 Sep 14 17:42 tty
crw-rw-rw-  1 root   root    1, 9 Sep 14 17:42 urandom
crw-rw-rw-  1 root   root    1, 5 Sep 14 17:42 zero

Also I can see that FTL is successfully using the /dev/shm:

root@slb-mel-00:~# ls -lahc /dev/shm/
total 13M
drwxrwxrwt 2 root root  260 Sep 15 20:57 .
drwxr-xr-x 7 root root  440 Sep 14 17:42 ..
-rw------- 1 root root 348K Sep 14 21:40 FTL-clients
-rw------- 1 root root  240 Sep 14 21:40 FTL-counters
-rw------- 1 root root 104K Sep 15 20:47 FTL-dns-cache
-rw------- 1 root root 132K Sep 15 20:17 FTL-domains
-rw------- 1 root root   88 Sep 14 21:40 FTL-lock
-rw------- 1 root root 8.0K Sep 14 21:40 FTL-overTime
-rw------- 1 root root 4.0K Sep 15 05:10 FTL-per-client-regex
-rw------- 1 root root  12M Sep 15 20:59 FTL-queries
-rw------- 1 root root   12 Sep 14 21:40 FTL-settings
-rw------- 1 root root 200K Sep 15 20:53 FTL-strings
-rw------- 1 root root  20K Sep 14 21:40 FTL-upstreams