Pihole down after 4.0 fresh install

This is very strange. I don't recall having seen this before. Pinging @DanSchaper and @Mcat12 to see if they know more about this.

This is on a fresh openvz vps, with Debian 9
With a rather old kernel though 2.6.32-042stab127.2

Since its openvz, i cannot upgrade it

I tried googling some stuff for OpenVZ and sudo but to no avail. Can you post the output of mount ?

edit I do found that Operation not permitted may indeed be a OpenVZ triggered message as it does say the same if you, e.g., want to enable swap or set the global date of the machine from within the virtual container.

  • Does sudo /usr/bin/pihole-FTL debug output anything? If not, we know at least that you container doesn't even try to start the binary...
  • Please copy the binary into your home folder and try to sudo-run it there. Does it still give the same message?

Strang thing is that sometimes i can start ftl as root, right now i cannot. Operation not permitted
Ill try in home

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Aug 20 12:03:29 2018 from 89.205.224.31
root@vps4:~# mount
/dev/ploop20076p1 on / type ext4 (rw,relatime,barrier=1,data=ordered,balloon_ino=12)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755,gid=500)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio,name=beancounter)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=26212k,mode=700)
root@vps4:~#

Okay, nothing strange in your mount output (I was looking to a line mounting /usr with some strange options, but that's not there).

Makes me think more and more that this is only a VM issues we maybe cannot actually solve...

But why did 3.2/3.3 run flawlessly then?

Manually starting as root is working though(./pihole-FTL), i have it up and running now

Versions before v4.0 used dnsmasq as DNS server. From v4.0 on, we include dnsmasq in our pihole-FTL and, hence, our daemon requires more privileges (such as binding to protected ports such as 53, etc.).

Where did you start it? In /usr/bin or a copy in some other directory? It may be that the OpenVZ virtualization restricts /usr/bin somehow such that external applications are not allowed there...

How can i checkout with V3.3?

I run it outside /user/bin
In my root dir now

@sander_kerkhoff

I don't think you should use sudo when you are already root. You use sudo or su when you're a regular user that needs root permissions. Using sudo when already root might not be kosher (a quick google sounds like it should be fine, people do it all the time regardless of root from muscle memory).

It's not going to be ALL openvz VPS, but perhaps something the host provider may need to enable/add permissions for case by case? On an unrelated openvz thing a few weeks ago, at another provider's VPS, some iptables stuff wasn't working (that worked fine at another openvz provider) and I had to open a service ticket. I told them what I was doing and what the iptables error was. They quickly enabled the permissions in the host and iptables would work again. That might have been kernel modoles for networking or something along those lines.

I have an openvz on ImpactVPS. Upgraded to v4 today after using v3.3 FTL branch and appears to still be working fine.

[root@pi etc]# hostnamectl
Static hostname: pi.example.net
Icon name: computer-container
Chassis: container
Machine ID: 60f02f4c0c4d47e68bae5bbf1b35ece7
Boot ID: f8526c4588894e0ab52b4edd1a29f36e
Virtualization: openvz
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 2.6.32-49-pve
Architecture: x86-64

Note, I only use root user on centos. If root gets "operation not permitted", I'd bet a dozen doughnuts that its an openvz permission issue that can only be solved by the host admin (your VPS provider). I would open a ticket, tell them about the errors you're getting, such as:
"systemd[1]: pihole-FTL.service: Failed to set invocation ID on control group /system.slice/pihole-FTL.service"

Chances are, they've run into those key words and will know what to do.

@DL6ER What commands can I run on my openvz vps to compare against sander_kerkhoff and confirm its lack of linux capabilities permissions/support? Since the alternative service file you asked him to try didn't work, that might mean his problem is NOT linux capabilities and something else. Has your code for starting as root without capabilities only been code reviewed so far, or it was tested and confirmed fixed the Ubuntu 16.04 server mentioned in one of the report problem posts?

I don't get an error when running setcap on my openvz centos 7. @sander_kerkhoff probably will get the operation not permitted. He can probably just open a service ticket with his provider asking for the right permissions to run the command below.

[root@pi /]# getcap "$(which pihole-FTL)"
/usr/bin/pihole-FTL = cap_net_bind_service,cap_net_admin,cap_net_raw+eip
[root@pi /]# setcap CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_NET_ADMIN+eip "$(which pihole-FTL)"
[root@pi /]#

Has this problem been observed on openvz on Centos 7 also, or just Ubuntu and Debian?

Well, or just something on top. The more reports I see here, the more I start thinking that most - if not all virtualizations, except for Docker - do not support the great features of Linux capabilities. Admittedly, only few applications actually really utilize them (probably because they are not super straightforward), but security related software should be concerned with a minimal set of capabilities for their applications.

It has been reported several times to have solved the seen issues with

but I did not create a list of operating systems / environments where it has helped.

Hmm, I'd suggest to do exactly what you already did

i switched over from openvz to a kvm vps.
I had pihole running again in 5 minutes.

If a dev @DL6ER wants to try ot things on my openvz, pm me. I have no use for it at this time

1 Like

Thanks for the offer but I don't think we have the manpower to start supporting OpenVZ anytime soon. I'm happy to implement some things like the starting as root if setcap failed, but currently my workload is too high to start any new endeavors.

@DL6ER just to close the loop. I reinstalled pihole on a fresh debian vm. I immediately switched to the latest dev branch, which includes all your recent fixes as of Aug 21, and so far no issues. Only thing I noticed is that after initial install, and a reboot, the block list was empty. So I had to update gravity again manually.

1 Like

Actually, that getcap/setcap doesn't seem to matter (red herring). I have an openvz box that started off with 3.3 or earlier and was upgraded to v4. It continued to work.

Today, I reinstalled centos 7 on that VPS that was previously working on v4. Since the installer doesn't support openvz, the script failed part way through. While looking at what to edit in the basic-install.sh (ie, don't call setStaticIPv4()) , I thought I could just put the correct parameters in the setupVars.conf and try for an automated install and skip the bad setting static IP code.

Well, the automated install finished, but ran into the problem where DNS doesn't start and I'm getting "-bash: /usr/bin/pihole-FTL: Operation not permitted".

So I do think that a fresh install of v4 is doing something new/different than an install of v3.3 upgraded to v4.

Edit: Actually, it does look like the capabilities. If I run your pihole-FTL-linux-x86_64 in /usr/bin, then it starts. But if I reboot, something is setting the caps again and breaks it. So that might be why other users might not have had success.

I would expect the setcap to happen during install, one time, not during service start. That just seems wrong place.

But ugh, I think I'll need to figure out how to get a working older version and just leave it at that. Just sucks that installing and using pihole has only gotten more difficult over time instead of easier. But I guess that comes with adding more features.

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.