Upgrade error when restarting pihole-FTL service (free disk space issue)

General Beta 5.0
1

Hi,

Version 4 works like a charm. Thanks to all developers.

Today I saw there was a Beta 5 version so let's try.
Upgrade from latest 4 to 5 stops at
Restarting pihole-FTL service

Back to version 4 now by restoring from backup
What can I do to upgrade?

Debug Token:
I think I can't provide one because upgrade fails and pihole does not resolve names.

System:
VM (Debian Buster amd64)
Debian up to date except security updates because of an error about repository not being signed.
Pi-hole Version v4.3.2 Web Interface Version v4.3.2 FTL Version v4.3.1

2

Which commands did you use to change to beta 5.0, and on which of them did this error occur?

3

From ' Announcing a Beta test of Pi-hole 5.0!' and the template of a new topic.

apt-get update
apt-get dist-upgrade

echo "release/v5.0" | sudo tee /etc/pihole/ftlbranch
pihole checkout core release/v5.0
pihole checkout web release/v5.0

At the '...checkout core...' command the upgrade stops.

4

Let's take a look and see if there are any related details in the install log. Please post the output of these commands from the Pi terminal:

ls -lha /etc/pihole/install.log

cat /etc/pihole/install.log

5

Before I will start the upgrade: Is this command correct?
The file is empty.
'echo "release/v5.0" > /etc/pihole/ftlbranch' puts the text in the file.

6 
-rw-r--r-- 1 root root 1.1K Jan 22 22:13 /etc/pihole/install.log

  [✓] Installing scripts from /etc/.pihole

  [i] Installing configs from /etc/.pihole...
  [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
  [✓] Copying 01-pihole.conf to /etc/dnsmasq.d/01-pihole.conf

  [i] Installing blocking page...
  [✓] Creating directory for blocking page, and copying files
  [i] Backing up index.lighttpd.html
      No default index.lighttpd.html file found... not backing up

  [✓] Installing sudoer file

  [✓] Installing latest Cron script

  [✓] Installing latest logrotate script
  [i] Backing up /etc/dnsmasq.conf to /etc/dnsmasq.conf.old
  [✓] man pages installed and database updated
7

Yes. This is the first of three commands you run to move all parts of Pi-Hole to the 5.0 beta branch.

8

What is the purpose of this command? After this command the file is empty.
I changed this command (see previous post) to put the text in the file.

Now upgrading after deleting self scripted backups.
Disk was full.
It is now putting (big) blocklists in db tables.

Why did the 'Disk space check' pass while the disk was full?
I created a symbolic link from /etc/pihole/pihole-FTL.db to pihole-FTL.db on a second disk.

9

This moves FTL to the release 5.0 branch, and it is expected that there is no output from the command. This is a method of minimizing the number of branches to download, since this will download with one of the subsequent commands.

This is a problem. What does df -ah show for disk space?

I don't know. I would have to look at the code or have one of the devs answer this.

This file is not where the blocklists are being stored. They are in /etc/pihole/gravity.db
What is the output of the following:

ls -lha /etc/pihole

10 
Filesystem      Size  Used Avail Use% Mounted on
sysfs              0     0     0    - /sys
proc               0     0     0    - /proc
udev            468M     0  468M   0% /dev
devpts             0     0     0    - /dev/pts
tmpfs            97M  4.2M   93M   5% /run
/dev/vda1       1.9G  1.6G  261M  86% /
securityfs         0     0     0    - /sys/kernel/security
tmpfs           482M  116K  482M   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           482M     0  482M   0% /sys/fs/cgroup
cgroup2            0     0     0    - /sys/fs/cgroup/unified
cgroup             0     0     0    - /sys/fs/cgroup/systemd
pstore             0     0     0    - /sys/fs/pstore
bpf                0     0     0    - /sys/fs/bpf
cgroup             0     0     0    - /sys/fs/cgroup/rdma
cgroup             0     0     0    - /sys/fs/cgroup/pids
cgroup             0     0     0    - /sys/fs/cgroup/net_cls,net_prio
cgroup             0     0     0    - /sys/fs/cgroup/blkio
cgroup             0     0     0    - /sys/fs/cgroup/perf_event
cgroup             0     0     0    - /sys/fs/cgroup/cpuset
cgroup             0     0     0    - /sys/fs/cgroup/memory
cgroup             0     0     0    - /sys/fs/cgroup/freezer
cgroup             0     0     0    - /sys/fs/cgroup/cpu,cpuacct
cgroup             0     0     0    - /sys/fs/cgroup/devices
systemd-1          -     -     -    - /proc/sys/fs/binfmt_misc
debugfs            0     0     0    - /sys/kernel/debug
mqueue             0     0     0    - /dev/mqueue
hugetlbfs          0     0     0    - /dev/hugepages
/dev/vdb1        47G  7.7G   37G  18% /data
/dev/vdb2       2.5G   59M  2.3G   3% /var/log
tmpfs            97M     0   97M   0% /run/user/1000
binfmt_misc        0     0     0    - /proc/sys/fs/binfmt_misc

/data was 100% now 18%

total 148M
drwxrwxr-x  3 pihole pihole 4.0K Jan 23 01:01 .
drwxr-xr-x 97 root   root   4.0K Jan 22 23:49 ..
-rw-r--r--  1 root   root   2.0K Jan 22 14:30 black.list
-rw-r--r--  1 pihole pihole    0 Jan 22 23:49 dhcp.leases
-rw-r--r--  1 root   root    596 Jan 22 23:49 dns-servers.conf
-rw-r--r--  1 pihole pihole   13 Jan 22 23:36 ftlbranch
-rw-r--r--  1 root   root     20 Jan 22 22:32 GitHubVersions
-rw-rw-r--  1 pihole pihole  14M Jan 23 01:01 gravity.db
-rw-rw-r--  1 pihole pihole  13K Jan 23 01:01 gravity.db-journal
-rw-r--r--  1 root   root    67M Jan 19 04:23 gravity.list
-rw-r--r--  1 root   root   1.1K Jan 22 23:49 install.log
-rw-r--r--  1 root   root    66M Jan 22 23:50 list.7.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root     32 Jan 23 01:00 localbranches
-rw-r--r--  1 root   root    124 Jan 19 04:23 local.list
-rw-r--r--  1 root   root     51 Jan 23 01:00 localversions
-rw-r--r--  1 root   root    234 Jan 22 23:49 logrotate
-rw-r--r--  1 pihole pihole 2.3M Jan 22 23:49 macvendor.db
drwxr-xr-x  2 root   root   4.0K Jan 22 23:50 migration_backup
-rw-rw-r--  1 pihole pihole   15 Jan 22 23:49 pihole-FTL.conf
lrwxrwxrwx  1 root   root     26 Jan 28  2019 pihole-FTL.db -> /data/pihole/pihole-FTL.db
-rw-r--r--  1 root   root   3.2K Jan 22 23:49 setupVars.conf
-rw-r--r--  1 root   root    323 Aug 13  2018 setupVars.conf.open
-rw-r--r--  1 root   root   3.2K Jan 22 23:49 setupVars.conf.update.bak
-rw-r--r--  1 root   root    321 Aug 13  2018 setupVars.conf.vpn
12

Are you able to upload a debug log now? If so, please upload and post the token here.

13

Looks like the root partition has space. This would pass the disk space check.

15

@jfb
No, sorry. When the upgrade failed I couldn't upload.
Then I saw the disk was full. Solved that.
So uploading debug info when upgrade is busy without errors makes no sense imo.

Now pihole is upgrading
[i] Adding adlist with ID 8 to database table...
(big list)

16

A 2Gig root partition vda1 is going to be really tight. Even after offloading /data to a separate disk vdb1, as we don't put anything in that directory. Logs going to vdb2 should be plenty though.

17

@DanSchaper
This was the problem. The disk mounted at /data was full. Now 18%
When the second disk (/data) was full the 'Disk space check' passed and 'Restarting FTL service' failed.
Probably because pihole-FTL.db is on the disk that had no free space.

18

We only check the root partition, and since we don't put any data in /data, we'd have no reason to check that. I don't think we'd even know it existed.

Edit: That's symlink'd so we really would have no knowledge that it was moved.

lrwxrwxrwx 1 root root 26 Jan 28 2019 pihole-FTL.db -> /data/pihole/pihole-FTL.db
19

@DanSchaper
Thanks for your answer. Fair enough only / is checked.
I don't know all my custumizations by heart and should have known before upgrading to v5.0.

After upgrade Pihole isn't upgraded.
I will open a new topic.

Update:
After reading Installation beta - #5 by Bat I understand the version number is not 5 but 4.3.2
Is my pihole installation up to date now?

[i] Checking for updates...
[i] Pi-hole Core: up to date
[i] Web Interface: up to date
[i] FTL: up to date
[i] Warning: You are using FTL from a custom branch (release/v5.0) and might be missing future releases.

[✓] Everything is up to date!

Pi-hole version is v4.3.2-386-g29f06a4 (Latest: v4.3.2)
AdminLTE version is v4.3.2-357-g3cac308e (Latest: v4.3.2)
FTL version is vDev-021cd83 (Latest: v4.3.1)

20

Yes, you are up to date. The version numbers show that you are on a version that is 386 commits greater than 4.3.2. The version number is based on git tags and that isn't set stable during a beta test. I know that is kind of confusing but it's a limitation of how we currently distribute the software.

2 Likes