Another Disk space question

Help Community Help
1

Details about my system:
This is my debug token

What I have changed since installing Pi-hole:
I just disabled logging today. This is the only change I've done to date.

Thank you for your help

The issue I am facing:
Disk space all used up and I don't know by what
this is the output from df -ah command

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       20G   17G  2.1G  90% /
none            492K  4.0K  488K   1% /dev
proc               0     0     0    - /proc
proc               0     0     0    - /proc/sys/net
proc               0     0     0    - /proc/sys
proc               0     0     0    - /proc/sysrq-trigger
sysfs              0     0     0    - /sys
sysfs              0     0     0    - /sys/devices/virtual/net
fusectl            0     0     0    - /sys/fs/fuse/connections
debugfs            0     0     0    - /sys/kernel/debug
securityfs         0     0     0    - /sys/kernel/security
pstore             0     0     0    - /sys/fs/pstore
mqueue             0     0     0    - /dev/mqueue
efivarfs           0     0     0    - /sys/firmware/efi/efivars
binfmt_misc        0     0     0    - /proc/sys/fs/binfmt_misc
proc               0     0     0    - /dev/.lxc/proc
sys                -     -     -    - /dev/.lxc/sys
none               0     0     0    - /sys/fs/cgroup
lxcfs              0     0     0    - /proc/cpuinfo
lxcfs              0     0     0    - /proc/diskstats
lxcfs              0     0     0    - /proc/loadavg
lxcfs              0     0     0    - /proc/meminfo
lxcfs              0     0     0    - /proc/stat
lxcfs              0     0     0    - /proc/swaps
lxcfs              0     0     0    - /proc/uptime
lxcfs              0     0     0    - /sys/devices/system/cpu/online
udev            7.8G     0  7.8G   0% /dev/full
udev            7.8G     0  7.8G   0% /dev/null
udev            7.8G     0  7.8G   0% /dev/random
udev            7.8G     0  7.8G   0% /dev/tty
udev            7.8G     0  7.8G   0% /dev/urandom
udev            7.8G     0  7.8G   0% /dev/zero
devpts             0     0     0    - /dev/pts
devpts             0     0     0    - /dev/ptmx
devpts             0     0     0    - /dev/console
devpts             0     0     0    - /dev/tty1
devpts             0     0     0    - /dev/tty2
none            492K  4.0K  488K   1% /proc/sys/kernel/random/boot_id
tmpfs           7.8G  764K  7.8G   1% /dev/shm
tmpfs           1.6G  100K  1.6G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock```
2

The command below will look at the size of all folders and display the top 20 largest.

sudo du / 2>/dev/null | sort -rn | head -n 20

(Edited to fix incorrect command)

3

This is the output

`du: cannot read directory '/lost+found': Permission denied
du: cannot read directory '/dev/.lxc/sys/kernel': Permission denied
du: cannot read directory '/dev/.lxc/sys/power': Permission denied
du: cannot read directory '/dev/.lxc/sys/class': Permission denied
du: cannot read directory '/dev/.lxc/sys/devices': Permission denied
du: cannot read directory '/dev/.lxc/sys/dev': Permission denied
du: cannot read directory '/dev/.lxc/sys/hypervisor': Permission denied
du: cannot read directory '/dev/.lxc/sys/fs': Permission denied
du: cannot read directory '/dev/.lxc/sys/bus': Permission denied
du: cannot read directory '/dev/.lxc/sys/firmware': Permission denied
du: cannot read directory '/dev/.lxc/sys/block': Permission denied
du: cannot read directory '/dev/.lxc/sys/module': Permission denied
du: cannot read directory '/dev/.lxc/proc/fs': Permission denied
du: cannot read directory '/dev/.lxc/proc/bus': Permission denied
du: cannot read directory '/dev/.lxc/proc/irq': Permission denied
du: cannot read directory '/dev/.lxc/proc/spl': Permission denied
du: cannot read directory '/dev/.lxc/proc/sys': Permission denied
du: cannot read directory '/dev/.lxc/proc/tty': Permission denied
du: cannot read directory '/dev/.lxc/proc/acpi': Permission denied
du: cannot read directory '/dev/.lxc/proc/scsi': Permission denied
du: cannot read directory '/dev/.lxc/proc/asound': Permission denied
du: cannot read directory '/dev/.lxc/proc/driver': Permission denied
du: cannot read directory '/dev/.lxc/proc/sysvipc': Permission denied
du: cannot read directory '/dev/.lxc/proc/pressure': Permission denied
du: cannot read directory '/dev/.lxc/proc/dynamic_debug': Permission denied
du: cannot read directory '/dev/.lxc/proc/1': Permission denied
du: cannot read directory '/dev/.lxc/proc/49': Permission denied
du: cannot read directory '/dev/.lxc/proc/96': Permission denied
du: cannot read directory '/dev/.lxc/proc/102': Permission denied
du: cannot read directory '/dev/.lxc/proc/103': Permission denied
du: cannot read directory '/dev/.lxc/proc/104': Permission denied
du: cannot read directory '/dev/.lxc/proc/105': Permission denied
du: cannot read directory '/dev/.lxc/proc/110': Permission denied
du: cannot read directory '/dev/.lxc/proc/111': Permission denied
du: cannot read directory '/dev/.lxc/proc/114': Permission denied
du: cannot read directory '/dev/.lxc/proc/146': Permission denied
du: cannot read directory '/dev/.lxc/proc/147': Permission denied
du: cannot read directory '/dev/.lxc/proc/148': Permission denied
du: cannot read directory '/dev/.lxc/proc/154': Permission denied
du: cannot read directory '/dev/.lxc/proc/263': Permission denied
du: cannot read directory '/dev/.lxc/proc/265': Permission denied
du: cannot read directory '/dev/.lxc/proc/272': Permission denied
du: cannot read directory '/dev/.lxc/proc/275': Permission denied
du: cannot read directory '/dev/.lxc/proc/319': Permission denied
du: cannot read directory '/dev/.lxc/proc/320': Permission denied
du: cannot read directory '/dev/.lxc/proc/321': Permission denied
du: cannot read directory '/dev/.lxc/proc/322': Permission denied
du: cannot read directory '/dev/.lxc/proc/323': Permission denied
du: cannot read directory '/dev/.lxc/proc/324': Permission denied
du: cannot read directory '/dev/.lxc/proc/325': Permission denied
du: cannot read directory '/dev/.lxc/proc/326': Permission denied
du: cannot read directory '/dev/.lxc/proc/450': Permission denied
du: cannot read directory '/dev/.lxc/proc/451': Permission denied
du: cannot read directory '/dev/.lxc/proc/452': Permission denied
du: cannot read directory '/dev/.lxc/proc/617': Permission denied
du: cannot read directory '/dev/.lxc/proc/5798': Permission denied
du: cannot read directory '/dev/.lxc/proc/5799': Permission denied
du: cannot read directory '/dev/.lxc/proc/5800': Permission denied
du: cannot read directory '/dev/.lxc/proc/5801': Permission denied
du: cannot read directory '/sys/kernel/debug': Permission denied
du: cannot read directory '/sys/fs/pstore': Permission denied
du: cannot read directory '/sys/fs/fuse/connections/42': Permission denied
du: cannot read directory '/sys/fs/fuse/connections/39': Permission denied
du: cannot read directory '/proc/tty/driver': Permission denied
du: cannot read directory '/proc/acpi/button': Permission denied
du: cannot access '/proc/5801/task/5801/fd/3': No such file or directory
du: cannot access '/proc/5801/task/5801/fdinfo/3': No such file or directory
du: cannot access '/proc/5801/fd/4': No such file or directory
du: cannot access '/proc/5801/fdinfo/4': No such file or directory
1012K /usr/share/perl/5.30.0/CPAN
1012K /usr/lib/python3/dist-packages/DistUpgrade
1008K /lib/systemd/system
996K /usr/lib/x86_64-linux-gnu/perl-base/auto
972K /usr/share/locale/uk
968K /usr/share/locale/uk/LC_MESSAGES
960K /usr/share/locale/fr
960K /usr/share/locale/de
956K /usr/share/locale/fr/LC_MESSAGES
956K /usr/share/locale/de/LC_MESSAGES

4

My apologies, the command I posted was no good. I've edited the post with an improved command that hides the errors and shows raw bytes so the sorting happens correctly and displays the top 20 results.

5

The output this time is this

17452380        /
16612360        /var
16292308        /var/log
15765056        /var/log/pihole
675924  /usr
524340  /var/log/journal
524336  /var/log/journal/61d5dc78c49a4796b74951eadef0dfa2
448928  /usr/lib
337928  /usr/lib/x86_64-linux-gnu
301620  /var/lib
167220  /var/lib/apt
167200  /var/lib/apt/lists
137272  /usr/share
127772  /var/lib/dpkg
92312   /etc
82820   /etc/pihole
77712   /usr/bin
47224   /lib
43020   /usr/lib/x86_64-linux-gnu/dri
27648   /usr/share/locale
6

Your Pi-hole log directory is 15GB. Did you have any special logging enabled? See how it is broken down with the below. That will help work out what to do to clear it out.

ls -l /var/log/pihole
7

I don't have any special logging no.

total 15765048
-rw-r--r-- 1 pihole pihole       22256 Jan 28 19:37 FTL.log
-rw-r--r-- 1 pihole pihole         146 Jan 28 14:57 FTL.log.1
-rw-r--r-- 1 pihole pihole       14792 Jan 28 14:57 FTL.log.2.gz
-rw-r--r-- 1 pihole pihole        4509 Jan 28 14:02 FTL.log.3.gz
-rw-r----- 1 pihole pihole 16095407713 Jan 28 15:01 pihole.log
-rw-r----- 1 pihole pihole    47880601 Jan  7 00:00 pihole.log.6.gz
-rw-r----- 1 root   pihole       46910 Jan 28 15:26 pihole_debug.log
-rw-r--r-- 1 root   root          6041 Jan 22 04:03 pihole_updateGravity.log
8

Your pihole.log file has grown to 16GB in size and that's where all the space has gone. The log file wouldn't normally get this large because the log rotation runs at midnight every 24 hours and moves the log along one, and after 5 days they drop off and are deleted. This is what mine looks like and you can see the way the last 5 days logs are there. At midnight, number 5 will be deleted and 4 will become 5 and so on.

-rw-r----- 1 pihole pihole 1621104 Jan 28 19:24 pihole.log
-rw-r----- 1 pihole pihole 1919616 Jan 28 00:00 pihole.log.1
-rw-r----- 1 pihole pihole 152483 Jan 27 00:00 pihole.log.2.gz
-rw-r----- 1 pihole pihole 172754 Jan 26 00:00 pihole.log.3.gz
-rw-r----- 1 pihole pihole 133010 Jan 25 00:00 pihole.log.4.gz
-rw-r----- 1 pihole pihole 247780 Jan 24 00:00 pihole.log.5.gz

You need to clear the log file to free up the space, and fix the log rotation so it happens automatically from now on. If you don't need the contents of the log files and are happy to delete them, you can delete the stray number 6 log and zero out the huge current log like this, and check they are sorted afterwards:

sudo rm /var/log/pihole/pihole.log.6.gz
sudo truncate -s 0 /var/log/pihole/pihole.log
ls -l /var/log/pihole

To fix the logrotate, check that logrotate is configured and available

cat /etc/pihole/logrotate
grep -v "#" /etc/cron.d/pihole
ls -l /usr/sbin/logrotate
2 Likes
9

As an extra check, what is the output of these commands?

echo ">stats >quit" | nc localhost 4711
echo ">querytypes >quit" | nc localhost 4711
10 
-rw-r--r-- 1 pihole pihole  1157 Jan 29 01:34 FTL.log
-rw-r--r-- 1 pihole pihole 23696 Jan 29 00:00 FTL.log.1
-rw-r--r-- 1 pihole pihole   153 Jan 28 14:57 FTL.log.2.gz
-rw-r--r-- 1 pihole pihole 14792 Jan 28 14:57 FTL.log.3.gz
-rw-r----- 1 pihole pihole     0 Jan 29 01:48 pihole.log
-rw-r----- 1 root   pihole 46910 Jan 28 15:26 pihole_debug.log
-rw-r--r-- 1 root   root    6041 Jan 22 04:03 pihole_updateGravity.log

it looks like this now.

root@Pi-Hole:~# ls -l /usr/sbin/logrotate
-rwxr-xr-x 1 root root 84056 Jan 21  2019 /usr/sbin/logrotate (in green color)

root@Pi-Hole:~# echo ">stats >quit" | nc localhost 4711
domains_being_blocked 362814
dns_queries_today 48137
ads_blocked_today 4504
ads_percentage_today 9.356628
unique_domains 2680
queries_forwarded 26197
queries_cached 16543
clients_ever_seen 64
unique_clients 64
dns_queries_all_types 48137
reply_UNKNOWN 985
reply_NODATA 4586
reply_NXDOMAIN 5295
reply_CNAME 22845
reply_IP 14090
reply_DOMAIN 48
reply_RRNAME 12
reply_SERVFAIL 105
reply_REFUSED 0
reply_NOTIMP 0
reply_OTHER 0
reply_DNSSEC 0
reply_NONE 0
reply_BLOB 171
dns_queries_all_replies 48137
privacy_level 0
status enabled
11

That date is a few years off.

12

Log files grow proportionally with the number of DNS requests as received by Pi-hole.
Pi-hole's log rotation happens on a daily patttern - it does not put any restrictions on file size. In order to grow to that kind of multi-gigabyte size, Pi-hole must have seen several millions of DNS requests.

It's still true that Pi-hole's log file wouldn't normally get this large - because you don't normally see millions of DNS requests in a home network with perhaps a few dozens of devices.

Removing the large log file will cure the symptom, but you may want to keep an eye open on why there have been excessive DNS requests in the first place. A DNS loop of sorts would be the prime suspect for causing them, but I see no indications for that in your debug log.

However, jfb's observation coincides with a few lines from your debug log:

*** [ DIAGNOSING ]: Pi-hole log

-rw-r----- 1 pihole pihole 15G Jan 28 15:01 /var/log/pihole/pihole.log
   -----head of pihole.log------
   Jan  7 00:00:41 dnsmasq[478]: query[A] time.nist.gov from 192.168.0.145
   Jan  7 00:00:41 dnsmasq[478]: config error is REFUSED (EDE: blocked)
   Jan  7 00:00:41 dnsmasq[478]: Rate-limiting time.nist.gov is REFUSED (EDE: blocked)
   Jan  7 00:00:41 dnsmasq[478]: query[A] time.nist.gov from 192.168.0.225
   Jan  7 00:00:41 dnsmasq[478]: config error is REFUSED (EDE: blocked)
   Jan  7 00:00:41 dnsmasq[478]: Rate-limiting time.nist.gov is REFUSED (EDE: blocked)
   Jan  7 00:00:41 dnsmasq[478]: query[A] pool.ntp.org from 192.168.0.145

This indicates that some clients are desperate to get accurate time information, to the extent that their requests got rate-limited by Pi-hole).

As you use are using unbound, an accurate time would also be absolutely required for the machine running your unbound and Pi-hole.

You should verify that your Pi-hole's time and time zone information are correct.

In addition, your router is distributing its own IP as local DNS server via DHCP next to Pi-hole's:

*** [ DIAGNOSING ]: Discovering active DHCP servers (takes 10 seconds)
   Scanning all your interfaces for DHCP servers
   
   * Received 300 bytes from eth0:192.168.0.1
     Offered IP address: 192.168.0.72
     DHCP options:
      Message type: DHCPOFFER (2)
      dns-server: 192.168.0.72
      dns-server: 192.168.0.1
      router: 192.168.0.1
      --- end of options ---

This would allow your clients to by-pass Pi-hole via 192.168.0.1 at their own discretion.
Pi-hole has to be the sole DNS server for your clients.

1 Like
13

Apparently, ntp serice is off. I don't know how that happened.
how do I enable it?

root@Pi-Hole:~# timedatectl status
               Local time: Sun 2023-01-29 13:32:52 UTC
           Universal time: Sun 2023-01-29 13:32:52 UTC
                 RTC time: n/a                        
                Time zone: Etc/UTC (UTC, +0000)       
System clock synchronized: yes                        
              NTP service: inactive                   
          RTC in local TZ: no                         
root@Pi-Hole:~# timedatectl timesync-status
Failed to query server: Connection timed out
14

status looks OK to me.

I wouldn't expect you to run an NTP server. It's not required.

Your above output shows that your host machine's NTP client s/w has successfully synced time:

Is your time zone indeed UTC+0 ?
Switch it if it isn't.

15

no, its EST
Got it corrected.

root@Pi-Hole:~# timedatectl status
               Local time: Sun 2023-01-29 09:36:47 EST             
           Universal time: Sun 2023-01-29 14:36:47 UTC             
                 RTC time: n/a                                     
                Time zone:  (EST, -0500)
System clock synchronized: yes                                     
              NTP service: inactive                                
          RTC in local TZ: no

Thank you so much @Bucking_Horn , @chrislph for all your help

17

You too @jfb

the logrotate command still returns wrong date however.

root@Pi-Hole:~# ls -l /usr/sbin/logrotate
-rwxr-xr-x 1 root root 84056 Jan 21  2019 /usr/sbin/logrotate

Anyway to correct it or will it correct itself later on now that the correct time zone is correct?

18

Check the permissions and contents of the Pi-hole logrotate setup

ls -l /etc/pihole/logrotate
cat /etc/pihole/logrotate

Check it's scheduled in cron

grep -v "#" /etc/cron.d/pihole

This will show the version installed. My latest Pi OS is logrotate 3.18.0

/usr/sbin/logrotate --version

and this will help confirm there's no unusual DNS usage

echo ">querytypes >quit" | nc localhost 4711

How does your Pi-hole log directory look now since it was cleaned? Has it grown a lot in the hours that have passed?

ls -l /var/log/pihole

If you run that command every day you should see the log files being bumped along one each day, the same way mine looked above.

Check to see what NTP activity might still be happening by looking for the 20 most recent NTP lines in there. Are they from right now, or perhaps they have even stopped.

sudo grep -i ntp /var/log/pihole/pihole.log | tail -n 20
20 
ls -l /etc/pihole/logrotate
cat /etc/pihole/logrotate

root@Pi-Hole:~# ls -l /etc/pihole/logrotate
-rw-r--r-- 1 root root 245 Jul 17  2022 /etc/pihole/logrotate
root@Pi-Hole:~# cat /etc/pihole/logrotate
/var/log/pihole/pihole.log {
        su root syslog
        daily
        copytruncate
        rotate 5
        compress
        delaycompress
        notifempty
        nomail
}

/var/log/pihole/FTL.log {
        su root syslog
        weekly
        copytruncate
        rotate 3
        compress
        delaycompress
        notifempty
        nomail
}

grep -v "#" /etc/cron.d/pihole :

root@Pi-Hole:~# grep -v "#" /etc/cron.d/pihole

33 3   * * 7   root    PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updateGravity >/var/log/pihole/pihole_updateGravity.log || cat /var/log/pihole/pihole_updateGravity.log

00 00   * * *   root    PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole flush once quiet

@reboot root /usr/sbin/logrotate --state /var/lib/logrotate/pihole /etc/pihole/logrotate

54 19  * * *   root    PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updatechecker
@reboot root    PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updatechecker reboot

/usr/sbin/logrotate --version :

root@Pi-Hole:~# /usr/sbin/logrotate --version
logrotate 3.14.0

    Default mail command:       /usr/bin/mail
    Default compress command:   /bin/gzip
    Default uncompress command: /bin/gunzip
    Default compress extension: .gz
    Default state file path:    /var/lib/logrotate/status
    ACL support:                yes
    SELinux support:            yes

ls -l /var/log/pihole:

 root@Pi-Hole:~# ls -l /var/log/pihole
total 18716
-rw-r--r-- 1 pihole pihole    74769 Jan 29 10:09 FTL.log
-rw-r--r-- 1 pihole pihole    23696 Jan 28 19:00 FTL.log.1
-rw-r--r-- 1 pihole pihole      153 Jan 28 09:57 FTL.log.2.gz
-rw-r--r-- 1 pihole pihole    14792 Jan 28 09:57 FTL.log.3.gz
-rw-r----- 1 pihole pihole 18974485 Jan 29 10:16 pihole.log
-rw-r----- 1 root   pihole    46910 Jan 28 10:26 pihole_debug.log
-rw-r--r-- 1 root   root       5640 Jan 28 22:33 pihole_updateGravity.log

sudo grep -i ntp /var/log/pihole/pihole.log | tail -n 20 :

Jan 29 10:14:24 dnsmasq[293]: cached ntp1.glb.nist.gov is 
Jan 29 10:15:16 dnsmasq[293]: query[A] ntp-g7g.amazon.com from 
Jan 29 10:15:16 dnsmasq[293]: cached ntp-g7g.amazon.com is 
Jan 29 10:15:37 dnsmasq[293]: query[A] pool.ntp.org from 
Jan 29 10:15:37 dnsmasq[293]: forwarded pool.ntp.org to 127.0.0.1#5335
Jan 29 10:15:37 dnsmasq[293]: reply pool.ntp.org is 
Jan 29 10:15:37 dnsmasq[293]: reply pool.ntp.org is 
Jan 29 10:15:37 dnsmasq[293]: reply pool.ntp.org is 
Jan 29 10:15:37 dnsmasq[293]: reply pool.ntp.org is 
Jan 29 10:15:39 dnsmasq[293]: query[A] pool.ntp.org from 
Jan 29 10:15:39 dnsmasq[293]: cached pool.ntp.org is 
Jan 29 10:15:39 dnsmasq[293]: cached pool.ntp.org is 
Jan 29 10:15:39 dnsmasq[293]: cached pool.ntp.org is 
Jan 29 10:15:39 dnsmasq[293]: cached pool.ntp.org is 
Jan 29 10:16:39 dnsmasq[293]: query[A] 0.nl.pool.ntp.org from 
Jan 29 10:16:39 dnsmasq[293]: forwarded 0.nl.pool.ntp.org to 127.0.0.1#5335
Jan 29 10:16:39 dnsmasq[293]: reply 0.nl.pool.ntp.org is 
Jan 29 10:16:39 dnsmasq[293]: reply 0.nl.pool.ntp.org is 
Jan 29 10:16:39 dnsmasq[293]: reply 0.nl.pool.ntp.org is 
Jan 29 10:16:39 dnsmasq[293]: reply 0.nl.pool.ntp.org
21

That all looked okay; the logrotate version is slightly old, when was the OS last updated?

There's a lot of NTP-related activity, has that settled down with the timezone now tweaked?

sudo grep -i ntp /var/log/pihole/pihole.log | tail -n 20

Now that a couple of days have passed, how does that log directory look now? Are the logs being rotated, and are the files growing large?

ls -l /var/log/pihole
1 Like
22

[Off topic] Just come here to say thank you. The original post was closed, but your step-by-step debugging process also helped me resolve my raspbian/pihole issue. Salute.