Implement logrotation for /var/log/pihole.log

Feature Requests Implemented
1

This idea came to me when I was working on this topic. Since log rotation is part of almost all linux distros, I checked if this could be used, ...
Solution:

  • Create /etc/.pihole/advanced/logrotate (the instructions for the rotation)
    /var/log/pihole.log {
    daily
    copytruncate
    rotate 1
    nocompress
    }
  • Modify /opt/pihole/piholeLogFlush.sh
    echo -n "::: Rotating /var/log/pihole.log ..."
    **#**echo " " > /var/log/pihole.log
    /usr/sbin/logrotate --force /etc/.pihole/advanced/logrotate
    echo "... done!"

considerations:

  • There aren't many log rotate directives, as they will be ignored by --force
  • The log will be available for the next 24 hours (rotate 1)
  • I opted for nocompress, this way the log remains readable, without additional actions
  • I tested various options to initialize the log after rotation, copytruncate seems to work best. This implies the log must exist! I tested this with a postrotate script (echo " " > /var/log/pihole.log) but that doesn't work, the new log remains empty.
  • You can't use the systems default logrotation scheme, since daily logrotation occurs at 06h25.

tested on pihole v.2.11.2, pihole flush aswell as cron

the developers could make yesterdays log available on the web interface by parsing the log /var/log/pihole.log.1

2

We are actually considering implementing this like for this quite some time now. However, we have to ensure first that it will work nicely on all platforms we support, which is quite a time-consuming task. And it got a little out of focus while working on FTL.

Let me ask you some questions about your precise suggestion:

If the user flushes it twice in a row, the log from the "day before" is gone, I suppose (that is how I'd expect it to work)?

What happens otherwise?

We could add missingok in case.

Timing might actually be an issue. We could do it with cron at midnight. What if the Pi is not running at midnight? Of course, there is the @reboot statement with cron: do you know if there is an option to do the logrotation only if it has not been done today? I hope you get what I mean. Pi-hole's not running 24h are seeing this issue and it would be nice if this approch would provide an elegant way to mitigate that problem.

3
4

The answer to your questions:

  • If the user flushes twice in a row, the log from the "day before" is gone: This is no different from the current situation, the current pihole flush command removes all the logging. I might even suggest to remove the 'flush' button from the web interface...

  • This implies the log must exist: If the log does not exist, it cannot be rotated and no new log will be created. Adding missingok doesn't solve this, it just suppresses the error message, witch I would not advise, an error message indicates something is wrong.
    To overcome the problem of a non existing log, add this to /opt/pihole/piholeLogFlush.sh. Tested this, works fine.
    /usr/sbin/logrotate --force /etc/.pihole/advanced/logrotate
    if [ ! -f /var/log/pihole.log ]; then
    echo " " > /var/log/pihole.log
    service dnsmasq restart
    fi
    Warning: I choose copytruncate because logging didn't resume using other options (always had to restart dnsmasq). My log is about 600Kb in size, I've got no idea what will happen if the log is several Mb in size. This should be tested!

  • Time might actually be an issue: I've modified /opt/pihole/piholeLogFlush.sh, witch is already triggered by cron at midnight (pihole flush). If the pi isn't running 24/7, there is no difference from the current situation e.g. no rotation.
    To my knowledge, there is no logrotate option to catch up missed events. You can find all logrotate options here.

  • I would advise against the @reboot cron option, als the log will be rotated at every reboot, witch I (and others) often do when I'm testing things. You could consider implementing @reboot (you need to add the root user - tested this) with a script that detects a non rotated log and rotate it only if this is detected. Example:
    line=$(head -n 1 /var/log/pihole.log)
    date=$(date +"%b %d")
    if [[ ${line} != *$date* ]];then
    /usr/sbin/logrotate --force /etc/.pihole/advanced/logrotate
    fi

  • I think the notifempty (saw this on github) directive is ignored by the command line force option.

  • The nomail instruction doesn't do anything, if nothing is configured, the mail you can recieve (see chapter 4.8 of this manual) comes from the cron daemon.

  • rotate 5 may be over the top. Even most system logs (see /etc/rotate.d/rsyslog) are rotated 4 times only.

  • You should probably check the precense of the logrotate package in install.sh

to summarize all this (tested)
/opt/pihole/piholeLogFlush.sh should look like this
#!/usr/bin/env bash
# Pi-hole: A black hole for Internet advertisements
# (c) 2015, 2016 by Jacob Salmela
# Network-wide ad blocking via your Raspberry Pi
# http://pi-hole.net
# Flushes /var/log/pihole.log
#
# Pi-hole is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 2 of the License, or
# (at your option) any later version.line=$(head -n 1 /var/log/pihole.log)

# detect the reboot parameter from cron
if [[ $@ == **reboot** ]];then
date=$(date +"%b %d")
if [[ ${line} != *$date* ]];then
/usr/sbin/logrotate --force /etc/.pihole/advanced/logrotate
fi
else
echo "::: Rotating /var/log/pihole.log ..."
/usr/sbin/logrotate --force /etc/.pihole/advanced/logrotate
echo "::: ... done!"
fi
#catch a non existing log
if [ ! -f /var/log/pihole.log ]; then
echo " " > /var/log/pihole.log
service dnsmasq restart
fi

/etc/cron.d/pihole should look like this:
# Pi-hole: A black hole for Internet advertisements
# (c) 2015, 2016 by Jacob Salmela
# Network-wide ad blocking via your Raspberry Pi
# http://pi-hole.net
# Updates ad sources every week
#
# Pi-hole is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 2 of the License, or
# (at your option) any later version.
#
# This file is under source-control of the Pi-hole installation and update
# scripts, any changes made to this file will be overwritten when the softare
# is updated or re-installed. Please make any changes to the appropriate crontab
# or other cron file snippets.

# Pi-hole: Update the ad sources once a week on Sunday at 01:59
# Download any updates from the adlists
59 1 * * 7 root PATH="$PATH:/usr/local/bin/" pihole updateGravity

# Pi-hole: Update Pi-hole! Uncomment to enable auto update
#30 2 * * 7 root PATH="$PATH:/usr/local/bin/" pihole updatePihole

# Pi-hole: Flush the log daily at 00:00 so it doesn't get out of control
# Stats will be viewable in the Web interface thanks to the cron job above
00 00 * * * root PATH="$PATH:/usr/local/bin/" pihole flush >/dev/null

@reboot root /opt/pihole/piholeLogFlush.sh reboot

notice the reboot parameter on the @reboot line, it makes piholeLogFlush check the date of the pihole log (first line)
notice the redirection on the midnight (00 00) line, a mail will only be sent if somethings wrong.

/etc/.pihole/advanced/logrotate should look like this:
/var/log/pihole.log {
daily
copytruncate
rotate 4
compress
delaycompress
nomail
}

to make life easy for you, I've made an online archive with the required files.

5

I see you edited the post 12 times (last time 7min ago). I've read this version, so please make anew post if you implement further changes

6

I have a log that is several MB in size (about one dozen quite active clients). It works without problems. dnsmasq keeps the log file open while running and therefore if you move the file it will happily write into the moved file instead of the new one (as the pointer to the virtual memory location does not change and renaming does not change where the file is physically located but only its name).

I tested this already (you can see it is in the code of the pull request): The log has been flushed at midnight. When I reboot, logrotate states that the file is not older than one day and simply skips the rotation procedure without having us to worry about it. Tried with three reboots within an hour - no rotation happened for me - did you see something different that would require implementing the changes you suggested (with checking the date)?

EDIT: You can see this when running the command manually with the verbose option:

# /usr/sbin/logrotate -v /etc/.pihole/advanced/logrotate
reading config file /etc/.pihole/advanced/logrotate

Handling 1 logs

rotating pattern: /var/log/pihole.log  after 1 days (5 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/pihole.log
  log does not need rotating

(note that the log was several hundred KB is size at this time, so the line about empty log files is only informative)

7

I don't think your @reboot line in cron is doing anything. I've tested this, the way your github modifications specify and get a log entry (error - syslog). If you add the user (root), the log is flushed when the system is rebooted, actually no different from pihole flush.
The command line --force option actually instructs logrotate to rotate, no matter what the conditions are.
To test this, I used yesterdays log and simply copied it into todays log, I think I've covered all scenarios, even a non existing log (that should never occur).

@reboot root /opt/pihole/piholeLogFlush.sh reboot

8

The line --force is not there!

Screenshot at 2017-01-13 11-30-25.png992×242 21.5 KB

9

Sorry, I missed that, will need to test again, however, NOT seeing a username there, I'm getting an error
Jan 13 10:22:55 raspberrypi cron[368]: Error: bad username; while reading /etc/cron.d/pihole
Jan 13 10:22:55 raspberrypi cron[368]: (systempihole) ERROR (Syntax error, this crontab file will be ignored)
Jan 13 10:22:55 raspberrypi cron[368]: (CRON) INFO (Running @reboot jobs)

10

Okay, so both of us have found something. It seems that I really have to do add the user there for Raspbian (that is not necessary under Ubuntu, where it defaults to root).

For debugging I changed the cron command to:

@reboot root /usr/sbin/logrotate -v /etc/.pihole/advanced/logrotate > /home/pi/logrotate 2>&1

That works because:

# cat /home/pi/logrotate
reading config file /etc/.pihole/advanced/logrotate

Handling 1 logs

rotating pattern: /var/log/pihole.log  after 1 days (5 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/pihole.log
  log does not need rotating

From syslog:

Jan 13 11:53:45 raspberrypi CRON[495]: (root) CMD (/usr/sbin/logrotate -v /etc/.pihole/advanced/logrotate > /home/pi/logrotate 2>&1)
11

the problem with linux flavors, there's always something different...

12

I missed that one. No, having no log at all is perfectly fine since the users can always decide to completely disable logging (even during the first install) - a log file will never be present and forcing a reboot of dnsmasq is this case is not a good thing to do.

logrotate is now implemented and will become available in the next release.