REQUEST: Option to send logs to a remote logserver

Feature Requests
21

It doesn't work well with the rsyslog service - it never attends the logs from "/var/log/pihole.log" - however the FTL ones are sent.

Forward logs to remote server

local0.* action(type="omfwd" target="10.172.211.15" port="1514" protocol="udp" action.resumeRetryCount="100" queue.type="linkedList" queue.size="10000")

Define extra log sources:

module(load="imfile" PollingInterval="30")
input(type="imfile" File="/var/log/pihole.log"
Tag="pihole"
severity="info"
Facility="local0")
input(type="imfile" File="/var/log/pihole-FTL.log"
Tag="pihole-FTL"
severity="info"
Facility="local0")

22

This would be a very nice option especially for Synology NAS owners. Synology has a syslog server that receives logs via port 514 and archives them. I know TP-Link access points are capable of sending their logs to a log server and it would be nice if pi-hole could do the same.

23

You should be able to do that now with rsyslog. Have you tried that approach yet?

24

Dan, thanks for your excellent suggestion. I got it working as follows:

  1. In /etc/rsyslog.conf add:

    module(load="imfile") #provides way to convert text to syslog

  2. Uncomment 4 lines for imud and imtcp in /etc/rsyslog.conf:

    module(load="imudp")
input(type="imudp" port="514")

module(load="imtcp")
input(type="imtcp" port="514")

  3. Add the remote log server IP and port to /etc/rsyslog.conf:

Specify your log server IP and port * . * @192.168.0.34:514

Note: Look up the syntax. My exact statement caused odd formatting in my post so I tweaked it.

  1. Create /etc/rsyslog.d/pihole.conf with these lines:

    $InputFileName /var/log/pihole.log
$InputFileTag pihole
$InputRunFileMonitor
$InputFilePersistStateInterval 1000

  2. Optionally create /etc/rsyslog.d/piholeftl.conf with these lines:

    $InputFileName /var/log/pihole-FTL.log
$InputFileTag pihole-ftl
$InputRunFileMonitor
$InputFilePersistStateInterval 1000

  3. Restart rsyslog -- systemctl restart rsyslog

  4. It's probably necessary to set “Network at boot” in the raspi-config system options so that NAS access is available when the Pi comes up. I had already done this for automated backups to my NAS.

My imfile setup evidently uses "legacy configuration directives" but I'm not about to change it since they work.

The Synology NAS log server can be configured for either UDP or TCP. I suppose the setup can be done with either imudp or imtcp depending on how you configure things. I'm using UDP on the NAS side.

I did have to tweak a NAS log center setting that spits out a warning if you exceed a "loggings per second" threshold. Mine was 10 by default and I bumped it to 150. That's not a PI-HOLE issue, just an FYI for NAS owners.

Its actually better to do this via rsyslogd since my NAS log archive now includes additional Pi system info (such as sudo sessions and commands) along with my pihole log.

Again, thank you very much for your suggestion to use rsyslog.

UPDATE: Much to my chagrin, the rsyslog approach worked fine up to a point, then it stopped sending pihole.log records. I believe the problem relates to the rsyslog imfile module which uses something called imfile-state files to keep track of which parts of the monitored log file have already been processed. The imfile doc states that it supports file rotation but it seems to have problems in that area. I stopped pihole-FTL, stopped rsyslog, removed all pihole logs, then restarted pihole-FTL and rsyslog. The log records are once again being handled correctly, but I expect that to fail at some point (midnight perhaps). I have an rsyslog debug file set up to get more diagnostics. It may be a few days till I can investigate further.

UPDATE 2: Forgive my frequent editing. I checked this morning and the pihole log records on the NAS side ended at 23:58:12 just before pihole midnight log file processing occurred. Evidently whatever happens then is confusing the rsyslog imfile positioning in the file being monitored. It's possible the imfile reopenOnTruncate will help, but it's going to be a few days till I can try it out.

2 Likes
25

Just a hint: you should use the files in /var/log/pihole/*

The ones you used are just symlinks for historic reasons and will be removed at some point.

26

Thanks! I just started another test with the imfile reopenOnTruncate feature set and I'll let it run across the midnight hour. After this new test I'll switch to the proper log file names per your reply. I was tempted to try a pihole flush to test it but prefer to just see if it keeps logging after midnight with reopenOnTruncate.

27

The rsyslog handling of the pihole log file is working great now. It didn't have any problems at midnight when the log file rotation was done. However, I think the main reason it's working now is that I inadvertently removed the rsyslog work directory /var/spool/rsyslog. In preparation for my test I did an "rm -r" command intending to remove the imfile-state files within that directory. It didn't occur to me that the directory would be removed too (I'm not exactly a linux guru). Without a working directory set up rsyslog evidently tries to use "/" but I don't think it could write there. Presumably imfile is not using state files at this point and everything is working perfectly.

My reading of the imfile documentation about state files leads me to suspect that if I restart rsyslog it might read the entire pihole log file and send it to my NAS via port 514 -- if so, I can live with that scenario. I will restart rsyslog to check this out but restarting rsyslog isn't something I would ever do routinely so I can live with extra log data being sent to my NAS if that's how this works.

I did make some changes to my initial rsyslog configuration:

  1. I removed /etc/rsyslog.d/pihole.conf and /etc/rsyslog.d/piholeftl.conf

  2. I changed /etc/rsyslog.conf and replaced my initial module(load="imfile") with these 2 lines:

module load=("imfile" PollingInterval="30")
input(type="imfile" File="/var/log/pihole/pihole.log" Tag="pihole:" deleteStateOnFileDelete="off" PersistStateInterval="0" reopenOnTruncate="on")

The 4 statements from my previous reply that load imtcp and imudp for port 514 processing were left in the conf file, as was the log server IP and port number statement.

I'm not doing anything with pihole-FTL.log at this point.

I'm not suggesting that this setup is a best practice by any means, but it's working nicely for me. If anyone with rsyslog expertise wants to suggest improvements just let me know and I'll tweak my setup.