REQUEST: Option to send logs to a remote logserver

Pablo_Repolles · July 10, 2023, 6:31pm

It doesn't work well with the rsyslog service - it never attends the logs from "/var/log/pihole.log" - however the FTL ones are sent.

Forward logs to remote server

local0.* action(type="omfwd" target="10.172.211.15" port="1514" protocol="udp" action.resumeRetryCount="100" queue.type="linkedList" queue.size="10000")

Define extra log sources:

module(load="imfile" PollingInterval="30")
input(type="imfile" File="/var/log/pihole.log"
Tag="pihole"
severity="info"
Facility="local0")
input(type="imfile" File="/var/log/pihole-FTL.log"
Tag="pihole-FTL"
severity="info"
Facility="local0")

Tim_Dockter · January 25, 2024, 12:58am

This would be a very nice option especially for Synology NAS owners. Synology has a syslog server that receives logs via port 514 and archives them. I know TP-Link access points are capable of sending their logs to a log server and it would be nice if pi-hole could do the same.

DanSchaper · January 25, 2024, 6:20pm

You should be able to do that now with rsyslog. Have you tried that approach yet?

Tim_Dockter · January 26, 2024, 12:52am

Dan, thanks for your excellent suggestion. I got it working as follows:

In /etc/rsyslog.conf add:

module(load="imfile") #provides way to convert text to syslog

Uncomment 4 lines for imud and imtcp in /etc/rsyslog.conf:

module(load="imudp")
input(type="imudp" port="514")

module(load="imtcp")
input(type="imtcp" port="514")

Add the remote log server IP and port to /etc/rsyslog.conf:

Specify your log server IP and port * . * @192.168.0.34:514

Note: Look up the syntax. My exact statement caused odd formatting in my post so I tweaked it.

Create /etc/rsyslog.d/pihole.conf with these lines:

$InputFileName /var/log/pihole.log
$InputFileTag pihole
$InputRunFileMonitor
$InputFilePersistStateInterval 1000

Optionally create /etc/rsyslog.d/piholeftl.conf with these lines:

$InputFileName /var/log/pihole-FTL.log
$InputFileTag pihole-ftl
$InputRunFileMonitor
$InputFilePersistStateInterval 1000

Restart rsyslog -- systemctl restart rsyslog
It's probably necessary to set “Network at boot” in the raspi-config system options so that NAS access is available when the Pi comes up. I had already done this for automated backups to my NAS.

My imfile setup evidently uses "legacy configuration directives" but I'm not about to change it since they work.

The Synology NAS log server can be configured for either UDP or TCP. I suppose the setup can be done with either imudp or imtcp depending on how you configure things. I'm using UDP on the NAS side.

I did have to tweak a NAS log center setting that spits out a warning if you exceed a "loggings per second" threshold. Mine was 10 by default and I bumped it to 150. That's not a PI-HOLE issue, just an FYI for NAS owners.

Its actually better to do this via rsyslogd since my NAS log archive now includes additional Pi system info (such as sudo sessions and commands) along with my pihole log.

Again, thank you very much for your suggestion to use rsyslog.

UPDATE: Much to my chagrin, the rsyslog approach worked fine up to a point, then it stopped sending pihole.log records. I believe the problem relates to the rsyslog imfile module which uses something called imfile-state files to keep track of which parts of the monitored log file have already been processed. The imfile doc states that it supports file rotation but it seems to have problems in that area. I stopped pihole-FTL, stopped rsyslog, removed all pihole logs, then restarted pihole-FTL and rsyslog. The log records are once again being handled correctly, but I expect that to fail at some point (midnight perhaps). I have an rsyslog debug file set up to get more diagnostics. It may be a few days till I can investigate further.

UPDATE 2: Forgive my frequent editing. I checked this morning and the pihole log records on the NAS side ended at 23:58:12 just before pihole midnight log file processing occurred. Evidently whatever happens then is confusing the rsyslog imfile positioning in the file being monitored. It's possible the imfile reopenOnTruncate will help, but it's going to be a few days till I can try it out.

yubiuser · January 27, 2024, 6:03pm

Just a hint: you should use the files in /var/log/pihole/*

The ones you used are just symlinks for historic reasons and will be removed at some point.

Tim_Dockter · January 29, 2024, 3:02am

Thanks! I just started another test with the imfile reopenOnTruncate feature set and I'll let it run across the midnight hour. After this new test I'll switch to the proper log file names per your reply. I was tempted to try a pihole flush to test it but prefer to just see if it keeps logging after midnight with reopenOnTruncate.

Tim_Dockter · January 29, 2024, 3:56pm

The rsyslog handling of the pihole log file is working great now. It didn't have any problems at midnight when the log file rotation was done. However, I think the main reason it's working now is that I inadvertently removed the rsyslog work directory /var/spool/rsyslog. In preparation for my test I did an "rm -r" command intending to remove the imfile-state files within that directory. It didn't occur to me that the directory would be removed too (I'm not exactly a linux guru). Without a working directory set up rsyslog evidently tries to use "/" but I don't think it could write there. Presumably imfile is not using state files at this point and everything is working perfectly.

My reading of the imfile documentation about state files leads me to suspect that if I restart rsyslog it might read the entire pihole log file and send it to my NAS via port 514 -- if so, I can live with that scenario. I will restart rsyslog to check this out but restarting rsyslog isn't something I would ever do routinely so I can live with extra log data being sent to my NAS if that's how this works.

I did make some changes to my initial rsyslog configuration:

I removed /etc/rsyslog.d/pihole.conf and /etc/rsyslog.d/piholeftl.conf
I changed /etc/rsyslog.conf and replaced my initial module(load="imfile") with these 2 lines:

module load=("imfile" PollingInterval="30")
input(type="imfile" File="/var/log/pihole/pihole.log" Tag="pihole:" deleteStateOnFileDelete="off" PersistStateInterval="0" reopenOnTruncate="on")

The 4 statements from my previous reply that load imtcp and imudp for port 514 processing were left in the conf file, as was the log server IP and port number statement.

I'm not doing anything with pihole-FTL.log at this point.

I'm not suggesting that this setup is a best practice by any means, but it's working nicely for me. If anyone with rsyslog expertise wants to suggest improvements just let me know and I'll tweak my setup.

spaceman_sproggit · January 6, 2025, 9:52pm

Tim, firstly I want to thank you for going to the trouble to document your approach here - it is very kind of you and extremely helpful.

Unfortunately, second, I wanted to note that I have tried to follow your approach but have hit some errors with regards to the rsyslog.conf editing. Before I get to the detail of those problems, I wanted to share the following for anyone looking to take this approach, hitting issues, then needing to diagnose rsyslog problems.

If you type the following in to your Pi at a console prompt:-

sudo rsyslogd -N1 -o /tmp/consolidated-rsyslog.conf -f /etc/rsyslog.conf

you will basically get back a validation of your configuration file, performed by your rsyslogd service [assuming it is running!] but without making changes to your running config...

Now to the errors I saw...

First, I got an error prompted by your syntax:-
'module load=("imfile" PollingInterval="30")'
but I was able to overcome the first part of this by changing the statement to:-
'module(load"imfile" PollingInterval="30")'

Next, I got an error "on or before line 11", where in my file, that is the line that contains:-
{Wildcard}@172.16.101.2:514
as this is the IP address and listening port of a Syslog Server on my QNAP NAS. I know this is working as my router is already writing to it successfully... I managed to solve this issue once I realised that this particular syntax has been deprecated. From the syslog.com documentation I developed this version of the statement:-

{Wildcard}action(type="omfwd" target="172.16.101.2" port="514" protocol="udp"
action.resumeRetryCount="100"
queue.type="linkedList" queue.size="10000")

but note that this will generate some slightly misleading behaviour - the "queue.size" parameter basically buffers up the updates and then sends them in a block to your syslog server, so you won't see real-time updates if you "tail -f" your remote syslog output file, which might leave you wondering if there is a break somewhere. You might want/need to trim that down or remove it during setup.

But it definitely works...

Or, it could be local software differences. In my case my Pi4B tells me that it is running rsyslogd version 8.2102.0 (aka 2021.02).

I would strongly suggest, to anyone who is experimenting with Tim's suggestion, to make use of the rsyslogd validation step, to help you diagnose any errors you might find.

If it helps anyone, I got the guidance on how to specify the remote syslog server from this page:

Tim_Dockter · January 7, 2025, 3:22am

Thanks for adding important information to the rsyslog setup spaceman_sproggit. I apologize if I didn't enter some statements in the correct syntax. To be honest I installed Raspberry Pi OS Bookworm in mid February 2024 and when I did that I stopped using the rsyslog remote server approach in favor of a simple cron job that runs just after midnight when the log rotation is done.