Did the update came with some syslog formating changes? I'm missing the "source ip" host requesting the record which in turn messed up my elk dashboard.
Every syslog line previously had the source ip of the client in its syslog message
Thanks
Can you post an example of what you are seeing, and in which file(s)?
Not right now as I'm on mobile right now. I'll update this post asap.
In short, the previous log message had the requesting source ip in the beginning of each log message, and this filed is missing in the recent version, so I can't get the correlation between blocked url and requesting client asking for the blocked domain
I'll check the previous log formating style and update this post...
old:
Dec 26 12:29:46 lxDNS01 dnsmasq[36864]: 192.168.254.123/1234 query[A] update.microsoft.com from 192.168.254.254
%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:date} %{HOSTNAME:hostname} %{SYSLOGPROG}: %{REQUESTING_CLIENT_IP}/%{POSINT:Source_Port} query[%{WORD:query_type}] %{HOSTNAME:domain_request} from %{IPV4:request_from}
new:
Dec 26 12:29:46 lxDNS01 dnsmasq[36864]: query[A] update.microsoft.com from 192.168.254.254
%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:date} %{HOSTNAME:hostname} %{SYSLOGPROG}: query[%{WORD:query_type}] %{HOSTNAME:request} from %{IPV4:request_from}
As you can see "192.168.254.123/1234"/%{REQUESTING_CLIENT_IP}/%{POSINT:Source_Port} does not show up anymore ...
for future reference: seems like the update did change "-q, --log-queries" and its previous setting of "--log-queries=extra"
Log the results of DNS queries handled by dnsmasq. Enable a full cache dump on receipt of SIGUSR1. If the argument "extra" is supplied, ie --log-queries=extra then the log has extra information at the start of each line. This consists of a serial number which ties together the log lines associated with an individual query, and the IP address of the requestor.
If I explicitely set it back to "extra" the log message is like it was before.
Thanks
Further background information for you: We switched to extra logging back in the days when our statistics generation still relied on parsing the log file (Pi-hole v3.0). It allowed us to uniquely couple replies to queries (what you mentioned as well) and hence improved the accuracy of our statistics from formally about 99% to 100%. This change, however, also caused the log file to grow much faster and, hence, we decided to roll back this behavior to the pre-v3.0 behavior (no extra logging).
Just as a warning: It is likely that your changes will be reset by future Pi-hole updates as we re-generate all config files during updates (because, behind the scenes, we're using some kind of pihole -r in the update process). Also, we might decide at some point to disable logging on devices by default (also largely to reduce the stress on the SD card), but in this case we will provide an opt-in feature to have it permanently enabled. If we don't forget, we might also implement the opt-in such that the user can chose between simple and extra logging in pihole.log so things will stay as they are during any future updates.
thanks for going so much into detail.
I'm and always was aware of the fact that pihole updates will overwrite any manual changes to various files as I added a conf.d file which sets syslog facility to daemon - and thats gets overwriten every time too.
The annoying thing is that dnsmasq is complaining about syslog facitlity beeing configured 2 times (the update process sets it to some value and my conf.d file sets it to another).
Is there some way to overcome this behaviour or am I doing something wrong in general?
The problem is that the order in which dnsmasq reads in the config files is non-deterministic, hence it is unclear if it reads first log-queries or log-queries=extra if both exist. This gives rise to the uncertainty which of the two should have overwritten the other one and, as this is an unresolvable situation, it is a fatal error.
I'm currently on vacation and stay away from the Pi-hole codebase, but I will add the addition of a toggle option (simple/extra) to my ToDo list so that this setting will survive future updates. I will notify you when this is done and tell you what to do in order to have the setting already survive the next update.
Thanks for your effort but please note:
The possibility of having the option to keep the setting of log queries extra would only make sense for me if this is applicable to all settings you overwrite via update in this file (e.g syslog facility)
Okay, thanks for the clarification. We will discuss internally how to proceed with this. There is the huge problem of detecting if changes are meant to be like they are in the files or if something broke. Hence, we currently allow only a subset of things to be changed and always re-write the configuration on, both, repairs and updates, parsing all possible allowed customization on-the-fly. All these special ifs are already at the limit of being maintainable and adding even more features that are needed by only a minority of users is actually pushing us beyond our limits as - and this should not be taken lightly - every possible combination of options needs to be tested for new releases as Pi-hole got way to complex to be able to forecast all implications a simple code change might have.