Request pihole.log per-request sequence numbers

Feature Requests Closed or Out Of Scope
1

Hi,. I would like to be able to integrate requests that I can retrieve from /var/log/pihole.log and then associate ALL of the individual log entries from a 'session', i.e. a specific request, so that I can use that information in a security logging environment.
as an example in my current pihole.log is;

Jul 14 19:07:12 dnsmasq[814]: query[A] login.live.com from 192.168.1.66
Jul 14 19:07:12 dnsmasq[814]: forwarded login.live.com to 1.1.1.1
Jul 14 19:07:12 dnsmasq[814]: reply login.live.com is <CNAME>
Jul 14 19:07:12 dnsmasq[814]: reply login.msa.msidentity.com is <CNAME>
Jul 14 19:07:12 dnsmasq[814]: reply login.msa.akadns6.net is <CNAME>
Jul 14 19:07:12 dnsmasq[814]: reply vs.login.msa.akadns6.net is 40.90.23.229
Jul 14 19:07:12 dnsmasq[814]: reply vs.login.msa.akadns6.net is 40.90.23.153
Jul 14 19:07:12 dnsmasq[814]: reply vs.login.msa.akadns6.net is 40.90.23.218

I'd find it beneficial for this to be enhanced a little such as;

Jul 14 19:07:12 dnsmasq[814] 1000-1: query[A] login.live.com from 192.168.1.66
Jul 14 19:07:12 dnsmasq[814] 1000-2: forwarded login.live.com to 1.1.1.1
Jul 14 19:07:12 dnsmasq[814] 1000-3: reply login.live.com is <CNAME>
Jul 14 19:07:12 dnsmasq[814] 1000-4: reply login.msa.msidentity.com is <CNAME>
Jul 14 19:07:12 dnsmasq[814] 1000-5: reply login.msa.akadns6.net is <CNAME>
Jul 14 19:07:12 dnsmasq[814] 1000-6: reply vs.login.msa.akadns6.net is 40.90.23.229
Jul 14 19:07:12 dnsmasq[814] 1000-7: reply vs.login.msa.akadns6.net is 40.90.23.153
Jul 14 19:07:12 dnsmasq[814] 1000-8: reply vs.login.msa.akadns6.net is 40.90.23.218

Now, the example '1000-n', is only trying to show that a specific query 'session' is in this case (some sequence) #1000 and it just happens to have 8 individual detail log records associated with the 'session' (query). Further, the previous 'session' would have been #999 and the subsequent session (query) would be #1001, each might have its own unique number of individual detail quantity (1-n).
Note: as some of the monitoring (FTL) is clearly showing that 'sessions' (queries) are being counted and thus are know to be unique, of course they are because each 'session' (query) results in a response to the submitter that 'counter' could very well be the session counter (the #- part) and the 'session' (query) 1-n count would be associated with that. I would not really be overly concerned about the session # restarting with a restart of pihole but persisting it would also be acceptable, until possibly some day as in your higher-end examples of millions of queries per day results in a substantially large session number (session id) and someone worries about signed/unsigned 32-bit counters (they do still exist...). Anyway, some sort of session counter that yields uniqueness....
My usage would then be to pull the entries from the /var/log/pihole.log and generate a series of log events that I would send to a SIEM, e.g. ArcSight, and long term storage for such needs as audit and analytics. I'd still have to do some parsing around merging/concantenating the query line with the various processing/response lines but that would be doable.
I have noted others asking for external logging (syslog) feature so if that existed then I could use that and do the log entry merging on the remote host but of course depednign on the protocol used to transport, e.g. udp/tcp, loss could occur of records so i am fine doing my work on the pi-hole host.
Thank you,
Doug

2

How would Pi-Hole delineate a "session"? Pihole sees a constant stream of DNS requests from all clients and processes them as received. A client could have multiple applications open, each generating DNS queries. Pi-Hole has no information that would tell it which of the requests should be grouped together, as only the DNS request goes to Pi-Hole. There is no context associated with any DNS requests.

Edit - ahh, now I see what you want. Just group all activities that directly follow a single request as a session. No context required.

I believe the Pi-Hole log is generated by dnsmasq, not Pi-Hole. The devs will know.

3

pihole.log is generated by dnsmasq, not Pi-hole. Look at dnsmasq's man page for details, especially log-queries=extra:
http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html

4

the simplest way to delineate a session that I can think of is the source IP/Port (regardless of protocol) as that is one-half of the classic network 'session' identification, i.e. a sourceIP/sourcePort/sourceProtocol/destinationIP/destinationPort (of course within some timerange where there is no session FIN(ish)/R(e)S(e)T in a connection oriented protocol like TCP but not UDP). But whatever is on the destination end, e.g in this case the pi-hole, needs to keep track of those sessions and then the order of each message generated.

5

AND, using log-queries=extra by modifying /etc/dnsmasq.d/01-pihole.conf as suggested after reading the link as suggested produced EXACTLY what I am looking for. OF COURSE, things still get dumped into /var/log/pihole.log, pointed to by that config file in property log-facility, in some processing order that starts with the query but is then sequence influenced by local caching and referred (forwarded) speed and other such sources of delay (latency). Taking the file, e.g. grabbing a chunk or doing some 'tail -F' or some such, and doing a sort as with gnu sort something like "sort -t ' ' -s -k 4,4 -k 5,5n -k 1,3" gets things ordered up by the dnsmasq unique id (pid), then the session id that dnsmasq generates due to log-queries=extra along with (probably redundant but...) the date/time (h'mmm I need to now see if I can get it to express date/time as yyyy-mm-dd 24h:mm:ss) and that '-s' keeping things per-session in their original logged order so I can see the resolution step down through if forwarded on, e.g. I send to quad-one but one may like quad-9, quad-8, ….., and finally the possibly multiple IP's enumerated. Loving this.

6

thank you to jfb and mcat12!!!
loving this.
getting exactly what I was looking for.
off to do a little record concatenation and enhancement and feed the results to me SIEM!
AGAIN
Thank you!
Doug

7

OK to close this feature request to your solution?

8

closure -> yes
thank you