Tail network log?

Is it possible to tail the network log aka the log shown in '/admin/queries.php' as I have logging disabled I noticed the tail options only work with logging enabled, but I like the recent logs only as they purge automatically every 24 hours.

Can you somehow tail that log instead?

I seen a post on arch forums saying it's stored in ram at /run/log/pihole but I don't have that on my system?

Try in /var/log/pihole/

That is referring Pi-hole's Query Log.
This is a log of DNS requests received by Pi-hole during the last 24 hours, as kept in Pi-hole's memory.

Logs are not purged after 24 hours.
Pi-hole just keeps the last 24 hours in-memory.

Pi-hole populates its long term database (at /etc/pihole/pihole-FTL.db) from that in-memory log, and the database may in turn be used to prepopulate in-memory data, e.g. after a restart.

In addition, Pi-hole logs several lines for each DNS request in /var/log/pihole/pihole.log* by default. Those logs are rotated daily and kept for 5 days.

A live view of log statements as submitted to that log is available via Tools | Tail pihole.log (or via pihole -t).

That is essentially a more verbose output of what the Query Log provides, though the Query Log offers better interaction features like filtering or in/excluding domains from blocking.

As each of the aforementioned components may be configured separately:
How exactly did you disable logging?

All these logs are empty for me.
Assuming this is because query logging is disabled.

All of the logs you say are empty for me.
The DB file however I can't open as headless system ATM.

I disabled the logging during setup.

I see logs in the URL I said previously though.

I am looking to see if it's possible to do something similar to 'pihole -t' but for these logs in memory.

Bumping?

Even if headless, you can query the query database with SQL commands. Our documentation page has details.

1 Like

Thanks for the reply.
Just to confirm so I'd be able to see the 24 logs that are stored in memory by querying the database?

Yes, and for much longer than 24 hours. The query data (by default) goes back 365 days

1 Like
Summary

Hmm, alright thanks. Slowly getting where I want however...

When I perform the SQL

SELECT * FROM queries WHERE client IS "IP_HERE" ORDER BY timestamp ASC;

The newest appears to be 5 days ago?

Nvm I was being dumb typing the wrong internal ip :rofl:

A new issue has arrived.

The SQL query is perfect and does what I wanted when using SQL to filter the results queried.

However...

My issue now is that the SQL query seems to not update fast enough.

When I go to a new site on a device connected to pihole as the DNS provider the SQL query (even ran directly on the db) won't update immediately.

I've noticed it does update immediately if I go to 'https:/localhost/admin/queries.php?all'

Is this a thing I need to configure or?

Ideally I'd like to have the ability to view the updates immediately via the SQL command.

EDIT; would it be 'DBINTERVAL' in the FTL config? Changing this seems to give an effect

Indeed, that option controls how often Pi-hole would sync in-memory queries to the database, defaulting to once per minute.
Note that decreasing this value results in a higher number of write accesses, potentially increasing i/o overhead and sd card wear.

You had previously marked my earliest reply as solution, so assuming that your issue was solved. :wink:

Could you elaborate what you are trying to achieve?
What keeps you from (re)enabling logs and using pihole -t?

As mentioned, logrotate takes care that logs are kept for 5 days before being removed.

This is what I was after basically.
Thanks for confirming.

That I don't think I intentionally did, my bad.

I don't want long term data only 24 hours. I noticed when installing that with logs disabled it still logged data for a day this was perfect for me.

Changing the config file allows me to do what I wanted now :slight_smile:

Then your current approach of interacting with Pi-hole's long term database via SQL seems to contradict your intention. :wink:

You could consider to disable Pi-hole's long term db altogether (via MAXDBDAYS) and use Pi-hole's logging as suggested earlier.
That way, you'd deal only with 5 days worth of data, instead of a full year.

Alternatively, you could try to use Pi-hole's API to retrieve in-memory data.

Of course, the kind of information you'd like to extract could also impact your choice.

How so? When I am querying the database for in memory data? I did notice this data is a year as mentioned above though and was confused by this however since I thought having logs/long term data disabled prevented this, however as you mentioned I went ahead and set MAXDBDAYS a while back anyways.

As far as I knew up until recently the only way to grab the in memory data minus using piholes web interface was via SQL as answered here initially.

The outcome has achieved what I initially wanted albeit in a slightly convoluted way.

I wasn't aware of this, how do I do that?

Tldr I basically just wanted a way to list 5 recent domains from the in memory data shown in the pihole web interface to use within a Python script. If there is an API it will make things much easier of course.

As mentioned:

If you are interacting with that database, you are not accessing in-memory data (as you've noticed already :wink: ).

You could check whether Pi-hole's v5 API >getallqueries would meet your requirements.

Clearly I am a little confused.
You quoted yourself saying Pihole populates the db FROM the memory.

Yes I found out there is more data, why is this when I have logging disabled? And you are saying the database is NOT the memory data sounds to me like you are contradicting yourself. :confused:

Thanks I'll take a look. For now though I've achieved what I needed like I said in a albeit roundabout way.

Only blocked or every single domain?

The 5 most recent blocked domains can be read using the API: >recentBlocked (5)

Note:
If you query the API every second, the API will show the most recent data for each interval, but there is no way to guarantee how many domains will be accessed in one second interval:

  • In a quiet network, maybe zero domains will be queried in a specific interval and the API will send the same 5 domains from previous request.
  • In a busier network, maybe hundreds of domains will be queried in a second, but the API will only send the last 5.

If you want a complete list of DNS request received by Pi-hole you will need to enable logging, tail the log and filter the results.

EDIT: or use

>getallqueries-time  1483964295        1483964312       (17)
                  <start_timestamp> <end_timestamp> + optional (<max_number_of_results>)

Pi-hole's in-memory data structures (as reflected by the UI's Query Log) hold the most current and accurate information about queries processed by Pi-hole for at most the last 24 hours.
That data is volatile, i.e. it is lost on power-downs or process termination.

Pi-hole will commit the most recent in-memory data to its long-term database in regular intervals.
That long-term database is persistent, so it survives power-downs and reboots, but it may not be accurate if Pi-hole didn't get a chance to commit the latest interval to the database (e.g. due to sudden power cuts).
By default, Pi-hole will keep up to a year's worth of data in its long-term database.

As you've observed:

This is because you are querying the database, which technically would always lag behind the state of Pi-hole's in-memory data.

In addition, when processing a DNS query, Pi-hole will also log several lines immediately to a log file. As mentioned, those logs are managed by logrotate and kept for at most 5 days.
Like the database, log files are usually persistent (at least when not using a ramdisk for storing them), and as they are updated immediately, it can also be expected to be more accurate than the database (e.g. in the event of power cut).

This is what you've opted to disable during Pi-hole's installation, while Pi-hole's long-term database and its in-memory representation are still active.

You've stated that you don't want long term data, hence my recommendation to disable Pi-hole's long-term database and re-enable logging.
You could then use pihole -t, or tail the logs yourself.

Alternatively, you could try to disable both db and logging, and use Pi-hole's API to retrieve in-memory data instead.

Okay. So I am a little confused still, I thought during installation that disabling logging would do what it says on the tin and not let stuff get saved into a database for a year :sweat_smile:.

Any how. I think disabling both the database and logging (which I thought was what I did during install, what makes it especially confusing is the UI long term data is all blank so initially I was sure I had all kinds of logs disabled), will be what is what I want.

If you are confirming doing this I will still see the in memory data as reflected in the web UI query log, and be able to use the API still to fetch those?

So the in memory data is kept for at most 24 hours?

I feel like the installer should be more clear that even after disabling logging, in memory data will be still saved to a database and not only that you have the logerate thing keeping things for 5 days.

So I disabled the db by setting MAXDBDAYS=0

How do I disable this Logerate thing you were saying.

Now using the API how do I list top 5 recent domains NOT blocked? I see using >getallqueries (5) will display 5 but this includes blocked domains also, I need to filter all blocked types, I could do this via the DB not sure about the API looking at the documentation.

Edit; I seem to have achieved this by going into the API settings in the web UI which I actually didn't notice for a while, and unticking "show blocked domain entries" but of course this then completely removes the ability to see them at all which is not ideal.

Another issue I have is the API command to show per client basis won't allow me to add the limit of (5)?

Additional edit:
Now having the DB disabled in ftl.log it constantly complains that there is no such table etc.