Long Term data not or only written after checking the same?

[Chris]

Please follow the below template, it will help us to help you!

Expected Behaviour:

System reliably populating long term data

Actual Behaviour:

Lost long term data after flushing of logs (after installation of V4.4) and rebooting

Debug Token:

https://tricorder.pi-hole.net/pet5ooc4bp

======================================

Dear All,

I am pretty much confused with my current Pi-Hole Installations.

Having updated yesterday to V4.4 (RPi 4, Raspbian Buster) I noticed that the dashboard didn’t show data for the Queries over last 24 hrs, neither for Clients (over time).
Looking into the Query Log and also into the Long Term Data did show reasonable information though.

For trying to fix things I did go to settings and flushed my logs – which finally also resulted in a loss of the information within the long term data of the last 24 hrs.
(Happened to both of my installations)

Subsequently the Queries over last 24 hrs, neither for Clients (over time) did build up again and look reasonable as of now.

Doing a reboot (after a apt update / upgrade) did not lead to any data loss.

For the long term data, it also shows information (except those lost 24 hrs) – and based on the investigation I ran I also checked the database location (all should be very much default) and the respective file-time and noticed that the file-time is partially not changing for quite some time (especially at the secondary / backup installation where requests are for sure way less) – despite I understand it should be written every minute..?

Besides I noticed an entry at the dashboard under Top Blocked domains that does show a hit and a frequency – but no Domain. How can this be the case?

So overall – being more a newbie - I am rather confused what happened and what is going on and especially how to avoid to running into more severe issues I maybe can’t see as of now….

Thanks a lot for your help!

Chris

P.S.: Tried to find something that would be applicable to my situation but didn’t find anything that would fit…

PPS.: Edited to avoid improve readability

[DanSchaper]

I notice you have /etc/pihole/setupVars.conf with QUERY_LOGGING=false. Is that intentional?

[Chris]

I for sure didn't enter that manually in that file.

What I did though was to disable the Query Logging at the GUI Level - I do not need a continuously updating log at the GUI - and did that for reducing write operations...

Being able to use the "Recent Queries (showing up to 100 queries)" following the "Query Log" at the dashboard seemed to be good enough so far

[Chris]

Just to ensure I am not running into a misunderstanding:

I suppose disabling "Query Logging" via the (Admin) GUI leads to the mentioned setting in the /etc/pihole/setupVars.conf - right?

This should not cause the long term data to be affected somehow - correct?

As I can (and always could) see data following the "Query Log" via the (Admin) Dashboard I would also understand this is not affected by the respective setting - correct?

I understood that disabling "Query Logging" via the (Admin) GUI only disables the availability of displaying the DNS request "real time" (located under Tools @ the (Admin) GUI - not sure, haven't used that for ages) and consequently reduces write operations - am I mistaken in this regards?

In case I am not mistaken I'd be happy to understand how a flushing of the logs can impact the long term data (given that this is written regularly, usually every minute).

As said it's no tragedy, but I don't want to ignore some hiccup that could maybe lead to a real problem - just because of a lack of knowledge....

Is there maybe any easy way to check the sanity of an installation?
(eg recommended log files to check for failed write operations or such,...?)

Thanks a lot for your time and help!

[DanSchaper]

@Chris

I suppose disabling “Query Logging” via the (Admin) GUI leads to the mentioned setting in the /etc/pihole/setupVars.conf - right?

Yes, that toggle sets the configuration to disable queries being written to the log files.

@Chris

I understood that disabling “Query Logging” via the (Admin) GUI only disables the availability of displaying the DNS request “real time” (located under Tools @ the (Admin) GUI - not sure, haven’t used that for ages) and consequently reduces write operations - am I mistaken in this regards?

No, that is correct, no queries are being logged.

@Chris

In case I am not mistaken I’d be happy to understand how a flushing of the logs can impact the long term data (given that this is written regularly, usually every minute).

If there's nothing being logged then there's nothing to go in the long term database.

@Chris

Is there maybe any easy way to check the sanity of an installation?
(eg recommended log files to check for failed write operations or such,…?)

Everything looks to be operating as requested. Logging was disabled and this is what happens when you stop logging queries.

[Chris]

@Chris

I suppose disabling “Query Logging” via the (Admin) GUI leads to the mentioned setting in the /etc/pihole/setupVars.conf - right?

Yes, that toggle sets the configuration to disable queries being written to the log files.

I understood that disabling “Query Logging” via the (Admin) GUI only disables the availability of displaying the DNS request “real time” (located under Tools @ the (Admin) GUI - not sure, haven’t used that for ages) and consequently reduces write operations - am I mistaken in this regards?

No, that is correct, no queries are being logged.

Sorry, for being nagging but this is really confusing to me.

According to this discussion I would expect that the “Disable Query Logging” button (which I pressed months ago) does NOT stop the long term data logging.

@Chris

In case I am not mistaken I’d be happy to understand how a flushing of the logs can impact the long term data (given that this is written regularly, usually every minute).

If there’s nothing being logged then there’s nothing to go in the long term database.

Is there maybe any easy way to check the sanity of an installation?

(eg recommended log files to check for failed write operations or such,…?)

Everything looks to be operating as requested. Logging was disabled and this is what happens when you stop logging queries.

I didn't change the setting at all - for the time until Feb. 26 ~20:00 the long term data shows queries, and starting from Feb 27 ~20:00 (after I pressed the "Flush logs" button and did a reboot) it also shows queries (as shown in the attached jpg)

My apologies again, but I really don't get my head around this...

[DanSchaper]

@Chris

According to this discussion I would expect that the “Disable Query Logging” button (which I pressed months ago) does NOT stop the long term data logging.

I believe this behavior was changed after users suggested that disabling logging should completely disable logging and Privacy Levels were introduced. But @PromoFaux can help clarify that.

[Chris]

@DanSchaper

I believe this behavior was changed after users suggested that disabling logging should completely disable logging and Privacy Levels were introduced. But @PromoFaux can help clarify that.

According to the docs the long-term database can be disabled by setting DBFILE= in /etc/pihole/pihole-FTL.conf.

My /etc/pihole/pihole-FTL.conf doesn't have any entry but 'PRIVACYLEVEL=0'

Here it's stating

We update the database file periodically and on exit of FTL DNS (triggered e.g. by a service pihole-FTL restart ). The updating frequency can be controlled by the parameter DBINTERVAL and defaults to once per minute. We think this interval is sufficient to protect against data losses due to power failure events.

Hence I can even less explain the missing long term data...

[PromoFaux]

Yeah, so AFAIK (cleverer people may correct me) Disabling logging does not stop logging to the database, just the file at /var/log/pihole.log, and I don't think it will cause the long term data to be affected.

That said, it's not an area that I've delved too deep into, I tend to set and forget when I set up Pi-hole on my network, so it's not something I have too much expertise in.

Outside the box thinking here.. is everything OK with the hardware your Pi-hole is installed on? I wonder if there is the possibility of a corruption on the storage causing this?

[Chris]

@PromoFaux

Outside the box thinking here… is everything OK with the hardware your Pi-hole is installed on? I wonder if there is the possibility of a corruption on the storage causing this?

The same behaviour happened on 2 devices / installations (same activity after the last upgrade to 4.4) - so it would be quite a coincidence that both show hardware issues at the same time.
(Nothing else running on those Raspi's)

Besides the SD Card is only used for booting - the / directory has been moved to an SSD connected via USB 3 (Raspi 4, Raspian Buster)

But I am very much a newbie to Linux and I do hope the hardware is ok - but is there an easy way on how to check for corruption on the storage?

Thanks a lot!

Ps.: What just crossed my mind: there was another update like the day before (4.3.??) - and I might NOT have rebooted the RasPi / OS after this previous upgrade.. Could that have caused this hiccup / a stop writing to the long term database?

[jfb]

@PromoFaux

Disabling logging does not stop logging to the database, just the file at /var/log/pihole.log , and I don’t think it will cause the long term data to be affected.

This is correct.

[deHakkelaar]

@Chris

The same behaviour happened on 2 devices / installations (same activity after the last upgrade to 4.4) - so it would be quite a coincidence that both show hardware issues at the same time.

Does below one show any ?

dmesg | grep -i voltage

[Chris]

@deHakkelaar

Does below one show any ?

dmesg | grep -i voltage

This doesn’t return anything on either device...

[jfb]

@Chris

Having updated yesterday to V4.4 (RPi 4, Raspbian Buster) I noticed that the dashboard didn’t show data for the Queries over last 24 hrs, neither for Clients (over time).

Are you running any adblockers on the browser from which you are loading this page? If so, disable it and reload the page.

Also, some screen captures would help us understand your problem. You can copy and paste screen shots directly into a reply.

[Chris]

@jfb

Are you running any adblockers on the browser from which you are loading this page? If so, disable it and reload the page.

Currently things are looking good - but I haven't flushed the logs (via the GUI button) - but should that actually affect the long term data...?

@jfb

Also, some screen captures would help us understand your problem. You can copy and paste screen shots directly into a reply.

I fear there is little to show except what I pretty much shared before already...missing data...

Sorry for not having more..

[jfb]

@Chris

I haven’t flushed the logs (via the GUI button) - but should that actually affect the long term data…

It should not. But, an adblocker can stop the web admin page from properly displaying the data. If it is totally missing, an adblocker can cause that.

In your screen shot below, are you sure the data is missing, and not that there just was no traffic to the device for this period? If there is a second DNS available to clients (and particularly the router), the traffic may have jumped to the other DNS server. For the period of missing data, what is in file /var/log/pihole.log

[Chris]

@jfb

It should not. But, an adblocker can stop the web admin page from properly displaying the data. If it is totally missing, an adblocker can cause that.

Thanks for pointing this out - I ensured that nothing gets blocked from my Pi-Holes.

@jfb

In your screen shot below, are you sure the data is missing, and not that there just was no traffic to the device for this period? If there is a second DNS available to clients (and particularly the router), the traffic may have jumped to the other DNS server. For the period of missing data, what is in file /var/log/pihole.log

I am perfectly sure this data was gone after I pressed "Flush Logs" @ the GUI and rebooted subsequently.
I can exclude that there hasn't been any traffic - and the same happened to my Primary Pi- Hole.

Here the screenshot, just taken:

Network is configured to have only the 2 Pi-Holes as DNS - the router is only queried by the Pi-Holes as conditional forwarder as it acts as DHCP.

/var/log/pihole.log is empty - understanding because I disabled query logging via the GUI button...