CPU usage grows until next reboot - caused by td-agent-bit

The issue I am facing:

This is an observation of reboots vs. CPU usage.

image

Details about my system:

My setup is Raspberry Pi 4 - 2GB version. Installed on top of Ubuntu.
My blog post covering the installation is here: https://lowtek.ca/roo/2021/pi-hole-a-black-hole-for-advertisements/ along with links to the base OS setup.

What I have changed since installing Pi-hole:

Because I had IPv6 setup - I was getting lots of numbered clients - by adding REFRESH_HOSTNAMES=ALL I was able to have the pi-hole resolve all of my clients to the local DNS names - this was very helpful.

It is possible that reboot to reboot - this setting is having an increasing number of client names to look up -- but I don't have an ever growing number of devices on my home network - so the number of reverse lookups should stop growing (where as the CPU usage appears to grow and grow).

I only have 14 days of data, as that is the current limit in my prometheus system.

Does anyone have the same observation? What could it be that is using more and more CPU? (I have more metrics for the system - so maybe I can splunk into the details to figure it out)

What is the scale on the vertical axis? I see time since boot which I assume is in hours, but what about CPU? CPU usage in Linux is typically reported in CPU units, where 1 would be 100% of a single CPU.

Are you running out of resources on the Pi? If you don't reboot, what is the outcome?

Oops - I should have included more details on scale.
CPU usage is 0-100%
Uptime is in days.

The pi-hole is very low CPU always - under 5% in general - however as you can see from the graph, there is slow growth over time. The fact that the uptime & CPU usage are closely related should not be taken with too much relevance, if I had tracked uptime in hours the slants would be very different.

Visualization aside - there is still CPU growth. It starts below 1%, and grows to 5%.

On the far left - you can see that my uptime hit 21 days - but the CPU usage hadn't significantly grown past 5%. I'm not having problems running out of resources.

This is also just the rough CPU usage graph - I don't actually know that the pi-hole software is using more and more CPU, only that something is. My pi-hole is dedicated hardware for pi-hole. The only additions are things to allow me to monitor the usage, and I tried to be good about blogging all the steps I took.

I wouldn't worry about the small growth in CPU use. It could be any app or service on the Pi that is consuming the CPU (not necessarily Pi-hole). And, you are using a fraction of the available CPU.

True - this far from being a problem. I just notice that the CPU usage and uptime were related - and squashing or understanding the growth is an interesting exploration.

Mostly I was posting here to document my journey. I do appreciate people taking the time to read this thread.

I would open a separate terminal window and run htop in that window. Sort by CPU, then you can watch resources by individual services.

Well - this is interesting

htop tells me that td-agent-bit seems to be eating a lot of CPU relative to the other things on the system. This is weird, because the logging agent shouldn't really be using very much CPU at all.

Related - while poking at this - I discovered that something is not quite right with my forwarding of logs - because I don't seem to consistently get log data from the pi-hole machine to my centralized log server.

At this point - I'm going to blame all of the CPU growth on the logging agent.

Yup -- restarting the td-agent-bit service radically cut my CPU usage down (but I expect it to grow slowly over time). I think this proves this is nothing to do with pi-hole at all.

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.