I performed the upgrade to pihole v6 this morning on my Raspberry Pi (Raspberry Pi Model B Rev 1) and the installation was slow but seemed to go fine. After v6 was running, however, there are a lot of things going on.
First, a note. I have 2 separate PiHoles running on my network. Besides this one, the other is running in a Proxmox VM and after seeing these issues I've decided to keep the other one at v5 for now. Short story, the Pi can run on UPS for a lot longer than my Proxmox system can.
The web interface is crazy slow after the first page load (or two). CPU usage also seems to constantly be high and I nearly always have a note in Pi-hole diagnosis that the long term load average is larger than the single processor. What's notable, however, is that if I haven't touched the web interface for a while the system from command line doesn't seem as loaded. It's almost as if the built in webserver is just not nearly as efficient as ngix was (and yes, I did tell it to disable ngix during upgrade)
It seems that clients on the network are, at times, stopping use of this instance which I assume is because it's being slow to respond at times. Notably with this running all day and the dashboard is indicating the DNS requests are WAY below normal (the other pi-hole on the VM running v5 seems to be taking nearly all queries which is fine but I feel like it's an indication of unresponsiveness)
This is the most notable and confusing one. After the upgrade I tried running the gravity update and it failed:
[i] Building tree...
[✗] Unable to build gravity tree in /etc/pihole/gravity.db_temp
[i] If you have a large amount of domains, make sure your Pi-hole has enough RAM available
I tried this a few times. I ran out of time early this morning so I just let it run with what it had (it still seemed to be blocking everything in the database prior to update) and when coming back home late tonight without touching the web interface I used the already logged in ssh session on my computer to launch pihole -g and this time it did work ok. It seems like maybe the web server is using the RAM that it needs to update gravity?
So, with all of these things going on I am wondering if there was a change in minimum requirements that eliminate the older Pis from having a decent experience? Or maybe any Pis (since I noticed you also removed the temperature -- maybe most people aren't using Pis anymore)?
I want to stress that all of this was working perfectly on this Pi while running v5 -- I honestly was impressed at just how stable it really was.
Please upload a debug log and post just the token URL that is generated after the log is uploaded by running the following command from the Pi-hole host terminal:
I'm not sure if it matters that it's done after a failed gravity update or not so I decided to try updating gravity via the web interface and it completed without error it seems but what I did notice (and I noticed this prior but after posting the first message) is that only my first list says content was modified recently -- all of the other lists say 7 days so maybe it still isn't updating correctly and using backups instead? I ran the diagnostics (and upload) after completing that.
(I assume that since you're asking for the logs, etc, that Pi-hole is still intended to be able to run on older Pis and I'm just encountering issues?)
As a single reference point, my Pi-hole is happily running on a NanoPi Neo with 256M of RAM with ~800k strong blocklists, along with Wireguard (granted mine is a 4 core CPU, so my web UI will be snappier). Others in our team are running Zeros without issues.
Your debug log shows that your router is indeed distributing two IPv4 addresses for DNS.
Note that switching DNS servers then would happen at your client's discretion.
They'll likely have noticed that one of them was down during upgrades, making them prefer the other one. It would be entirely up to the clients if and when they would decide to switch over to the v6 one. As long as they are happy with your v5, they may never consider to do so.
During upgrade, Pi-hole updates database schemas to the most recent version. While this database conversion may take up both time and resources, it happens only once, during upgrading.
Also, as RAM was identified as a potential bottleneck during updates, we've released an immediate mitigation by having the temporary database created as a file. If you did start your v6 upgrade before that was released, then you may indeed have observed RA;M shortages.
I'm fairly confident that you manage to fix your slow UI behaviour after reading the linked info, and then you should be back to normal.
Ok, I'm glad to hear the intent is not to eliminate the lower power devices. That's the biggest thing I was looking for.
It does seem like switching back to http helped. I often don't use https locally but the notice on the login page made me think (incorrectly I now realize) that this was possibly a sign of something being deprecated.
Agreed and I understand that's likely why the first day of the upgrade it didn't get a lot of requests but I am occasionally getting name resolution errors on devices on my network. These clear up pretty quickly by a couple refreshes but it seems to me that these are situations where the v6 running on the pi fails to respond and the browser or application times out before the OS switches to the v5 pi-hole. Consequently I am still seeing high load warnings appearing in the diagnosis. I hadn't touched anything last night or this morning and just checking now there was a sudden drop off of requests just before a warning that happened at 04:27 saying the load average was high. I never had any of these things happen before the upgrade to v6 on the pi. I did just run the diagnostics again just in case. The new token is https://tricorder.pi-hole.net/xqww9NxV/
This is probably the best explanation for most of the issues that I saw (especially the failing gravity update). Maybe I missed it somewhere but if there isn't a message showing up somewhere it may be helpful to add a message to the update letting people know this. Had I saw such a message I would have walked away and not tried to update gravity or other things until later.
My first (and only) update involving v6 was to v6.0.4 so I suspect the mitigation is included?
The http vs https move seems to have helped the UI stuff for certain but that was really only half of the story. Having the load average going high while I'm not touching the web UI is still a concern. I am also concerned that gravity isn't updating all of the lists even though it's not giving an error. After running gravity, even if nothing has changed on the upstream lists, it'll always update the "Content last updated on" for each of the lists that I use. Currently only the first list is getting this
The first list being 2 days ago was after the gravity update started running ok but all of the additional lists still show 9 days ago and that time stamp is prior to the v6 update.
Adding to this I found this in syslog on one of the machines on my network. This is just a snip of these messages. They continue until 04:34 but there are also runs of these messages at other times of day yesterday as well as a few close to noon today.
Mar 3 04:00:33 gigserv1 systemd-resolved[1018]: Using degraded feature set TCP instead of UDP for DNS server 192.168.1.254.
Mar 3 04:01:04 gigserv1 systemd-resolved[1018]: Using degraded feature set UDP instead of TCP for DNS server 192.168.1.254.
Mar 3 04:01:20 gigserv1 systemd-resolved[1018]: Using degraded feature set TCP instead of UDP for DNS server 192.168.1.254.
Mar 3 04:01:50 gigserv1 systemd-resolved[1018]: Using degraded feature set UDP instead of TCP for DNS server 192.168.1.254.
Mar 3 04:02:06 gigserv1 systemd-resolved[1018]: Using degraded feature set TCP instead of UDP for DNS server 192.168.1.254.
Mar 3 04:02:37 gigserv1 systemd-resolved[1018]: Using degraded feature set UDP instead of TCP for DNS server 192.168.1.254.
Mar 3 04:02:53 gigserv1 systemd-resolved[1018]: Using degraded feature set TCP instead of UDP for DNS server 192.168.1.254.
Mar 3 04:03:13 gigserv1 systemd-resolved[1018]: Using degraded feature set UDP instead of TCP for DNS server 192.168.1.254.
Mar 3 04:03:29 gigserv1 systemd-resolved[1018]: Using degraded feature set TCP instead of UDP for DNS server 192.168.1.254.
Mar 3 04:03:59 gigserv1 systemd-resolved[1018]: Using degraded feature set UDP instead of TCP for DNS server 192.168.1.254.
Mar 3 04:04:15 gigserv1 systemd-resolved[1018]: Using degraded feature set TCP instead of UDP for DNS server 192.168.1.254.
Mar 3 04:04:46 gigserv1 systemd-resolved[1018]: Using degraded feature set UDP instead of TCP for DNS server 192.168.1.254.
Mar 3 04:05:02 gigserv1 systemd-resolved[1018]: Using degraded feature set TCP instead of UDP for DNS server 192.168.1.254.
Mar 3 04:05:39 gigserv1 systemd-resolved[1018]: Using degraded feature set UDP instead of TCP for DNS server 192.168.1.254.
Mar 3 04:05:54 gigserv1 systemd-resolved[1018]: Using degraded feature set TCP instead of UDP for DNS server 192.168.1.254.
I haven't because the web interface is tolerable now and, correct me if I'm wrong, I would rather save CPU cycles for the resolver. Even with not being on the web interface it often takes an extremely long time to log in to the system via ssh which is another reason.
That being said, the output of the webserver.threads config is "0"
I am not well versed with htop so I'm not sure if you are looking for some kind of history feature that it may have but currently there are 2 threads of pihole-FTL that are spiking up to a combined 50% of CPU in frequent bursts. Occasionally I'll see a third pihole-FTL process also spike alongside those but only momentarily. Just to reiterate this is without me even accessing the web interface.
I can try this if you think it will help with performance but it really seems to me at this point that the webserver is no longer an issue in it's self. Having the system take 30-45 seconds to get me to a shell after ssh login at times is more concerning to me than the performance of the web interface (which is now tolerable).
The SSH login delay has made me think of something else and, whether or not it's true in this specific case or not, may be something that is happening with a lot of others running on a Pi..... The SD card. I'm not sure at this point but I recall on other systems I would frequently see SSH login delays when there is high IO load. My speculation is that the upgrade from v5 to v6 was very intense on the SD card and may have pushed many SD cards over the limit.
I am about to head out of town for a funeral so I won't likely be able to dig in anymore for the next few days but I am going to see if I can order a brand new SD card and either do a fresh install and use the transporter to restore backups or just copy the current image onto the new card. I'm not seeing any signs of corruption but I am curious if the SD card is suddenly a bottleneck because it's degraded.
Just wanted to update now that I'm back home.... started having a lot of slow loads and some name resolution errors on my network again so I logged into the v6 pihole via SSH and found that the "wait" in top was pretty high (jumping frequently to above 50%) and found a kworker process consuming about 40% CPU (kworker/0:1H-kblockd). I could be wrong but I interpret this as being the kernel working on a block device (in this case, the SD card) which is starting to make me think my theory that the upgrade process hammered my SD card enough to degrade it. I went ahead and ordered a brand new SD card which should arrive next week. I will update this thread when I can get that replaced.
The SD card actually arrived this morning. I moved the image to the new card and the pi booted fine but I am still seeing heavy loads and it's still occasionally timing out on requests (client devices momentarily can't resolve domains). Watching top over a period of time shows that pihole-FTL does spike up to 40% CPU still and iowait is still occasionally jumping up to 50-60%. Unfortunately I don't know what these numbers typically were before the v6 upgrade but I do know that the system load was basically always under 1 and now it's pretty consistently running between 0.9 and 1.3
