Thank you very much for the detailed analysis!
I will do my best to address all your points one by one:
Gap in log - another DNS
Currently there are no other DNS servers on the network.
There are no settings for this unfortunately on the ISP router and it is very limited (only settings are DHCP on/off, IPv4 pool, etc.), but as soon as I turn off DHCP on the ISP router and set Pi-hole to be the single DHCP server on the network, clients start to use Pi-hole as the default and only DNS.
If I turn off the DNS service on the Pi-hole machine, the name resolution does not work.
If the ISP router is set as DHCP, the clients start to use the ISP provided DNS servers.
Web UI access issue
I access the Pi-hole Web UI by IP only and it was not available by IP at the time of hang.
Other Web UIs were accessible at that time (by IP as well).
DHCP/RA
Interesting. I have set Pi-hole as DHCP and turned off this setting on the ISP router.
The clients seem to get IPs and DNSMASQ names as normal.
ProxMox network
Currently the ProxMox VMs/CTs are directly getting IPs from the LAN (from Pi-hole), and have their own virtual NICs communicating on the host's "eno1" (Eth0), bridged through "vmbr0".
Managed switch
Hmm, I would not think that the switch has anything to do with this, as there are no special rules or anything set on it currently.
ISP router and distributions
This is something I am not sure of, as the router is very limited. However I believe it is in IPv4 mode only.
Currently, and at the time of creating this debug log, the DHCP is turned off on the ISP router and there are no IPv6 settings accessible.
The "Enable additional IPv6 support (SLAAC + RA)" option is currently turned off on the Pi-hole.
(As the router does not have IPv6 settings and this option above is turned off, I am not sure where the devices get IPv6 addresses from.)
VM (CT) and upstream DNS
I currently have the DNS set manually to these values on both Pi-hole and in the ProxMox Pi-hole CT as well. The physical host has different (manual) DNS settings.
Clarification:
ProxMox physical host (manual) DNS settings:
192.168.0.222 (Pi-hole CT)
192.168.0.1 (ISP router)
1.1.1.1 (Cloudflare)
ProxMox Pi-hole CT (manual) DNS settings:
192.168.0.1
1.1.1.1
1.0.0.1
8.8.8.8
8.8.4.4
Pi-hole (manual) upstream DNS settings:
192.168.0.1
1.1.1.1
1.0.0.1
8.8.8.8
8.8.4.4
My way of thinking with this was that if Pi-hole gets a DNS query then:
A. it is a local query and can resolve it -> (OK)
B. it is a local query, but can't resolve it -> 192.168.0.1 (OK)
C.a. it is an external query -> 192.168.0.1 (OK)
C.b. it is an external query -> 192.168.0.1 (X) -> 1.1.1.1 (OK)
As the Pi-hole runs on the CT, I have provided the same DNSs for the CT as well, similarly to the Pi-hole upstreams.
Also:
The other VMs/CTs on the same ProxMox host get the host's DNSs for themselves.
So the ProxMox host needs to have the Pi-hole set as primary DNS (for the other VMs/CTs), plus if the Pi-hole is not available for some reason, then the other 2 DNS (ISP router and Cloudflare) should be sufficient.
Upstreams
The DHCP server is turned off on the ISP router and Pi-hole deals with the DHCP leases, as well as with the DNSMASQ names.
This resolution currently works.
Also there are no IPv6 settings on the ISP router at all.
If would seem sensible to remove your router from upstreams and enable Pi-hole's Conditional Forwarding to your router instead.
In any case, you should verify that your router distributes Pi-hole's IPv4 as DNS server exclusively, and also that it does not advertise any IPv6 DNS server address at all, or at least only one of your Pi-hole machine's stable IPv6 addresses.
Could you please clarify if in this case these would still be needed?
UI access
This is an interesting phenomenon.
So I noticed that when I leave the Web UI open in my browser for some time (>=15-30 mins?), let it sit/go browsing on other pages, and when getting back to it and trying to click on the interface items, the page freezes for a few seconds then starts rapidly flashing (~10 flashes/sec) for around 2-4 seconds (probably the session has expired and it is trying to refresh) and then it drops me back to the login page (to log back in again).
I did not think of this too much so far, but it was a bit weird. I am not sure what would be the root cause of this.
Additional info to this:
- the browser is Brave, up-to-date
- I have other Web UIs open to the ProxMox host and VMs/CTs, but they don't produce this phenomenon
- there are no plugins added to the browser other than ABP
3rd party software
Currently I am not running anything like that and I am not querying Pi-hole in any ways.
I hope all this helps some.
Once again, I appreciate your time and effort!