UI panels spinning for both pihole in HA pair

Help
#1

Expected Behavior:
Unable to load admin page or panels become responsive if scripts loaded from cache.

Actual Behavior:
Admin dashboard either does not load or panels become unresponsive for the following panels:

  • query types
  • queries answered by
  • top permitted domains
  • top blocked domains panels.

In incognito, the admin page just never loads but default landing page does. When you do a force-reload/cache-less load of /admin/, the following files are not retrieved from the lighthttpd server (network requests stall):

  • bootstrap.min.js
  • select2.min.css
  • default-dark.css
  • adminlte.min.js
  • Chart.min.js
  • logo.svg
  • donate.gif
  • ip-address-sorting.js
  • utils.js
  • queries.js
  • footer.js

pihole1 utilization: >336000 queries, >2500 per hour.
pihole2 utilization: ~84000 queries, ~100 per hour.

no high CPU usage
restarting does not restore functionality
/var/log/pihole.log{,.1} sizes being 38MiB & 67MiB, respectively, on pihole1 and 16MiB & 7.4MiB on pihole2.

Env info:

  • Xen VM instance (2core, 2GiB, 24GiB VHD on local SSD), Guest Agent 8.0
  • Ubuntu Server 20.04 w/kernel 5.4.0-70-generic (fully updated)
  • lighttpd 1.4.55
  • php 7.4.3
  • pihole 5.2.4
  • adminLTE 5.4
  • FTL 5.7
  • unbound 1.9.4-2ubuntu1.1apt
  • browser client - Windows 10 Pro w/Chrome 89.0.4389.114 32b

Debug tokens:
pihole1 debug: https://tricorder.pi-hole.net/dn1yanai2x
pihole2 debug: https://tricorder.pi-hole.net/j73k1h829h

#2

Please follow our template - you have not provided debug tokens for either of these Pi-hole instances.

#3

You're right! I've updated with the two logs now.

#4

Just checking in on this to see if anyone was able to look at the two debug files.

#5

I just ran "pihole flush" on pihole1 and then reboot it and now the interface is back to working function.

This feels like the database just grew to a point where it was unsupportable with the current design.

#6 
   2021-04-07 13:52:07: (mod_fastcgi.c.421) FastCGI-stderr: PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/admin/api_FTL.php on line 324
   2021-04-07 13:52:13: (mod_fastcgi.c.421) FastCGI-stderr: PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/admin/api_FTL.php on line 324
#7

The database is not involved in any of these. The data is provided from FTL which has them in its memory. You have "HA" in your title. What do you mean? Are the two Pi-holes somehow synchronized? If so, how?

#8

I mean I have two piholes, on the same LAN, that all hosts are told about via DHCP and that they are configured the same as to support the same effective state across both instances. While I would agree this is not true HA, the effective result is similar enough for my usecase.

#9

So PHP itself is running out of memory... then something else is artificially limiting it as they both have memory_limit = -1 set:

root@ubu2004-pihole1:~# cat /etc/php/7.4/cli/php.ini |grep memory_limit
memory_limit = -1

root@ubu2004-pihole2:~# cat /etc/php/7.4/cli/php.ini |grep memory_limit
memory_limit = -1
#10

You pasted two listings of cli.

#11

My apologies, you are correct sir [Note to self ord('l') != ord('g')].

root@ubu2004-pihole1:~# cat /etc/php/7.4/cgi/php.ini |grep memory_limit
memory_limit = 128M

root@ubu2004-pihole2:~# cat /etc/php/7.4/cgi/php.ini |grep memory_limit
memory_limit = 128M

I'll mod these to -1 and test to see if things continue working once the query log gets some length back.

#12

Just to be sure, you posted the same file twice again. This time it was cgi.

Sorry, missed the hostname.