Pihole crashes after a while, reboot does not start

[qucklgw]

Please follow the below template, it will help us to help you!

Expected Behaviour:

Pihole doesn't crash at all or Pihole recovers after a reboot

Actual Behaviour:

Pihole would crash after a few hours. Running systemctl restart pihole-FTL doesn't work, rebooting doesnt work, pihole -r doesnt work either

Both systemctl restart and pihole -r will get stuck at starting pihole-FTL

I have 2 VM running the same configuration, both has the same issue after a while.

Debug Token:

https://tricorder.pi-hole.net/a4u888n6y0

[mibere]

Any useful information in the output of
sudo journalctl --no-pager -b -u pihole-FTL.service ?

[qucklgw]

root@pi-hole0:~# systemctl status pihole-FTL

• pihole-FTL.service - LSB: pihole-FTL daemon
  Loaded: loaded (/etc/init.d/pihole-FTL; generated)
  Active: activating (start) since Sun 2019-08-18 02:12:42 +08; 1min 20s ago
  Docs: man:systemd-sysv-generator(8)
  Cntrl PID: 471 (pihole-FTL)
  Tasks: 4 (limit: 505)
  Memory: 82.1M
  CGroup: /system.slice/pihole-FTL.service
  |-471 /bin/bash /etc/init.d/pihole-FTL start
  |-563 su -s /bin/sh -c /usr/bin/pihole-FTL pihole
  |-566 sh -c /usr/bin/pihole-FTL
  `-567 /usr/bin/pihole-FTL

Aug 18 02:12:42 pi-hole0 systemd[1]: Starting LSB: pihole-FTL daemon...
Aug 18 02:12:43 pi-hole0 pihole-FTL[471]: Not running
Aug 18 02:12:43 pi-hole0 su[563]: (to pihole) root on none
Aug 18 02:12:43 pi-hole0 su[563]: pam_unix(su:session): session opened for user

Command ran after a reboot. lighttpd is running, I can get the login page fine.

Update:
Okay, weird, now the service starts.

Update2:
So here's another vm that crashed, running on Ubuntu 18.04 instead of Debian 10. This is the log after a crash and a reboot. pihole-FTL seems to be running at 100% cpu usage.
https://tricorder.pi-hole.net/k42f59vvej

Restarting pihole-FTL will just take forever and timeout

[jfb]

You appear to have a large number of queries in a 24 hour period. This can cause difficulties when FTL tries to read them from the database on restart.

[2019-08-18 02:26:12.620 673] Resizing "/FTL-queries" from 71794688 to 72024064

What is the output of this command from the Pi Terminal:

echo ">stats" | nc localhost 4711

[qucklgw]

No output, since probably because pihole-FTL failed to start.
Here's the output on another VM that was used as a secondary DNS

qucklgw@pi-hole1:~$ echo ">stats" | nc localhost 4711
domains_being_blocked 815015
dns_queries_today 7769314
ads_blocked_today 20134
ads_percentage_today 0.259148
unique_domains 101035
queries_forwarded 5899561
queries_cached 1849594
clients_ever_seen 2782
unique_clients 2635
dns_queries_all_types 7769314
reply_NODATA 255066
reply_NXDOMAIN 6266402
reply_CNAME 544493
reply_IP 504415
privacy_level 0
status enabled
---EOM---

Yea, I'm testing it at a site with about ~1000 users, and I suspect some of the hosts were infected and spamming DNS requests.

Would using anonymous or no statistic mode help? Or am I better using a different DNS solution for a site this big?