Constant random DNS crashes

xolinlevh · March 1, 2023, 3:48pm

Expected Behaviour:

PiHole should be resolving DNS queries normally
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
-pi 3 model b

Actual Behaviour:

Randomly, DNS resolution will stop. Can be every few days, few weeks, or this past monday it was constantly failing as soon as id reboot it.

Debug Token:

https://tricorder.pi-hole.net/Ti3yvikh/

Heres a shot showing the last 30 days....dips in graph are times the DNS wasnt resolving. I'm able to SSH in and restart the DNS server and things come back up...but, its really annoying

phlmike · March 1, 2023, 5:06pm

Same issue, I can't see your debug logs (obviously), but have you seen log entries like:

[2023-03-01 11:55:00.072 641/T669] ERROR: SQL query "DELETE FROM network_addresses WHERE lastSeen < 1646153700;" failed: database is locked (SQLITE_BUSY)
[2023-03-01 11:56:00.015 641/T669] Encountered error while trying to store client in long-term database

Or entries like:

[✗] Integrity errors in /etc/pihole/pihole-FTL.db found.

*** in database main ***
Main freelist: size is 160212 but should be 160211
On tree page 380224 cell 0: 2nd reference to page 349760
Page 230182 is never used
Page 230183 is never used
wrong # of entries in index domain_by_id_domain_idx
row 13495798 missing from index idx_queries_timestamps

chrislph · March 1, 2023, 5:18pm

Upload a new debug log using the Execute database integrity check option in Tools > Generate debug log, or the command

pihole -d -c

If you're seeing those errors it implies your database became corrupted at some point, perhaps due to a power loss or power supply voltage issues. You can always start over with a fresh FTL database.

sudo service pihole-FTL stop
sudo mv /etc/pihole/pihole-FTL.db /etc/pihole/pihole-FTL-old.db
sudo service pihole-FTL start

if it keeps happening you can check for power supply voltage issues.

phlmike · March 1, 2023, 5:29pm

Power loss... My battery backup's batteries died completely on me. When the power went out, the UPS didn't stay up for the Hypervisor to shut down.

xolinlevh · March 1, 2023, 5:45pm

Attached log body here...sorry thought the link before would work

debug log removed by moderator

jfb · March 1, 2023, 5:49pm

Your debug log doesn't show any potential problems, other than this:

   [2023-03-01 01:00:04.215 22199/T22223] SQLite3 message: database is locked in "SELECT name FROM network_addresses WHERE name IS NOT NULL AND ip = ?;" (5)
   [2023-03-01 01:00:04.216 22199/T22223] getNameFromIP("192.168.0.39") - SQL error prepare: database is locked

Do you have any other processes running that directly manipulate the query database?

chrislph · March 1, 2023, 5:49pm

The link before would have been fine, I'm just a normal user and cannot read them. I only suggested redoing it with the database integrity check option as that will help staff see if there really is a database consistency problem. It's best to delete the actual log from your previous post – only post the token URL, not the full log.

xolinlevh · March 1, 2023, 6:05pm

Shouldnt....runs on its own Pi and i dont do anything other than random updates when needed.

xolinlevh · March 7, 2023, 12:59am

Ran that command the other day, same issue just happened again

https://tricorder.pi-hole.net/bgpmCdx9/

Bucking_Horn · March 7, 2023, 7:24am

No.
The dips in the graph do represent times at which your Pi-hole has received fewer queries, and that Pi-hole did actually process those fewer.

This may just suggest periods of lowered DNS activities, or perhaps that some of your some clients would be by.passing Pi-hole at times.

As with the previous, your most recent debug log does not contain any pointers to Pi-hole crashing.

Quite possibly, your observation does not relate to crashes:

One of your client's has been rate-limited.:

*** [ DIAGNOSING ]: Pi-hole diagnosis messages
 count  last timestamp       type         message     blob1   blob2
 -----  -------------------  -----------  -------     ------  ------
 1      2023-03-06 18:55:20  RATE_LIMIT   192.168.0.8 1000    60

Pi-hole's web UI would also have alerted you about this.
And if you didn't restart Pi-hole in the meantime, that message should also be still available via Tools | Pi-hole diagnosis.

Rate limiting happens when a client sends excessive amounts of DNS requests.
By chance, your debug log nicely demonstrates this for your 192.168.0.8:

*** [ DIAGNOSING ]: contents of /var/log/pihole

-rw-r--r-- 1 pihole pihole 75K Mar  6 18:58 /var/log/pihole/FTL.log

  -----tail of FTL.log------
  [2023-03-06 18:55:06.518 983M] Imported 236531 queries from the long-term database
  [2023-03-06 18:55:06.523 983M]  -> Total DNS queries: 236531
  [2023-03-06 18:55:06.523 983M]  -> Cached DNS queries: 14405
  [2023-03-06 18:55:06.523 983M]  -> Forwarded DNS queries: 81680
  [2023-03-06 18:55:06.523 983M]  -> Blocked DNS queries: 139887
  [2023-03-06 18:55:06.523 983M]  -> Unknown DNS queries: 0
  [2023-03-06 18:55:06.523 983M]  -> Unique domains: 2931
  [2023-03-06 18:55:06.523 983M]  -> Unique clients: 35
  [2023-03-06 18:55:06.524 983M]  -> Known forward destinations: 2
  [2023-03-06 18:56:06.591 983/T1000] Still rate-limiting 192.168.0.8 as it made additional 99363 queries
  [2023-03-06 18:57:06.652 983/T1000] Still rate-limiting 192.168.0.8 as it made additional 128788 queries

Note that those would show as spikes in the graph rather than dips.

Commonly., you'd have to investigate your client for reasons why it would send those excessive amounts of DNS requests. In some cases, clients may be desperate for a specific domain that is blocked by Pi-hole, in which case How do I determine what domain an ad is coming from? may help.

Lucky for you, your debug log actually contains a few lines from /var/log/pihole/pihole.log that show one such domain when your client at 192.168.0.8 was being rate limited:

*** [ DIAGNOSING ]: Pi-hole log
-rw-r----- 1 pihole pihole 343M Mar  6 18:58 /var/log/pihole/pihole.log

  -----tail of pihole.log------
  Mar 6 18:58:53 dnsmasq[983]: config error is REFUSED (EDE: blocked)
  Mar 6 18:58:53 dnsmasq[983]: Rate-limiting v7event.stats.avast.com is REFUSED (EDE: blocked)
  Mar 6 18:58:53 dnsmasq[983]: query[AAAA] v7event.stats.avast.com from 192.168.0.8
  Mar 6 18:58:53 dnsmasq[983]: config error is REFUSED (EDE: blocked)
  Mar 6 18:58:53 dnsmasq[983]: Rate-limiting v7event.stats.avast.com is REFUSED (EDE: blocked)

Chances are you should be able to mitigate this.
In a recent topic, a German user succeeded by configuring a specific reply for those requests, see Probleme mit dem RATE_LIMIT, ein Client sendet zu viele Anfragen - #12 by Keen_Barete.

Let's try to apply their solution to your Pi-hole - please
a) create a custom dnsmasq configuration file, e.g. at /etc/dnsmasq.d/42-block-avast.conf) with the following contents:

# block AVAST telemetry
local=/analytics.ff.avast.com/
local=/v7event.stats.avast.com/

b) exempt above domains (analytics.ff.avast.com, v7event.stats.avast.com) from blocking in Pi-hole via Pi-hole's Domain management
c) restart Pi-hole's DNS resolver via Settings or by running pihole restartdns

system · March 28, 2023, 7:25am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.