Behavior when database is busy

jpgpi250 · August 19, 2021, 7:48am

installed

Current Pi-hole version is development v5.3.1-100-g98867d8d
Current AdminLTE version is tweak/blocking_while_busy v5.5.1-58-gb4a26d3
Current FTL version is tweak/blocking_while_busy vDev-2edd512

testing may take a while...

jpgpi250 · August 19, 2021, 8:33am

I've run pihole -g like twenty times now, the text 'gravity database is not available' doesn't appear in the log file anymore...

DROP.zip (40.1 KB)

to ensure something needed to be dropped, I ran

#!/bin/bash

domains=("userlocation.googleapis.com." "ssl.google-analytics.com." "incoming.telemetry.mozilla.org." "settings-win.data.microsoft.com.")

# initialize RANDOM only once, to avoid using the same entry evry time
RANDOM=$$$(date +%s)

while :; do
	selecteddomain=${domains[ $RANDOM % ${#domains[@]} ]}
	dig @192.168.2.57 A ${selecteddomain}
	sleep 5
done

on another pi

jpgpi250 · August 20, 2021, 7:42am

Although I didn't get a notification from this topic, I ran pihole -up this morning and got an update.

Current Pi-hole version is development v5.3.1-100-g98867d8d
Current AdminLTE version is tweak/blocking_while_busy v5.5.1-58-gb4a26d3
Current FTL version is tweak/blocking_while_busy vDev-6c6188f

after running the update, I checked for status type 15, and yes, their back 69 entries after the new FTL became active, 15 unique domains

DEBUG_QUERIES and DEBUG_FLAGS aren't active anymore (space on TMPFS is limited)

In the pihole-FTL log, I found (41 times):

[2021-08-20 09:09:57.387 9764M] Gravity database is busy (whitelist)
[2021-08-20 09:09:57.388 9764M] Gravity database is busy (blacklist)

and (90 matches)

[2021-08-20 09:11:40.007 9764M] ERROR: Gravity database not available
[2021-08-20 09:11:40.007 9764M] get_client_groupids("192.168.2.228") - SQL error step: database is locked

I assume this is normal, some FTL thread is still trying to acces the database.

I'm NOT sure how you determine what you consider to be a client initiated resubmitted query, and thus a normal situation. Could you please explain what to look for, to identify duplicates (that shouldn't happen) and resubmitted (that is normal)?

Will you be inviting additional testers, possibly make this part of dev?

Thanks, again, for your time and effort.

DL6ER · August 20, 2021, 9:14am

This can only be seen when you have DEBUG_QUERIES enabled. Each query for which a line like

exists is a new query. These lines are triggered by the code that receives DNS packets at the socket. If there were duplicates in the Query Log/database where no corresponding line in pihole-FTL.log exists - then those would be erroneous. I haven't seen anything indicative for something like this in your data, so far.

Sure, I'll open a PR soon(ish).

This was just a minor tweak to DNS cache debug logging. No change at all when DEBUG_QUERIES is not enabled.

DL6ER · August 23, 2021, 8:12am

github.com/pi-hole/FTL

Customizable locking while database is busy

pi-hole:release/v5.9 ← pi-hole:tweak/blocking_while_busy

opened 08:11AM - 23 Aug 21 UTC

DL6ER

+228 -101

**By submitting this pull request, I confirm the following:** - [X] I have re…ad and understood the [contributors guide](https://github.com/pi-hole/pi-hole/blob/master/CONTRIBUTING.md). - [X] I have checked that [another pull request](https://github.com/pi-hole/FTL/pulls) for this purpose does not exist. - [X] I have considered, and confirmed that this submission will be valuable to others. - [X] I accept that this submission may not be used, and the pull request closed at the will of the maintainer. - [X] I give this submission freely, and claim no ownership to its content. **How familiar are you with the codebase?:** ## 10 --- When the gravity database is locked/busy, Pi-hole allows all queries because it has no way of detecting if they should be blocked or not. However, for very large lists and/or on very busy networks and/or on very (!) slow devices, this may open a windows of up to several seconds where undesired queries may get through, ads and unwanted telemetry could be the result. This PR adds a config option (`REPLY_WHEN_BUSY`) to customize what happens to queries while the database is locked: 1. `ALLOW` - as before, just allow all queries when the database is busy 2. `BLOCK` - block all queries when the database is busy. This uses the configured `BLOCKINGMODE` (default `NULL`) 3. `REFUSE` - refuse all queries which arrive while the database is busy 4. `DROP` - just drop the queries, i.e., never reply to them at all. In my opinion, option 4 is the best one because it will typically cause clients to retry the query after some timeout of typically a few seconds. I.e., the DNS resolution will only be delayed but we do never hand out incorrectly either allowed or blocked queries. `REFUSE` first sounded like a good option, too, however, it turned out that many clients will just immediately retry, causing up to several thousands of queries per second. This does not happen in `DROP` mode.

github.com/pi-hole/AdminLTE

Add new blocked by database status and NONE reply type

pi-hole:release/v5.6 ← pi-hole:tweak/blocking_while_busy

opened 08:14AM - 23 Aug 21 UTC

DL6ER

+23 -13

**By submitting this pull request, I confirm the following:** - [X] I have r…ead and understood the [contributors guide](https://github.com/pi-hole/pi-hole/blob/master/CONTRIBUTING.md), as well as this entire template. - [X] I have made only one major change in my proposed changes. - [X] I have commented my proposed changes within the code. - [X] I have tested my proposed changes, and have included unit tests where possible. - [X] I am willing to help maintain this change if there are issues with it later. - [X] I give this submission freely and claim no ownership. - [X] It is compatible with the [EUPL 1.2 license](https://opensource.org/licenses/EUPL-1.1) - [X] I have squashed any insignificant commits. ([`git rebase`](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)) --- **What does this PR aim to accomplish?:** Add new "blocked by database" status and `NONE` reply type **How does this PR accomplish the above?:** Add interpretation for status 15 **What documentation changes (if any) are needed to support this PR?:** None

DL6ER · August 23, 2021, 6:22pm

The feature has been merged into the FTL v5.9 beta branch and will be included in the next Pi-hole release.

jpgpi250 · August 23, 2021, 6:43pm

is there a branche (like pihole checkout dev) that has all the latest (core, web, ftl) on board, or do I have to get them induvidually?

DL6ER · August 23, 2021, 6:59pm

Yes, because

is just a shortcut to get all the development branches. We don't have something similar for beta branches (with or without heterogeneous versions).

PromoFaux · August 23, 2021, 7:37pm

Except on docker where we have the :beta-v5.9 tag which get's rebuilt nightly to try and keep up with beta development!

jpgpi250 · August 23, 2021, 7:48pm

Just to let you know, the results of this, after updating to

Pi-hole version is development v5.3.1-100-g98867d8d (Latest: v5.3.1)
AdminLTE version is release/v5.6 v5.5.1-65-g224f906 (Latest: v5.5.1)
FTL version is release/v5.9 vDev-9a53df6 (Latest: v5.8.1)

DL6ER · August 23, 2021, 8:00pm

Thanks, this looks promising. We should make DROP the new default in a future Pi-hole release.

jpgpi250 · August 23, 2021, 8:25pm

Word of warning (again), if you're going to make DROP the default.

Correct me if I'm wrong

on new deployed (from scratch) pihole, the pi will not be a client of pihole-FTL, all queries will be resolved by whatever was configured as the resolver during installation
on an existing pihole, that has been upgrade from pihole v4 or earlier, the pi will be a client of pihole-FTL, all queries will be submitted to pihole-FTL.

This implies as you can see in my screenshots (I've changed the behaviour to pre-v5, e.g. my pi is a pihole-FTL client), some queries from 127.0.0.1, during pihole -up / pihole -g are dropped.

Because the pi (127.0.0.1) almost immediatly retries, there isn't a detectable problem, but it does happen (DROP on the pi). I don't know how docker, a VM, or a slower pi (ZERO) will behave. I'm running things on a 3B.

DL6ER · August 30, 2021, 5:35pm

This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.