PiHole 5 suddenly stops to resolve DNS requests

schuettecarsten · September 21, 2020, 9:22am

Please follow the below template, it will help us to help you!

Actual Behaviour:

pihole FTL stops resolving DNS requests for unknown reasons - sometimes after some hours, sometimes after a few seconds. I cannot see anything in the logs. I tried to update to the latest development branch, without success. After restarting DNS resolver using 'pihole restartdns' or 'service pihole.FTL restart', it works again, but after some time it stops to resolve DNS requests again.

'pihole -t' shows the following output when the issue occurs:

root@raspi1:/home/pi# pihole -t
  [i] Press Ctrl-C to exit
11:27:00: forwarded 71.178.168.192.in-addr.arpa to 192.168.178.1
11:27:00: reply 192.168.178.71 is kasa71-hs100.fritz.box
11:27:00: reply 192.168.178.71 is HS100.fritz.box
11:27:00: query[PTR] 5.6.6.0.8.c.e.c.0.b.f.8.4.f.1.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.d.f.ip6.arpa from 127.0.0.1
11:27:00: config fd00::21f4:8fb0:cec8:665 is NXDOMAIN
11:27:00: query[PTR] c.5.e.1.3.e.d.3.1.7.f.2.5.5.8.c.0.0.0.0.0.0.0.0.0.0.0.0.0.0.d.f.ip6.arpa from 127.0.0.1
11:27:00: config fd00::c855:2f71:3de3:1e5c is NXDOMAIN
11:27:00: query[PTR] 1.3.0.5.5.e.5.7.3.8.d.8.9.8.4.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.d.f.ip6.arpa from 127.0.0.1
11:27:00: config fd00::5489:8d83:75e5:5031 is NXDOMAIN
11:27:01: query[A] login.microsoftonline.com from fd00::c855:2f71:3de3:1e5c

That's it, no more output and noting happens on the log. When I restart FTL service from a different shell, log continues for some time, before it again stops on a query line.

Expected Behaviour:

pihole.FTL should work without interruptions

Debug Token:

https://tricorder.pi-hole.net/znv6s3r2dd

jfb · September 21, 2020, 3:38pm

If you are looking for stability, the dev branch is not what you want to be running. It is developmental, and not a stable release.

I recommend going back to the stable master with pihole checkout master, then we can troubleshoot from there.

schuettecarsten · September 21, 2020, 6:49pm

I tried several versions without success. Even "master" versions sometimes stop working. And also reinstalling or reconfiguring the while system did not help. I am back on master now, I recreated the pihole-FTP.db, and resolving still stops sometimes for unknown reasons. Nothing special in pihole-FTL.log, nothing in pihole -t output...

  Current Pi-hole version is v5.1.2.
  Current AdminLTE version is v5.1.1.
  Current FTL version is v5.2.

technicalpyro · September 22, 2020, 8:19pm

I have also been seeing this issue on the dev branches
Please add

DEBUG_LOCKS=true
DEBUG_DATABASE=true

in /etc/pihole/pihole-FTL.conf and pihole restartdns

the log file (likely the entire one) is of interest when the outage happens and can be uploaded to tricorder once you have restored DNS and then we can take a look

Bucking_Horn · September 22, 2020, 10:12pm

Just to complement technicalpyro's hint for uploading files into Pi-hole's tricorder: You can execute

cat /var/log/pihole-FTL.log | pihole tricorder

or substitute with any relevant other log, and then post the token(s) here.
(see also How do I debug my Pi-hole installation? for other ways)

DL6ER · September 23, 2020, 1:55pm

Yeah, development contains the change that Pi-hole stores one hostname per address instead of one hostname per device (so multiple hostnames per device are possible). This is important in case of ISP-provided IPv6 hostnames. It required a non-backwards-compatible extension to the database.

anon55913113:

[2020-09-23 04:35:00.622 20242/T20250] Waiting for lock in resolveClients() (/root/project/src/resolve.c:346)
[2020-09-23 04:40:00.007 20242/T20248] Waiting for lock in DB_thread() (/root/project/src/database/database-thread.c:55)
[2020-09-23 04:59:00.053 20242/T20249] Waiting for lock in GC_thread() (/root/project/src/gc.c:47)

[2020-09-23 08:32:21.804 20242M] Waiting for lock in _FTL_new_query() (/root/project/src/dnsmasq_interface.c:520)

Just so I get this right: There was nothing in between 04:59:00 and 08:32:21 ? There was also no activity at all in between?

It was exactly what we fixed for clients in pi-hole/FTL#889 but this time, the bug showed up during resolving the host name of upstream destinations which was not active for some time. This has never happened on our test systems, so we didn't see this before. Sorry.

github.com/pi-hole/FTL

Do not block shared memory when inactive upstreams are skipped

pi-hole:development ← pi-hole:fix/development_locked_old_upstreams

opened 01:51PM - 23 Sep 20 UTC

DL6ER

+9 -6

**By submitting this pull request, I confirm the following:** - [X] I have re…ad and understood the [contributors guide](https://github.com/pi-hole/pi-hole/blob/master/CONTRIBUTING.md). - [X] I have checked that [another pull request](https://github.com/pi-hole/FTL/pulls) for this purpose does not exist. - [X] I have considered, and confirmed that this submission will be valuable to others. - [X] I accept that this submission may not be used, and the pull request closed at the will of the maintainer. - [X] I give this submission freely, and claim no ownership to its content. **How familiar are you with the codebase?:** ## 10 --- This fixes a variant of the bug in `development` which was forgotten in https://github.com/pi-hole/FTL/pull/889

It would be awesome if you could check whether

ftl checkout fix/development_locked_old_upstreams

fixes the issue you are seeing.

DL6ER · September 23, 2020, 7:24pm

Yeah, this somehow sounds plausible given the bug the mentioned PR fixes. BTW, it was already merged into development so no need to switch to the custom branch.

The issue was that when FTL saw an upstream destination which was not being used for two hours, it could end up in a dead-lock. This never happened to me, because my VoIP phone does some two queries every hour (24/7), so there is activity even when everything else is offline. This should now be fixed.

DL6ER · September 25, 2020, 6:36am

Hmm, interesting. How old was the gravity database you restored from the backup?

The error is not critical, the trigger is added thereafter. If it wasn't there, then it didn't need to be removed.

github.com

pi-hole/pi-hole/blob/6b536b7428a1f57ff34ddc444ded6d3a62b00a38/advanced/Scripts/database_migration/gravity/3_to_4.sql#L52


      
          CREATE TABLE domainlist_by_group
          (
          	domainlist_id INTEGER NOT NULL REFERENCES domainlist (id),
          	group_id INTEGER NOT NULL REFERENCES "group" (id),
          	PRIMARY KEY (domainlist_id, group_id)
          );
          
          DROP TRIGGER tr_whitelist_update;
          DROP TRIGGER tr_blacklist_update;
          DROP TRIGGER tr_regex_whitelist_update;
          DROP TRIGGER tr_regex_blacklist_update;
          CREATE TRIGGER tr_domainlist_update AFTER UPDATE ON domainlist
              BEGIN
                UPDATE domainlist SET date_modified = (cast(strftime('%s', 'now') as int)) WHERE domain = NEW.domain;
              END;
          
          DROP VIEW vw_whitelist;
          CREATE VIEW vw_whitelist AS SELECT domain, domainlist.id AS id, domainlist_by_group.group_id AS group_id
              FROM domainlist
              LEFT JOIN domainlist_by_group ON domainlist_by_group.domainlist_id = domainlist.id
              LEFT JOIN "group" ON "group".id = domainlist_by_group.group_id

It should have been created in the 2 to 3 step:

github.com

pi-hole/pi-hole/blob/6b536b7428a1f57ff34ddc444ded6d3a62b00a38/advanced/Scripts/database_migration/gravity/2_to_3.sql#L28


      
          DROP VIEW vw_regex;
          DROP TRIGGER tr_regex_update;
          
          CREATE VIEW vw_regex_blacklist AS SELECT DISTINCT domain
              FROM regex_blacklist
              LEFT JOIN regex_blacklist_by_group ON regex_blacklist_by_group.regex_blacklist_id = regex_blacklist.id
              LEFT JOIN "group" ON "group".id = regex_blacklist_by_group.group_id
              WHERE regex_blacklist.enabled = 1 AND (regex_blacklist_by_group.group_id IS NULL OR "group".enabled = 1)
              ORDER BY regex_blacklist.id;
          
          CREATE TRIGGER tr_regex_blacklist_update AFTER UPDATE ON regex_blacklist
              BEGIN
                UPDATE regex_blacklist SET date_modified = (cast(strftime('%s', 'now') as int)) WHERE domain = NEW.domain;
              END;
          
          CREATE TABLE regex_whitelist
          (
          	id INTEGER PRIMARY KEY AUTOINCREMENT,
          	domain TEXT UNIQUE NOT NULL,
          	enabled BOOLEAN NOT NULL DEFAULT 1,
          	date_added INTEGER NOT NULL DEFAULT (cast(strftime('%s', 'now') as int)),

Apparently, your backup was somehow broken, I still don't expect any issues from that.

system · October 16, 2020, 6:50am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.