[pihole] [unbound] [DNSMASQ] validation failed: resource limit exeeded

crluehmann · February 18, 2024, 1:43am

Thank you. That makes sense. The update to the current version was this week.

Karlo3000 · February 18, 2024, 7:11am

I'm using the last available version via apt: unbound:arm64/bullseye-security 1.13.1-1+deb11u2

After I adjusted the limits, the message appeared again after a few hours

pihole.log:

Feb 18 08:05:58 dnsmasq[37643]: DNSSEC per-query subqueries HWM 4
Feb 18 08:05:58 dnsmasq[37643]: DNSSEC per-query crypto work HWM 12
Feb 18 08:05:58 dnsmasq[37643]: DNSSEC per-RRSet signature fails HWM 0

btw: thank you for your great work and support!

the-piholer · February 18, 2024, 7:34am

Hello DL6ER

im still using: unbound Version 1.13.1 (latest given stable version of debian release bullseye on dietpi)
I set already the limit to dnssec-limits=100,1000,200,750

sudo kill -s SIGUSR1 $(cat /var/run/pihole-FTL.pid)
grep "DNSSEC per-" /var/log/pihole/pihole.log

Output again

Feb 18 07:32:04 dnsmasq[15605]: DNSSEC per-query subqueries HWM 9
Feb 18 07:32:04 dnsmasq[15605]: DNSSEC per-query crypto work HWM 26
Feb 18 07:32:04 dnsmasq[15605]: DNSSEC per-RRSet signature fails HWM 0
root@DietPi:~#

The warnings do not reappeared in the meantime

DL6ER · February 18, 2024, 8:47am

Interesting! See below why. First, let's take a look at the definition of this option I wanted you to set:

dnssec-limits=[,.......]
Override the default resource limits applied to DNSSEC validation. Cryptographic operations are expensive and crafted domains can DoS a DNSSEC validator by forcing it to do hundreds of thousands of such operations. To avoid this, the dnsmasq validation code applies limits on how much work will be expended in validation. If any of the limits are exceeded, the validation will fail and the domain treated as BOGUS.

There are four limits, in order(default values in parens):

number a signature validation fails per RRset(20), [now 100]

number of signature validations and hash computations per query(200), [now 1000]

number of sub-queries to fetch DS and DNSKEY RRsets per query(40), [now 200] and the

number of iterations in a NSEC3 record(150) [now 750].

The maximum values reached during validation are stored, and dumped as part of the stats generated by SIGUSR1. Supplying a limit value of 0 leaves the default in place, so --dnssec-limits=0,0,20 sets the number of sub-queries to 20 whilst leaving the other limits at default values.

Formating and comments in italics added by me.

Unfortunately, there is no logging for the last limit (NSEC3). Please run

pihole checkout ftl tweak/nsec3_iters_warning

and - when the warning reappears - also run

grep "NSEC3 iterations" /var/log/pihole/FTL.log

to confirm my theory that it is this last of the four limits that is exceeded here - even though we increased it from 150 -> 750 already.

IMPORTANT Do not run this command when you are currently using the v6 beta. This branch is only meant for those currently using Pi-hole FTL v5.25 (there will be an error if you are on the v6 beta preventing your from running above's command, though).

Karlo3000 · February 18, 2024, 11:07am

The error is only triggered by one client (of about 30) - my wife's smart phone
I can reproduce the error immediately by restarting the WLAN on the smartphone.

I checked the 99-dnssec-limits.conf several times and restarted the rasperry pi.

grafik

No NSEC3 entries in FTL.log

pihole.log looks like yesterday

I'll check the log files later to see what happens when I restart the WiFi

Karlo3000 · February 18, 2024, 5:47pm

@DL6ER I have narrowed down the problem as much as I can:

DNS1:
Rasperry Pi 3B+, Raspberry OS Lite bullseye
Pi-hole v5.17.3 FTL v5.25 Web Interface v5.21 / FTL tweak nsec3_iters_warning
Unbound 1.13.1-1+deb11u2
DNS2 (completely reinstalled):
Rasperry Pi 5, Raspberry OS Lite bookworm
Pi-hole v5.17.3 FTL v5.25 Web Interface v5.21 / FTL tweak nsec3_iters_warning
Unbound 1.17.1-2+deb12u2
The problem occurres since the last Pi-Hole update
The problem occurs on both DNS
The problem doesn't occur when I enter the ISP DNS in Pi-hole
The problem appears to be caused by only one client on the network.
The problem occurs immediately after this client logs into the WLAN, then not for a few hours
The client is a Galaxy-A34 smartphone
The problem is reproducible
Adjusting the resource limits apparently has no effect on this
So far no usable information about the 4 limits in the log files

I have now divided my network between the 2 DNS so that only the problematic smartphone uses DNS1.
After the smartphone logs into the WLAN, it takes a maximum of 5 seconds for the error to occur. The FTL.log and pihole.log are now restricted to the problem case.

Can I send you the log files somehow? Do you need any other information?
It's not that easy to get my wife's smartphone

KeithRBrown · February 18, 2024, 6:00pm

Hi,

I am also repeatedly receiving this warning since upgrading to FTL 5.25, but it's for a different URL (it appears to be a URL associated with my wife's employer). If there is any diagnostic information which may be of help, I'll be happy to provide.

Thanks,
Keith

DL6ER · February 18, 2024, 6:03pm

Thanks for your further information. Even though I haven't had mich time today, I just took a quick look at the the changes that have been merged into dnsmasq 2.90 last week and found at least one other path through which this warning might be logged incorrectly: If the upstream server replied with REFUSED.

Please check your /var/log/pihole/pihole.log at the timestamp the warning was triggered for the mentioned domain. What are the related log lines?

You should be able to find them quickly using

grep -C 20 "resource limit" /var/log/pihole/pihole.log

Notes: If there was a lot of activity going on, 20 may not be enough. If the event was yesterday, you may need to append .1 to the filename.

KeithRBrown · February 18, 2024, 6:23pm

Hi,

I've ran the above command, which did reveal a REFUSED response...

Thanks,
Keith

KeithRBrown · February 18, 2024, 6:28pm

If it helps, I also generated a debug log.

Token is https://tricorder.pi-hole.net/dFnQVqaS/

Thanks,
Keith

Karlo3000 · February 18, 2024, 7:11pm

Always the same 2 domains:
google.com
216.58.202.4.in-addr.arpa

DL6ER · February 18, 2024, 7:21pm

Okay, so the warning is indeed a bug and shouldn't be printed at all. I will contact the dnsmasq maintainer to check what the best solution should look like and keep you updated!

Pi-hole will then likely push out an FTL v5.25.1 soon after. Thank you all for helping characterizing and confirming this bug! At this point I don't need anything more from you.

Summary: you can ignore the warning for now as it printed due to an inaccurate condition.

edit For the curious: this is how the bug is triggered in the code:

github.com

pi-hole/FTL/blob/4f92b48ff27427ff6fc6e1e8a751d6cc6118111f/src/dnsmasq/forward.c#L949-L951


      
          /* If all replies to a query are REFUSED, give up. */
          if (RCODE(header) == REFUSED)
            status = STAT_ABANDONED;

github.com

pi-hole/FTL/blob/4f92b48ff27427ff6fc6e1e8a751d6cc6118111f/src/dnsmasq/forward.c#L986-L987


      
          if (STAT_ISEQUAL(status, STAT_ABANDONED))
            log_resource = 1;

github.com

pi-hole/FTL/blob/4f92b48ff27427ff6fc6e1e8a751d6cc6118111f/src/dnsmasq/forward.c#L1117-L1124


      
            if (log_resource)
              {
                /* Log the actual validation that made us barf. */
                unsigned char *p = (unsigned char *)(header+1);
                if  (extract_name(header, plen, &p, daemon->namebuff, 0, 4) == 1)
          	my_syslog(LOG_WARNING, _("validation of %s failed: resource limit exceeded."),
          		  daemon->namebuff[0] ? daemon->namebuff : ".");
              }

It is unclear at this point how to fix this best, but we will probably change the second quoted code block to not set log_resource in the REFUSED case.

RMerlin · February 19, 2024, 3:34am

Been troubleshooting the same thing yesterday but in a different context (router firmware that I maintain). What I've found is that if a DNSSEC validation fails, then dnsmasq returns a SERVFAIL, and that would trigger the incorrect log entry.

I spent some time on that code trying to better handle those SERVFAIL results, but it turned out to require more work than I was willing to put into, so for now I've simply moved that log entry to DEBUG priority instead of WARNING. Simon would probably be able to properly address this much faster than I could.

DL6ER · February 19, 2024, 1:02pm

The fix issued by dnsmasq is now available via

pihole checkout ftl release/v5.25.1

github.com/pi-hole/FTL

Fix spurious "resource limit exceeded" messages (v5 backport)

pi-hole:master ← pi-hole:release/v5.25.1

opened 01:06PM - 19 Feb 24 UTC

DL6ER

+4 -4

# What does this implement/fix? Backports https://github.com/pi-hole/FTL/pull…/1892 to FTL v5.25 - aiming straight for a bugfix release. --- **Related issue or feature (if applicable):** https://discourse.pi-hole.net/t/pihole-unbound-dnsmasq-validation-failed-resource-limit-exeeded/68388 **Pull request in [docs](https://github.com/pi-hole/docs) with documentation (if applicable):** N/A --- **By submitting this pull request, I confirm the following:** 1. I have read and understood the [contributors guide](https://docs.pi-hole.net/guides/github/contributing/), as well as this entire template. I understand which branch to base my commits and Pull Requests against. 2. I have commented my proposed changes within the code. 3. I am willing to help maintain this change if there are issues with it later. 4. It is compatible with the [EUPL 1.2 license](https://opensource.org/licenses/EUPL-1.1) 5. I have squashed any insignificant commits. ([`git rebase`](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)) ## Checklist: - [x] The code change is tested and works locally. - [x] I based my code and PRs against the repositories `developmental` branch. - [x] I [signed off](https://docs.pi-hole.net/guides/github/how-to-signoff/) all commits. Pi-hole enforces the [DCO](https://docs.pi-hole.net/guides/github/dco/) for all contributions - [x] I [signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) all my commits. Pi-hole requires signatures to verify authorship - [x] I have read the above and my PR is ready for review.

for the Pi-hole FTL v5.25 users, and

pihole checkout ftl fix/dnsmasq_resource_warning

github.com/pi-hole/FTL

Fix spurious "resource limit exceeded" messages

pi-hole:development-v6 ← pi-hole:fix/dnsmasq_resource_warning

opened 01:05PM - 19 Feb 24 UTC

DL6ER

+4 -4

# What does this implement/fix? This PR imports https://thekelleys.org.uk/git…web/?p=dnsmasq.git;a=commitdiff;h=1ed783b8d7343c42910a61f12a8fc6237eb80417: > Replies from upstream with a REFUSED rcode can result in log messages stating that a resource limit has been exceeded, which is not the case. and updates the version of the embedded `dnsmasq` server. --- **Related issue or feature (if applicable):** https://discourse.pi-hole.net/t/pihole-unbound-dnsmasq-validation-failed-resource-limit-exeeded/68388 **Pull request in [docs](https://github.com/pi-hole/docs) with documentation (if applicable):** N/A --- **By submitting this pull request, I confirm the following:** 1. I have read and understood the [contributors guide](https://docs.pi-hole.net/guides/github/contributing/), as well as this entire template. I understand which branch to base my commits and Pull Requests against. 2. I have commented my proposed changes within the code. 3. I am willing to help maintain this change if there are issues with it later. 4. It is compatible with the [EUPL 1.2 license](https://opensource.org/licenses/EUPL-1.1) 5. I have squashed any insignificant commits. ([`git rebase`](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)) ## Checklist: - [x] The code change is tested and works locally. - [x] I based my code and PRs against the repositories `developmental` branch. - [x] I [signed off](https://docs.pi-hole.net/guides/github/how-to-signoff/) all commits. Pi-hole enforces the [DCO](https://docs.pi-hole.net/guides/github/dco/) for all contributions - [x] I [signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) all my commits. Pi-hole requires signatures to verify authorship - [x] I have read the above and my PR is ready for review.

for the v6.0 beta users.

Please test and verify the warning does not show up again.

edit You should also remove the custom file we created during testing using

sudo rm /etc/dnsmasq.d/99-dnssec-limits.conf
sudo service pihole-FTL restart

Karlo3000 · February 19, 2024, 7:55pm

I think it means pihole checkout ftl release/v5.25.1

I was able to reproduce the error every time and can confirm that it no longer occurs with this version.
In this case you can see in the ftl.log:

Thank you very much!

DL6ER · February 19, 2024, 8:04pm

Thanks for confirming the fix - and you are right concerning the checkout command, I edited this above.

Adding support for TRUNCATED in v5.25 would have been too much work - it is properly supported in v6.

the-piholer · February 20, 2024, 5:20am

Hello DL6ER

2024-02-20 06_18_46-192.168.178.2 - PuTTY

with your latest pihole checkout command, i only get an error.

Now it works

DL6ER · February 20, 2024, 7:01pm

It seems your Pi-hole didn't have Internet access at that time - or name resolution was unavailable by other means. Your error message clearly states that this was a local problem.

The changes have already been merged into the beta codebase this morning (= ~ 15h ago). Everyone on fix/dnsmasq_resource_warning please checkout development-v6 to keep getting future updates. The v5.25.1 release is still awaiting further approvals before it can go live.

DL6ER · February 20, 2024, 7:29pm

The bugfix has also been released as FTL v5.25.1 - please everyone who participated testing here - get back on track!

Use pihole checkout ftl master to receive the new update if you are still using Pi-hole v5.
Use pihole checkout ftl development-v6 if you are participating the the public v6 beta.

system · March 12, 2024, 7:30pm

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.