Resizing "/FTL-queries" : Heavy cpu-load, "Lost connection to API"

#1

Please follow the below template, it will help us to help you!

Expected Behaviour:

Pihole runs fine, until it get queries from the following sites:
1st: current.cvd.clamav.net
2nd: fusion.tvaddons.ag
3rd: default._domainkey.flafnine.com
All this caused a “Resizing /FTL-queries” and stopped the FTL service and I had to reinstall the system from the beginning (in total 3 times now). I blacklisted the sites, but it did not help. I could not even find out, which client caused the queries. What can I do?
Thx for your support, oliaros

Actual Behaviour:

Heavy cpu load, “Lost connection to API”

Debug Token:

https://tricorder.pi-hole.net/go3lbqluk1

#2

Can you post the output of the following commands?

echo ">stats" | nc localhost 4711

ls -lh /etc/pihole/pihole-FTL.db

ls -lh /var/log/pihole.log*

1 Like
#3

Just to be clear, Pi-Hole is not getting queries “from” those sites, it is getting queries “for” those sites?

In addition to the commands from @mlbere, what is the output of these as well:

echo ">top-clients" | nc 127.0.0.1 4711

echo ">top-domains" | nc 127.0.0.1 4711

echo ">top-ads" | nc 127.0.0.1 4711
#7

Reply from @oliaros, posted by moderator.

pi@raspberrypi:~ $ echo “>stats” | nc localhost 4711
domains_being_blocked 113474
dns_queries_today 3341999
ads_blocked_today 868
ads_percentage_today 0.025972
unique_domains 884
queries_forwarded 3337072
queries_cached 4059
clients_ever_seen 15
unique_clients 15
dns_queries_all_types 3341999
reply_NODATA 54
reply_NXDOMAIN 1
reply_CNAME 1611
reply_IP 497
privacy_level 0
status enabled
—EOM—

pi@raspberrypi:~ $ ls -lh /etc/pihole/pihole-FTL.db
-rw-r–r-- 1 pihole pihole 531M Jun 7 08:45 /etc/pihole/pihole-FTL.db

pi@raspberrypi:~ $ ls -lh /var/log/pihole.log*
-rw-r–r-- 1 pihole pihole 1.8K Jun 7 08:08 /var/log/pihole.log
-rw-r–r-- 1 pihole pihole 2.3M Jun 7 00:03 /var/log/pihole.log.1
-rw-r–r-- 1 pihole pihole 6.8M Jun 6 16:03 /var/log/pihole.log.1.gz-2019060700.backup
-rw-r–r-- 1 pihole pihole 8.9M Jun 6 12:39 /var/log/pihole.log.2.gz
-rw-r–r-- 1 pihole pihole 595K Jun 6 00:01 /var/log/pihole.log.3.gz
-rw-r–r-- 1 pihole pihole 2.2M Jun 5 00:00 /var/log/pihole.log.4.gz
-rw-r–r-- 1 pihole pihole 14K Jun 4 07:05 /var/log/pihole.log.5.gz

Right now I had to reinstall the system from the beginning because it ran into high cpu load again and FTL stopped working.

Thanks for your assistance,
oliaros

#8

This appears to be the root of the problem. One or more clients is making a significant number of queries and these queries are stored in the long term database and some in memory. This overloads the memory and is likely causing the problems you see.

You can run these commands in the Pi terminal to see the top domains and the top requesting client.

echo ">top-clients" | nc 127.0.0.1 4711
 
echo ">top-domains" | nc 127.0.0.1 4711

echo ">top-ads" | nc 127.0.0.1 4711
#9

@jfb
Here is the output of the requested commands:

pi@raspberrypi:~ $ echo ">top-clients" | nc 127.0.0.1 4711
0 4710139 192.168.0.1 fli4l.allerstrasse.de
1 714 192.168.0.247 libreelec.allerstrasse.de
2 623 192.168.0.40 lizas-iphone.allerstrasse.de
3 465 192.168.0.20 lifebook-lan.allerstrasse.de
4 452 192.168.0.97 redminote4-redmi.allerstrasse.de
5 167 192.168.0.195 galaxy-s2.allerstrasse.de
6 60 127.0.0.1 localhost
7 46 192.168.0.121 spa2102-1.allerstrasse.de
8 40 192.168.0.122 spa2102-2.allerstrasse.de
9 11 192.168.0.198 chumby0.allerstrasse.de
@raspberrypi:~ $ echo ">top-domains" | nc 127.0.0.1 4711
0 4544958 piholenet.b-cdn.net
1 147445 106.203.110.36.in-addr.arpa
2 2675 mailing._domainkey.srv2.de
3 620 e6858.dsce9.akamaiedge.net
4 555 e4478.a.akamaiedge.net
5 474 zattoo.com
6 452 googlehosted.l.googleusercontent.com
7 369 stun.t-online.de
8 328 www-cdn.icloud.com.akadns.net
9 317 apidata.googleusercontent.com
---EOM---
pi@raspberrypi:~ $ echo ">top-ads" | nc 127.0.0.1 4711
0 79 graph.instagram.com
1 39 data.mistat.intl.xiaomi.com
2 17 app.adjust.com
3 16 googleads.g.doubleclick.net
4 11 analytics.ff.avast.com
5 9 settings.crashlytics.com
6 8 www.googleadservices.com
7 7 www.google-analytics.com
8 7 api.ad.intl.xiaomi.com
9 6 adservice.google.com
---EOM---

Thanx for your assistance,
oliaros

#10

That is a huge number of requests from your router (or client connected to your router), for the domain piholenet.b-cdn.net.

This is what is overloading your Pi-Hole and causing the problems you are seeing.

Please generate a new debug log, upload it and post the token here. The old debug log has expired.

#11

@jfb ,
sorry for the late answer, I had to setup the system again yesterday and then had to wait, until the huge number of requests happened again. This time the requests go to current.cvd.clamav.net from 192.168.0.1.

✓] Your debug token is: https://tricorder.pi-hole.net/ud6efzzpgi

Thanks for your support,
oliaros

[edit] after 20h runtime huge numbers of requests from 192.168.0.1 point to b1sync.zemanta.com[/edit]

#12

Blacklisting them makes it worse because they get a TTL of just two seconds.

My router overrunning Pi-hole when the upstream DNS server can’t be reached.

A automatic limiter would help. Pi-hole should detects a flood of the same requests are made, it wil drop all new ones until the first times out. Then it accept a new one and the rest is dropped again.

#13

The problem you are having is not a Pi-Hole problem. Pi-Hole is answering the DNS queries it receives, and the large volume is overwhelming Pi-Hole.

The solution is to determine what software or client is requesting those domains, and try to stop them at the source.

#14

You can suggest this on the dnsmasq mailing list: dnsmasq-discuss@lists.thekelleys.org.uk

As soon as they add this to dnsmasq, we will incorporate it into FTL. I agree with @jfb that this is a clear misbehavior of your router., however, I also see that your chances at getting this fixed are typically rather low on this level.

#15

Having that changed that at router level is even more a mission impossible with Mikrotik.

I started to use Unbound to solve a other problem that Pi-hole triggered the router the do.
hmmmm, I can look if the router self can hard limit it self in when creating those floods.

#16

Just to highlight this: My suggestion to send a mail to the dnsmasq mailing list was honest. This can very well result in the solution you need / want.

#17

I know but that will take time. I don’t think that I would be the first one asking for it and till now it is not implemented.

What is implemented is that I have now limited my router to 60 requests in a minute with a burst of three. If it reaches that limit it will be not allowed to do requests for 90 seconds. In that time that already present requests will time out.
These are only requests made by the router for itself and the client, Pi-hole, Unbound requests are not limited.

The router was the only one who could not control itself and the DNS is a bit rudiment but working as long you know the limitations.

#19

Have you looked through the mailing list archive? It’s completely indexed and searchable through your search engine of choice. Mind, I’d anyone thinks similarly, things would never get asked.

#20

A long long time ago I struggled with DNSSEC and DNSMASQ and that ended in a disappointment getting support. It was partly solved recently.

I searched and the only limiting is possible with DHCP and nothing with DNS. To me, dnsmasq is EOL and is only sporadic maintained on the moment.

When I look in the manual of Unbound I have several different options to limit DNS requests and upstream requests. But then Unbound is crashing when it has not upstream connection and the building/support of Unbound in Debian is death so getting later newer versions are troublesome.

So all kind of dilemmas and I have the feeling that I going to have to freeze the updating DNS programs I use.

#21

@all,
thx for the hints. When I started with PiHole v4.2.1 I had exactly the same problems as I have now. Somewhere in the forum it was suggested to use “pihole checkout ftl tweak/overhaul_overTime” to solve a (maybe completely different) problem. As a total newbee I tried it out and the PiHole ran without any problems for 9 days. Then v4.3.1 was announced and after I updated my PiHole the same problems started again. I tried to use the tweak again, but FTL refused start. It would be great if there was an easy way to downgrade to v4.2.1 to get a running system again. But unfortunately all the suggestions in the forum I could not comprehend.

BTW the problems occur even if Pihole is the only client of the router.

Thanks again, regards, oliaros

#22

@msatter

interesting approach! My router ist running DD-WRT but I have no idea how to restrict the access as you described.

Regards, oliaros

#24

Look for rate limit on Dd-wrt in your case.

But you know what you are doing because iptables is not childs play.

What you are expierences I have seen before here and the important part is not to blacklist it. You have to look wich client and program is causing this flood of requests and when.

#26

@all,
I reinstalled the system with an edge-image from a different source, then I reinstalled PiHole. The output of uname -a is the same as before:
pi@PiHole:~ $ uname -a
Linux PiHole 4.19.42+ #1219 Tue May 14 21:16:38 BST 2019 armv6l GNU/Linux
Something must be different: PiHole runs like a charme now, no huge number of requests any more.
I hope it stays like this.
Regards and thanks to all of you.
oliaros