Pihole+Unbound rclone+onedrive

Help Community Help
1

Please follow the below template, it will help us to help you!

If you are Experiencing issues with a Pi-hole install that has non-standard elements (e.g you are using nginx instead of lighttpd, or there is some other aspect of your install that is customised) - please use the Community Help category.

Expected Behaviour:

[Replace this text with what you think should be happening. Please include as much detail as possible including, but not limited to:
-operating system Raspberry Pi4
-hardware]

Actual Behaviour:

I use Pihole+PiVPN+Unbound on my Raspberrypi 4. All work's fine. The DHCP and blocklist are without any issue.

Now I tried to setup rclone on my synology NAS with onedrive. The problem is: Failed to sync: couldn't list files: Get "https://graph.microsoft.com/v1.0/drives/xxxxxx/items/xxxxx!32864/children?$top=1000": dial tcp: lookup graph.microsoft.com on 192.168.178.222:53: server misbehaving.

192.168.178.222 is my pihole. Unbound was installed with the official doc.. Now I read in this forum from rclone that the problem is the pihole. The workaround in the 1st post is not save according to some forum post below.

Have someone the same problem and a solution.

2

Let's see if you are indeed observing the same issue as claimed in the rclone forum post you link.

Run from your RPi 4, what's the output of:

echo ">top-domains (12) >quit" | nc localhost 4711
3

thanks for your fast replay.

The output is:
0 10565 graph.microsoft.com
1 3176 forum.rclone.org
2 1054 checkip.synology.com
3 901 discourse.pi-hole.net
4 589 www.google.com
5 556 d3p8zr0ffa9t17.cloudfront.net
6 556 www.google.de
7 547 api-global.netflix.com
8 520 checkipv6.synology.com
9 488 api.onedrive.com
10 439 connectivity-check.ubuntu.com
11 389 nrdp.prod.ftl.netflix.com

4

graph.microsoft.com turning up as the most requested domain would support that your rclone's DNS requests are subject to Pi-hole's rate limit (as introduced by Pi-hole's FTL v5.7).

Note that contrary to the opinion from the post you've linked, this is a client issue.
It is a client that decides to send a DNS request, and when and how often it does so.

`dig` returns a TTL of 4 hours for the `graph.microsoft.com` CNAME and of 5 minutes for the respective A records that it points to (click for details). 
~$ dig graph.microsoft.com

; <<>> DiG 9.11.5-P4-5.1+deb10u3-Debian <<>> graph.microsoft.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63530
;; flags: qr rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;graph.microsoft.com.           IN      A

;; ANSWER SECTION:
graph.microsoft.com.    14400   IN      CNAME   ags.privatelink.msidentity.com.
ags.privatelink.msidentity.com. 300 IN  CNAME   www.tm.prd.ags.trafficmanager.net.
www.tm.prd.ags.trafficmanager.net. 300 IN A     40.126.32.99
www.tm.prd.ags.trafficmanager.net. 300 IN A     20.190.160.25
www.tm.prd.ags.trafficmanager.net. 300 IN A     20.190.160.24
www.tm.prd.ags.trafficmanager.net. 300 IN A     20.190.160.97
www.tm.prd.ags.trafficmanager.net. 300 IN A     20.190.160.96
www.tm.prd.ags.trafficmanager.net. 300 IN A     20.190.160.164

;; Query time: 351 msec
;; SERVER: 192.168.1.2#53(192.168.1.2)
;; WHEN: Mon Aug 30 20:53:49 CEST 2021
;; MSG SIZE  rcvd: 232

A sensible approach for a client would be to keep working with an IP address until its TTL expires.
A client respecting that TTL would thus end up sending a new DNS query every 5 minutes, totalling to 12 DNS requests per hour, or 288 per day.
This would be far below Pi-hole's rate limit default of 1,000 DNS request per minute per client (equaling 60,000 per hour or 1,4 million DNS queries a day).

Based on your link, it seems a fair assumption that rclone is exceeding that limit.

Note that you may run into the same observation with other DNS servers as well, as rate limiting DNS requests is common practice for DNS servers (e.g. its around 1,500 queries per second per source IP for Google's DNS servers).

The most effective way to address your issue would be to
(i) find a way to stop rclone from sending such a large amount of unnecessary DNS requests.

You'd have to consult rclone's documentation and support in order to find out if and how that's supported.

If it isn't supported, you'd have two other options:
(ii) Install or enable a caching DNS stub resolver that uses Pi-hole as its only upstream on the client system that runs rclone.

While rclone would still send over thousand identical DNS requests per second, those would hit the stub resolver's cache on the same system before they hit Pi-hole over the network, i.e. the stub resolver would query Pi-hole once every 5 minutes and serve all other requests by rclone for the same domain from its cache.
This has the additional advantage of working with other DNS servers as well, and it should be also a tad bit faster than wiring all queries to Pi-hole.

(iii) Set Pi-hole's RATE_LIMIT to a higher value or disable it completely

See Pi-hole's documentation on Query Rate Limiting for details.

After you made your changes to /etc/pihole/pihole-FTL.conf, don't forget to restart Pi-hole's DNS resolver to apply them, e.g. by running

pihole restartdns
1 Like
5

Thanks for the quick and detailed answer.
I try it in the rclone forum for a solution.
When they have no answer is it bad in anyway to disable the RATE_LIMIT(for security,...)

The other problem is, that so many dns-request distort the pihole dashboard. I mean the %blockt go down.

6

No. We introduced rate limiting to ensure your network cannot be taken down by a single client going crazy overloading the DNS server on your Raspberry Pi. Even when you are using a Rapsberry Pi 4 you should take into account that Pi-hole is that resource-efficient that many still run it on one of the original Raspberry Pis from the very first days - which can obviously do a lot less than your much newer Raspberry Pi 4.

I see, however, this percentage doesn't really mean anything in your case. If you want to have useful numbers on the dashboard, there will be no other way than fixing the issue client-side (i.e. with rclone). This can be done as suggested above by

This seems to be exactly what the last post in the rclone forum recommends to do (before the topic was closed):

Based on packet captures I've looked at, rclone makes a TON of DNS requests for graph microsoft com during operations against OneDrive. The system where you're running rclone should have a local DNS resolver cache, and that cache should bear the brunt of the tens of thousands of requests rclone makes.

edit @daschmidt Maybe you can send the user mlody11 a direct message suggesting him to edit the solution into their original post (or, at least, to select the last post as solution). Otherwise, many users will miss it and will rather add the workaround (disabling rate limiting) instead of properly fixing it client-side.

1 Like
7

Thanks for your replay.

I'm really a amateur with linux. Have you a howto or a somethink for linux or docker.

Yes I writte him a direct message

8

The whole thing didn't leave me alone yesterday, which is why I have now put on a night shift and found a solution that suits me.

On my Synology I installed the DNS-Server with following configuration.

DNS-Server-Config

dnsserver
dnsserver983×557 37.4 KB

and changed the DNS-Server from the Synology to the own ip.

Network

netzwerk
netzwerk776×560 31.9 KB

I don't know if this is a right way or the best, but now it work's for me.

9

Your two screenshots above look identical?

Nevertheless, sounds like you fixed it. :slight_smile:

This would seem to match the enable part of:

10

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.