Add "PROXY protocol" Support -> quick win: DoH, DOT, DNSCrypt, Loadbalancing, dns rulesets with dnsdist

christian82 · February 11, 2020, 9:06pm

UPDATE:
"DNS X-Proxied-For" (XPF) is probably outdated, since today (20.03.2020) dnsdist also supports the PROXY protocol.

####################################################################

I have been running my own DNS infrastructure for some time now, including my own recursive DNS servers.
Currently the setup consists of dnsdist (DNS proxy) and unbound.
I have learned to appreciate dnsdist, because it allows to create DoH, DoT, DNSCrypt, load balancing and DNS rulesets very easily.

However, if you use dnsdist in front of pi-hole, only the dnsdist ip address will appear in the pi-hole logs. Since this problem is known (e.g. http Proxy), there is a draft "DNS X-Proxied-For" available. Dnsdist supports this in the newServer command (addXPF), but pi-hole does not understand the "DNS X-Proxied-For" entry.

It would be cool if we could include "DNS X-Proxied-For" in pi-hole, then you could take advantage of dnsdist.

https://tools.ietf.org/id/draft-bellis-dnsop-xpf-02.html

https://blog.powerdns.com/2018/03/30/dnsdist-1-3-0-released/

https://dnsprivacy.org/wiki/m/mobile.action#page/1278004

UPDATE:

Bucking_Horn · February 23, 2020, 1:36pm

Not sure whether it would be prudent to support this.

"DNS X-Proxied-For" is not a standard, not even an RFC yet. Its corresponding internet draft has been around since January 2017 and has expired in September 2018:

christian82 · February 23, 2020, 3:40pm

Since the "DNS X-Proxied-For" approach will probably not be further developed, the proxy protocol (v2) should be implemented in pihole.

dnsdist:

github.com/PowerDNS/pdns

dnsdist: Support proxy protocol for outgoing queries toward backends

opened 12:19PM - 10 Oct 19 UTC

closed 03:58PM - 21 Mar 20 UTC

rgacogne

feature request dnsdist

- Program: dnsdist - Issue type: Feature request ### Short description Usi…ng EDNS Client Subnet to forward the original source IP address to the backend causes several issues: - we can't forward the existing ECS information, if any ; - we need to parse the existing query to determine if there was already an EDNS record, and whether the ECS option was in use. In some cases we need to rewrite it ; - we can only forward the original source address, not the port, or the original destination address / port, and even less arbitrary data ; - altering the initial payload break TSIG signature, and might cause some issues with regard to compression if the EDNS record was not the last one in the DNS packet. XPF fixes a few of these issues but still require some parsing and editing of the packet, and does not provide any way to forward arbitrary data. Using a custom EDNS option would be possible but would still require doing some parsing, with the same issue with regard to compression, TSIG and so on. The proxy protocol [1] has a very simple design, pre-pending some header to the existing data but leaving it untouched otherwise. The second version is very efficient in term of generation and parsing, and allows for arbitrary data to be passed using the optional TLVs mechanism. It would be nice if dnsdist could support pre-pending such a header to the UDP or TCP payload before sending it to the backend. [1]: https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

github.com/PowerDNS/pdns

Add support for Proxy Protocol between dnsdist and the recursor

master ← rgacogne:ddist-proxy-protocol

opened 04:09PM - 27 Feb 20 UTC

rgacogne

+2527 -73

### Short description This PR implements: - outgoing Proxy Protocol (UDP, TC…P, healthchecks) in dnsdist ; - incoming Proxy Protocol (UDP, TCP) in the recursor. It allows dnsdist to pass the initial source and destination addresses and ports to the recursor, along with custom values. Compared to the (ab)use of EDNS Client Subnet, this doesn't prevent preserving an existing ECS value (zeo scope, notably) and provides the ports, along with custom values. Compared to ECS and XPF, this prevents having to parse the existing payload and inject either an EDNS option or a new record, preserving TSIG-signatures. Based on a rebased version of https://github.com/PowerDNS/pdns/pull/8448. Could use a good review :) ### Checklist I have: - [x] read the [CONTRIBUTING.md](https://github.com/PowerDNS/pdns/blob/master/CONTRIBUTING.md) document - [x] compiled this code - [x] tested this code - [x] included documentation (including possible behaviour changes) - [x] documented the code - [x] added or modified regression test(s) - [x] added or modified unit test(s)

Nginx should also support this (proxy_protocol on;)

HAProxy:

DanSchaper · March 1, 2020, 6:02pm

https://dnsdist.org/ is very nice software to front end a DNS server. Not just for this particular request but for all kinds of needs.

christian82 · March 1, 2020, 6:40pm

Unfortunately, dnsdist (or other proxies) is currently not very easy to use with pihole, because the pihole gui only shows the source IP of dnsdist.
Of course you can create a workaround with policy based routing (I don't think it works with dnsdist) but I would like to have an easier solution (I run everything with docker).

I thought XPF was the solution at first, but the proxy protocol is probably more common.

Unfortunately, a few things are still missing in order to make this work in the future.
My DoH setup:
Internet ---(HTTPS,443)---> Haproxy ---(HTTP,80, docker internal)---> dnsdist ---(dns,53, docker internal)---> pihole --(dns,53, docker internal)---> unbound

My DoT setup:
Internet ---(DoT,853)---> dnsdist ---(dns,53, docker internal)---> pihole --(dns,53, docker internal)---> unbound

Missing:
Haproxy ---(HTTP,80, docker internal)---> dnsdist. (only DoH)
- X-Forwarded-For support, only relevant if several services are to be used via port 443.
- dnsdist: dnsdist: DoH in reverse proxy: Accept HTTP Header for indicating real client (address) · Issue #8661 · PowerDNS/pdns · GitHub

dnsdist ---(dns,53, docker internal)---> pihole. (DoH and DoT)
- proxy protocol support
- dnsdist: https://github.com/PowerDNS/pdns/pull/8874
- pihole: nothing found for the proxy protocol yet, therefore this post

Which service would have to be adapted for the proxy protocol for pihole, pihole-FTL or dnsmasq? Which service provides the DNS port 53 for the clients?

christian82 · March 20, 2020, 4:50pm

Now dnsdist 1.5.0 (second release candidate) is available with Proxy Protocol and X-Forwarded-For support. Now pihole should follow.

I have tried to get go-mmproxy running in the pihole docker container, but unfortunately I haven't been able to get it to work. Installation was no problem, but the start with the init system s6-overlay is a problem.
https://github.com/path-network/go-mmproxy

christian82 · September 6, 2020, 10:26am

I would like to use pihole over the internet from the following devices:

iOS, MacOS, Androide, Firtzbox (a very well-known market for DSL-Rotuer in Germany)

Pihole is supposed to run on two virtual servers in the cloud as a docker container.

The use of port udp/53 is out of question from the following reasons:

udp/53 is not encrypted
on udp/53 already runs an authoritative dns server (powerdns), actually it runs behind dnsdist.
DNS Amplification Attack
- In Germany you get a friendly email from the "Bundesamt für Sicherheit" (BSI) asking you to disable the open resolver on port 53.

Therefore only DoH or DoT can be considered. A VPN is not the perfect solution (energy consumption, no redundancy of the dns server), even the Fritzbox can not handle it.

DoH or DoT is (or soon) supported by the devices.
iOS/MacOS: https://www.techradar.com/news/apple-devices-will-get-encrypted-dns-in-ios-14-and-macos-11
Android: Enable Private DNS with 1.1.1.1 on Android 9 Pie
Fritzbox : FritzOS 7.19: Bessere WLAN-Verschlüsselung WPA3 für AVMs Fritzboxen | heise online

But I do not find any possibility to activate DoT or DoH in Pihole.

It is not a problem to activate a DoT/DoH proxy in front of pihole, but the client IP address is lost, you only see the IP address of the proxy in pihole gui.
There are instructions to solve this with routing/route maps, but this is only a very unpleasant workaround. Also in Docker this is difficult to implement (or do you have a good manual for it).

I also don't think the implication of DoT or DoH directly in Pihole is a good solution.
Also note that port 443, 53, 853 is not exclusively used by pihole but by other DNS services, so a DNS load balancer must be installed.
it would make much more sense if the developers had a look at the "Proxy ptrotocol". Then you could simply add haproxy or dnsdist (version 1.5 or higher) in front of it and you would have solved the problem.

https://blog.powerdns.com/2020/07/30/dnsdist-1-5-0-released/
https://dnsdist.org/advanced/proxyprotocol.html

SOLUTION: Implementation of the "Proxy ptrotocol"

PS: With mmproxy I had unfortunately also no success: https://github.com/path-network/go-mmproxy

Tntdruid · September 6, 2020, 11:43am

You can do that whit ZeroTier its end to end encryption.

christian82 · September 6, 2020, 12:37pm

and what does this have to do with pihole and dot/doh?

Bucking_Horn · September 6, 2020, 3:24pm

Neither DNS-over-TLS nor DNS-over-HTTPS do require authentication.

The solution you seem to be striving for will hence result in you running an open resolver, regardless of ports used.

The only reliable way, known to me, to safely access a cloud-based Pi-hole without publically exposing it as an open resolver is by means of a VPN.

As for your other request:

You could explore EDNS(0) to fix this, see Support for add-subnet option from dnsmasq (ECS/EDNS0 Client Subnet).

christian82 · September 6, 2020, 4:26pm

An open resolver is not a problem with DoT/DoH, it is only a problem with udp/53 because a "DNS Amplification Attack" can be performed, as soon as TCP is used the problem does not exist.
For this reason only a authoritative-only name server should run on udp/53.

If you use DoH you can even use a custom URL (I already do), so guessing the URL is difficult.
But even if somebody uses my open resolver over TCP, this is not a security problem.

EDNS I have a closer look at, dnsdist can do that too.
https://dnsdist.org/advanced/ecs.html

christian82 · September 6, 2020, 6:08pm

EDNS works

christian82 · September 8, 2020, 7:52pm

@DanSchaper @DL6ER
Here the desired feature requests for the proxy protocol.

DanSchaper · September 8, 2020, 7:53pm

Don't ping me. I ignore people that do it.

DL6ER · September 8, 2020, 8:07pm

Can you provide some example for what is on the wire when using the proxy protocol? So far, there is nothing about how this is actually realized. The EDNS(0) data is shiped within the very same DNS request so working on this is fairly straightforward. Does "the proxy protocol" fit equally nice into the overall DNS framework?

I have had a quick look at the implementation PR of the procy protocol. It is a +2,527 lines of code change. Unfortunately, the readability of the code adding the proxy protocol to PowerDNS is ... well ... let's restirct ourselves to say that there is not a single line of comments in the source code unit that defines addProxyProtocol. This really makes it hard if not impossible to understand what they are doing there without further reading elsewhere.

Could you provide us with some examples like recorded pcaps so we can check if it is feasible to implement this in FTL? Another question is: What happens if you switch on this feature? Does Pi-hole still work as expected? If so, then they do not interfere much with the original DNS protocol and chances increase that we will be able to implement this in FTL.

I should, nevertheless, also mention that it is currently (and in the foreseeable future) beyond our capabilities to implement this when it turns out to be a lot of work. An external contribution to add this is obviously always welcomed. Note, however, that changes to the dnsmasq code needs to be submitted through the official dnsmasq project to ensure future compatibility so we can continue to easily integrate dnsmasq upstream updates.

christian82 · September 8, 2020, 8:37pm

We need Proxy Protocol v2 because it supports UDP.

Enable Proxy protocol · Cloudflare Spectrum docs
Use the Proxy Protocol to Preserve a Client’s IP Address
https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

I don't have a DNS server that supports the Proxy Protocol v2 with udp, I only use Postfix and Dovecot with the Proxy protocol with TCP.

Does Pi-hole still work as expected? -> No
Since the dns-port expects a proxy-protocol-header when the proxy protocol is enabled, normal clients can no longer connect. Only the proxies (e.g. dnsdist or haproxy) with activated proxy protocol can connect. But this is not bad, because clients in this setup never connect directly to pihole, but always via the proxy.

DL6ER · September 8, 2020, 8:42pm

I have seen your links above. I have also read the technical description which is the third link. However, this does only describe the protocol in a very abstract way. There is no mentioning of how it would integrate with DNS. Hence my question.

If we don't have any examples, it will be close to impossible to realize this.

Let me rephrase my question, maybe I was unclear: Can you record a pcap in the configuration of your proxy in the way you would use it when Pi-hole could read the proxy protocol? I want to see what is sent to Pi-hole by the proxy. That's the only way I can think about how to even start looking at this.

I'm not familiar with any of the tools you are using. If I'd first have to setup a separated network segment with these services ... you see the overhead.

christian82 · September 8, 2020, 9:08pm

The proxy protocol has nothing to do with dns, it is a general approach to keep the client IP address when passing the protocol through a proxy.

Here is an example script for the encoding, I don't have a pcap.

GitHub - ably/proxy-protocol-v2: A simple encoder and decoder for the proxy protocol v2 binary format

hadifarnoud · July 18, 2021, 6:52am

where are we with DNS-over-TLS nor DNS-over-HTTPS support? can I use my pihole as DoH server?

DL6ER · July 18, 2021, 8:07am

Sure, Pi-hole can be used in conjunction with any of these technologies when you want to do this. There are excellent user-provided guides summarized on https://docs.pi-hole.net as well as on this forum.