Nginx DNS-over-TLS doesn't show external IPs in Log

Hello guys,

I have installed Pihole as an DNS-over-TLS Server and used Nginx as Gateway.

Actually painpoint is to show the external IP in log… so I found @specterzz post:

but it doesn’t work… get the same failure like @nivong does:

Has anybody implement pihole as an DNS-over-TLS Server with " proxy_bind $remote_addr transparent;" Option? Thanks for advice

Is running a DoT gateway really what you want?

You may want to reconsider your motives for running a DoT gateway in front of Pi-hole (as opposed to using an upstream DoT proxy behind Pi-hole).

I do not see the use case for this DoT gateway:

a) for internal name resolution services within your home network, encryption is unneccessary. It just adds additional overhead, slowing down resolution and potentially using more power (apart from the fact that despite running a DoT server, some of your DNS traffic will still be plain DNS, depending on the OSs your are using).

b) when accessing your home network from public internet clients (e.g. your smartphones), you should use a VPN connection that already does encryption, so no need to encrypt DNS on top of that

c) if you do not provide encryption from Pi-hole to its upstream DNS servers, your DNS traffic will be plain DNS the moment it is leaving your Pi-hole anyway.

d) just in case you plan on running a publically available DNS server:
The Pi-hole team strongly discourages any configuration turning Pi-hole into an open resolver.

upstream DoT over Stubby is already implemented

Usecase: since Android 9 you can use your own systemwide DoT-DNS-Server, called private dns.
Google is pushing DoT and I would like to have my own secure DoT and DoH DNS-Server (PiHole)
DOH works great but DoT is my painpoint…

I know that running an unsecured and unencrypted open resolver is not recommended and really dangerous cause of DNS Amplification DDoS Attacks.

Hey Dual-O,

I will create and paste a full script for you. Give me a day or two.

1 Like

I see. Though I think the correct wording would be:
Starting with Android 9, setting a custom DNS server is restricted to those DNS servers supporting DNS-over-TLS, and you no longer can configure individual DNS servers for specific networks.

That is not an additonal option, it is a forced decision, limiting you in your configuration choices.

In addition, it may well force you into a more complicated configuration, though I am not convinced yet that setting up a DoT gateway would be your best option.

Howbeit, you have a valid point here:
Android’s limitation quite probably complicates the setup I would normally like to recommend - i.e. setting up a VPN and only route your roaming devices’ (like smartphones and laptops) DNS through it.
I don’t think the current guide on routing DNS via VPN will work for you. Alerting @jfb whether he is aware of an updated solution.

Meanwhile, if you want to stick to your gateway approach:
If your problem is similar to that of nivong, maybe his solution (i.e. adopting proper firewall and intrusion detection rules) would apply to you as well.

Also, it would probably help to know whether you actually run your Pi-hole in a network environment you control (e.g. your home network) or in some rented cloudspace (where you may be limited in network configuration options and some provider extorts a fee from you for not deleting your data on top of that).

Either way, I fear your problem is not exactly Pi-hole related, but rather a general network configuration issue. You may want to consider searching other forums (e.g. for nginx) for a solution as well.

Thanks for your reply.

DNS via Wireguard VPN is working great. But an always on VPN connection will drain my phone battery a lot. I tested it for a week and it consumes 10-15% of the battery. This is an great working option but I think not the best way for mobile phones…

Yes you are right but I think this community have some great PiHole/Nginx/Stubby/Unbound/Cloudflared specialitsts. So would be great if someone can help :slightly_smiling_face:

My recommendation is to support “DNS X-Proxied-For” -> quick win: DoH, DOT, DNSCrypt, Loadbalancing, dns rulesets with dnsdist.

https://dnsprivacy.org/wiki/m/mobile.action#page/1278004

Hello Christian,
nginx is a very simple/stable way to do DoT and DoH.
I don’t know dnsdist but it sounds like a good solution too.
So you have the same “Problem” with dnsdist in front of PiHole?
Would be great to find a solution for both tools in here.
regards

This (https://tools.ietf.org/html/draft-bellis-dnsop-xpf-04) is the solution, but it‘s only a draft now. Dnsdist support this draft.

This seems to address a problem that @dual-o has already solved by

EDIT: Besides, that “DNS X-Proxied-For” option is not a standard. Its corresponding internet draft has expired in September 2018.

Unfortunately a transparent proxy is not a nice solution, because the return routes have to be adjusted.

PROXY Protocol v2 support could also be a possibility

1 Like

Here is the script you can follow to install Pihole with DoT (NGINX) on VPC.



# ===============================================================================
sudo su

apt-get update && apt-get upgrade -y && apt-get autoremove -y

# Install necessary packages
apt-get install vim nginx certbot resolvconf python-certbot-nginx python3-certbot-dns-cloudflare dialog php7.2-fpm php7.2-zip php-sqlite3 iptables-persistent -y

# (Optional) Install Amazon EC2 Instant Connect
apt-get install ec2-instance-connect

# ===============================================================================
# Create NGINX server block for pihole web interface
# Replace pi.domain.com with your domain

echo "server {
        listen 80;
        listen [::]:80;
        root /var/www/html;
        server_name pi.domain.com;
        autoindex off;
        index pihole/index.php index.php index.html index.htm;
        location / {
                return 301 \$scheme://\$host/admin;
        }
        location ~ \.php$ {
                include snippets/fastcgi-php.conf;
                fastcgi_pass unix:/run/php/php7.2-fpm.sock;
        }
        location /*.js {
                index pihole/index.js;
        }
        location /admin {
                root /var/www/html;
                index index.php index.html index.htm;
        }
        location ~ /\.ht {
                deny all;
        }
}" >> /etc/nginx/sites-available/pihole
# ===============================================================================
# Create Symbolic link for the site

ln -s /etc/nginx/sites-available/pihole /etc/nginx/sites-enabled/pihole 
# ===============================================================================
# Verify configuration and reload NGINX

systemctl reload nginx
# ===============================================================================
# Cloudflare API credentials for certificate verification

echo "dns_cloudflare_email = XXXXXXXXX@XXXXX.XXXXX
dns_cloudflare_api_key = XXXXXXXXXXXXXXXXXX" >> ./cloudflare.ini
# ===============================================================================
# Change permission for cloudfare credentials file

chmod 400 ./cloudflare.ini
# ===============================================================================
# Generate certificate request using Certbot

certbot -i nginx -m admin@domain.com --dns-cloudflare-credentials ./cloudflare.ini --agree-tos --no-eff-email -d *.domain.com
# ===============================================================================
# Install Pihole Server

curl -sSL https://install.pi-hole.net | bash
# ===============================================================================
# Create NGINX Streams directory

mkdir /etc/nginx/streams/
# ===============================================================================
# Create new server block for DNS-Over-TLS Server Block
# We will be requesting the cert later in the stage

echo "upstream dns-servers {
  server [::1]:53;
  server 127.0.0.1:53;
}
server {
  listen 853 ssl;
  listen [::]:853 ssl;
  ssl_certificate /etc/letsencrypt/live/domain.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/domain.com/privkey.pem;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
  ssl_protocols            TLSv1.2 TLSv1.3;
  ssl_ciphers              HIGH:!aNULL:!MD5;
  ssl_handshake_timeout    10s;
  ssl_session_cache        shared:SSL:20m;
  ssl_session_timeout      4h;
  proxy_ssl off;
  #proxy_bind \$remote_addr transparent;
  proxy_pass dns-servers;
}


" >> /etc/nginx/streams/dns-over-tls
# ===============================================================================
# Enable SSL Stapling

echo "ssl_stapling on;
ssl_stapling_verify on;
resolver [::1] 127.0.0.1;" >> /etc/nginx/conf.d/stapling.conf
# ===============================================================================
# Create new NGINX streams directory

echo "stream {
        include /etc/nginx/streams/*;
}" >> /etc/nginx/nginx.conf
# ===============================================================================
# Enable NXDOMAIN response for blocked domains

echo "BLOCKINGMODE=NXDOMAIN" >> /etc/pihole/pihole-FTL.conf
# ===============================================================================
# Restart NGINX and Pihole Server

service nginx restart
systemctl restart pihole-FTL
# ===============================================================================
# Reset Pihole Web Interface password

pihole -a -p
# ===============================================================================
# Verify if DNS-Over-TLS is working or not

# ===============================================================================
# ===============================================================================
# Enabling NGINX Transparency Proxy
# https://www.nginx.com/blog/ip-transparency-direct-server-return-nginx-plus-transparent-proxy/
# ===============================================================================
# ===============================================================================
# Edit DNS-Over-TLS server block at /etc/nginx/streams/dns-over-tls

# Replace upstream IP with Server's IPv6 and IPv4 Interface IP
  server [XXXX:XXXX::XXXX]:53;
  server X.X.X.X:53;

# Add the NGINX Proxy Bind to the server block
  proxy_bind $remote_addr transparent;
# ===============================================================================
# Run NGINX as Root, instead of www-data

sed -i 's/www-data/root/' /etc/nginx/nginx.conf
# ===============================================================================
# Add IPtables marking rules to tag DNS response packets

ip6tables -t mangle -A OUTPUT -p tcp --sport 53 -j MARK --set-xmark 7
iptables -t mangle -A OUTPUT -p tcp --sport 53 -j MARK --set-xmark 7
# ===============================================================================
# Add IP rules to divert DNS response packets to NGINX

# IPv6
ip -6 rule add fwmark 7 lookup 99
ip -6 route add local ::/0 dev lo table 99

# IPv4
ip rule add fwmark 7 lookup 99
ip route add local 0.0.0.0/0 dev lo table 99
# ===============================================================================
# Restart NGINX service

service nginx restart
# ===============================================================================
# Save IP6tables & IPtables rules

ip6tables-save > /etc/iptables/rules.v6
iptables-save > /etc/iptables/rules.v4
# ===============================================================================
# Ignore localhost queries

echo "IGNORE_LOCALHOST=yes" >> /etc/pihole/pihole-FTL.conf
service pihole-FTL restart
# ===============================================================================
# Verify DNS-Over-TLS is working

# ===============================================================================
# Troubleshooting Commands

netstat -tulpane                              # Connection Tracking
pihole -r                                     # Reset/Reconfigure Pihole Server after IP Change
pihole tail                                   # Pihole logs
ip6tables -t mangle                           # IPtables IPv6 Mangle Table
ip6tables -t mangle                           # IPtables IPv4 Mangle Table
tcpdump -n -i any tcp port 853 -w dump.pcap   # TCPDump command to verify DNS-Over-TLS packets
# ===============================================================================

Mod Edit: External link removed and file contents added to post.

The script is wrong in some places.

# ===============================================================================
# Enable NXDOMAIN response for blocked domains

echo "BLOCKINGMODE=NXDOMAIN" | sudo tee /etc/pihole/pihole-FTL.conf

That will add an additional line to the config file and that is not correct. If you want to change the mode then you need to sed in changes to the existing mode configuration line. If it doesn’t exist then you add it.

Same as with

echo "IGNORE_LOCALHOST=yes" >> /etc/pihole/pihole-FTL.conf

Hello @specterzz,
thanks for your help. Unfortunately I have to say that it doesn’t work for me.
When I comment “proxy_bind $remote_addr transparent;” DoT works great but with localhost in log.
When I uncomment there is no dns resolve entry in pihole log. I think I have a firewall / route problem.

troubleshoot

root@DNS:~# ip6tables -t mangle -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A OUTPUT -p tcp -m tcp --sport 53 -j MARK --set-xmark 0x7/0xffffffff
root@DNS:~# iptables -t mangle -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A OUTPUT -p tcp -m tcp --sport 53 -j MARK --set-xmark 0x7/0xffffffff

netstat -tulpane

tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      0          45182636   3581/nginx: master
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      0          45182638   3581/nginx: master 
tcp        0      0 0.0.0.0:853             0.0.0.0:*               LISTEN      0          45182634   3581/nginx: master
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN      998        45085420   18695/pihole-FTL
tcp        0      0 127.0.0.1:8053          0.0.0.0:*               LISTEN      64707      21903662   23544/stubby  

tcpdump

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
14:55:45.882545 IP external-clientr-IP.30567 > external-server-IP.853: Flags [S], seq 2939463429, win 65535, options [mss 1400,sackOK,TS val 1129368 ecr 0,nop,wscale 9,tfo  cookiereq,nop,nop], length 0
14:55:45.882712 IP external-server-IP.853 > external-clientr-IP.30567: Flags [S.], seq 3817759696, ack 2939463430, win 28960, options [mss 1460,sackOK,TS val 950262223 ecr 1129368,nop,wscale 7], length 0
14:55:45.970164 IP external-clientr-IP.30567 > external-server-IP.853: Flags [.], ack 1, win 172, options [nop,nop,TS val 1129387 ecr 950262223], length 0
14:55:45.970470 IP external-clientr-IP.30567 > external-server-IP.853: Flags [P.], seq 1:518, ack 1, win 172, options [nop,nop,TS val 1129387 ecr 950262223], length 517
14:55:45.970514 IP external-server-IP.853 > external-clientr-IP.30567: Flags [.], ack 518, win 235, options [nop,nop,TS val 950262311 ecr 1129387], length 0
14:55:45.984321 IP external-server-IP.853 > external-clientr-IP.30567: Flags [.], seq 1:2777, ack 518, win 235, options [nop,nop,TS val 950262325 ecr 1129387], length 2776
14:55:45.984363 IP external-server-IP.853 > external-clientr-IP.30567: Flags [P.], seq 2777:3456, ack 518, win 235, options [nop,nop,TS val 950262325 ecr 1129387], length 679
14:55:46.012121 IP external-clientr-IP.30567 > external-server-IP.853: Flags [.], ack 1389, win 177, options [nop,nop,TS val 1129392 ecr 950262325], length 0
14:55:46.019810 IP external-clientr-IP.30567 > external-server-IP.853: Flags [.], ack 2777, win 182, options [nop,nop,TS val 1129393 ecr 950262325], length 0
14:55:46.034546 IP external-clientr-IP.30567 > external-server-IP.853: Flags [.], ack 3456, win 188, options [nop,nop,TS val 1129393 ecr 950262325], length 0
14:55:46.037556 IP external-clientr-IP.30567 > external-server-IP.853: Flags [P.], seq 518:611, ack 3456, win 188, options [nop,nop,TS val 1129393 ecr 950262325], length 93
14:55:46.037934 IP external-server-IP.853 > external-clientr-IP.30567: Flags [P.], seq 3456:3714, ack 611, win 235, options [nop,nop,TS val 950262379 ecr 1129393], length 258
14:55:46.038113 IP external-server-IP.853 > external-clientr-IP.30567: Flags [P.], seq 3714:3745, ack 611, win 235, options [nop,nop,TS val 950262379 ecr 1129393], length 31
14:55:46.038149 IP external-server-IP.853 > external-clientr-IP.30567: Flags [F.], seq 3745, ack 611, win 235, options [nop,nop,TS val 950262379 ecr 1129393], length 0
14:55:46.077625 IP external-clientr-IP.30567 > external-server-IP.853: Flags [P.], seq 611:1088, ack 3714, win 193, options [nop,nop,TS val 1129397 ecr 950262379], length 477
14:55:46.077686 IP external-server-IP.853 > external-clientr-IP.30567: Flags [R], seq 3817763410, win 0, length 0
14:55:46.080911 IP external-clientr-IP.30567 > external-server-IP.853: Flags [FP.], seq 1088:1119, ack 3746, win 193, options [nop,nop,TS val 1129399 ecr 950262379], length 31
14:55:46.080954 IP external-server-IP.853 > external-clientr-IP.30567: Flags [R], seq 3817763442, win 0, length 0
14:55:46.082974 IP external-clientr-IP.15785 > external-server-IP.853: Flags [S], seq 3777039265, win 65535, options [mss 1400,sackOK,TS val 1129399 ecr 0,nop,wscale 9,tfo  cookiereq,nop,nop], length 0
14:55:46.083047 IP external-server-IP.853 > external-clientr-IP.15785: Flags [S.], seq 4194415647, ack 3777039266, win 28960, options [mss 1460,sackOK,TS val 950262424 ecr 1129399,nop,wscale 7], length 0

I am looking at the same. There is not traffic on port 53 toward Pi-hole.

The Mangle rule is for traffic coming from Pi-hole back to Ngnix to tag it a the IP Rule/Route will deliver it to Nginx.

If you rewrite the TCPdump then you can see normal UDP 53 traffic but none coming from Nginx.

tcpdump -n -i any tcp port 53 -vv

%edit%

This way you can see if the mangle is present however you need first traffic from Nginx to get some back:

iptables -vL -t mangle

%edit 2%

I got it partly working and I had a expired certificate on my testing Pi.

The thing missing is the import part of having the client IP address being logged and I am going to check my config files for Nginx to see what it could be.

%edit 3%
I can now see the remote address but it not yet routed correctly back. The config in streams contains this line that has to be activated and it has a “” in it which is put in front of the IP address and so making it invalid:

Wrong: proxy_bind \$remote_addr transparent;

Correct: proxy_bind $remote_addr transparent;

%edit 4%

IT’S ALIVE!!!

I had also an error in the IP address in Mangle so no match.

The IP rule and route do not survive a reboot and looking for how make those two lines peramant.

ip6tables -t mangle                           # IPtables IPv6 Mangle Table
ip6tables -t mangle                           # IPtables IPv4 Mangle Table

Should be:

ip6tables -t mangle                           # IPtables IPv6 Mangle Table
iptables -t mangle                           # IPtables IPv4 Mangle Table

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.