Debug wireguard connection with PI Hole running in a podman container

The issue I am facing:

Hi,
I've set up a PI-hole container with podman. The container is running rootless and uses a ufw firewall rule to redirect the DNS requests from port 53 to 30053, which is used by the podman container. The setup works well in local network but does not work with a client using a wireguard connection to the PI-Hole.

If I run dig and check the network log in the PI-hole dashboard, I can see that PI-hole got the request from the wireguard IP of the client for the domain. The client on the other side did not get the IP. Instead there is only a timeout.

I already try to test parts of the route. I tested the following things:

  • I tested DNS over wireguard without a container. It works. Therefore I used another system, which installed dnsmasq on the host and uses the same wireguard configuration.
  • Avoiding the port redirection. Therefore I used a system in the local network and used the dig command with a custom port: dig +short @192.168.0.126 -p 30053 A foo.internal. In the local network it works, via wireguard it does not work.
  • Use different network handler for portman. I use the network handler --net=slirp4netns:port_handler=slirp4netns that PI-Hole gets the request from the real IP addresses of the clients, instead a single gateway IP, which is the default. I tested it with the default network configuration. In the local network it works, via wireguard it doesn't. I can also see the request from the different client In logs in the dashboard. The only difference is, that they are bundle in the single gateway IP address.

After all the tests, I thing the problem is the combination of the wireguard network and the podman network if the PI-Hole sends the answer back. But I cannot check this or find out where is the concrete problem. I know the setup is really specific therefore I ask if somebody has an idea how to continue with debugging. Or maybe had the same problem and know the solution :wink:

Best regards,
Simeon

Details about my system:

Host operation system: Ubuntu 24.04.1

Podman container configuration (quadlet):

[Unit]
Description=pihole DNS server and adblocker

[Container]
Image=docker.io/pihole/pihole:latest
ContainerName=pihole-service
AutoUpdate=registry
# DNS Server Port
PublishPort=30053:53/udp
# Dashboard Port
PublishPort=30054:80
# the slirp4netns handler passes the real IP addresses of the clients
# the default Podman network uses NAT, therefore a Client IP is 
# translated from 192.168.x.x to 10.0.x.x
# with the default Podman network, detecting an external device is nearly impossible
Network=slirp4netns:port_handler=slirp4netns
# configure the network interface name for pi-hole
Environment=INTERFACE="tap0"
# needs to be set otherwise INTERFACE="tap0" is not working
Environment=DNSMASQ_LISTENING="all"
Volume=/home/user/pihole/config/pihole:/etc/pihole
Volume=/home/user/pihole/config/dnsmasq:/etc/dnsmasq.d
# Contains custom domains
Volume=/etc/hosts:/etc/hosts:ro
Environment=TZ="Europe/Berlin"
# defines the environment variables: WEBPASSWORD, VIRTUAL_HOST and PIHOLE_DNS_
EnvironmentFile=/home/user/pihole/conf.env

[Service]
# together with WantedBy=multi-user.target default.target starts service at boot
Restart=always
TimeoutStopSec=70

[Install]
# together with Restart=always starts service at boot
WantedBy=multi-user.target default.target

wireguard server config:

[Interface]
Address = 192.168.6.0/24
ListenPort = 51820
PrivateKey = < cat /etc/wireguard/server.key>

PostUp = ufw route allow in on wg0 out on enp0s31f6
PostUp = iptables -t nat -I POSTROUTING -o enp0s31f6 -j MASQUERADE
PostUp = ip6tables -t nat -I POSTROUTING -o enp0s31f6 -j MASQUERADE
PreDown = ufw route delete allow in on wg0 out on enp0s31f6
PreDown = iptables -t nat -D POSTROUTING -o enp0s31f6 -j MASQUERADE
PreDown = ip6tables -t nat -D POSTROUTING -o enp0s31f6 -j MASQUERADE

[Peer]
PublicKey = < public key client >
AllowedIPs = 192.168.6.5/32

wireguard client configuration:

[Interface]
PrivateKey = <cat /etc/wireguard/wgvic.priv>
Address = 192.168.6.5/24
# DNS server of the router
# should be replaced with 192.168.0.126 if PI-Hole is working
DNS = 192.168.0.1
 
[Peer]
PublicKey = < cat /etc/wireguard/server.pub > 
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = wg.example.com:51820
PersistentKeepalive = 15

Port forwarding rule defined in the /etc/ufw/before.rules

# NAT table rules
*nat
:POSTROUTING ACCEPT [0:0]

# Forward traffic from port 53 to port 30053.
-A PREROUTING -p udp --dport 53 -d 192.168.0.126 -j DNAT --to-destination 192.168.0.126:30053

# Forward traffic from port 80 to port 30080.
-A PREROUTING -p tcp --dport 80 -d 192.168.0.126 -j DNAT --to-destination 192.168.0.126:30080

# Don’t masquerade local traffic.
-A POSTROUTING -s 192.168.0.0/24 -j MASQUERADE

# this COMMIT is required a this point
COMMIT

If the request registers in Pi-hole, and Pi-hole's QueryLog shows anything but N/A in its Reply column, then your issue would lie with networking rather than DNS.

Your description reads as if you already narrowed it down to be a 'podman replies to a wireguard originating request' issue.

The one piece of advice that I can think of:
You may want to check your MTU for your wireguard endpoint as well as your podman machine and match your podman' slirp4netns:mtu against that.
If MTUs differ, you may also consider to adjust the MTU in your wireguard configuration.
Use ip route get <wg endpoint IP address> to get hold of the MTU of an endpoint.
Wireguard would default to 1420, but only if it fails to detect its peer endpoint's MTU. If Wireguard and Podman disagree on MTU, you may lose packets.

If that doesn't work, you should consider to also consult forums specialising in podman and wireguard issues, to increase your chances for better and more useful advice.

Thanks for the fast answer. No, the Reply column never shows N/A. Ether it shows IP for a existing domain or NXDOMAIN for a non existing domain. I also think it is a network problem.

I set the MTU of the wireguard configuration on server and client to 1500 and also for the container with slirp4netns:port_handler=slirp4netns,mtu=1500. Unfortunately it didn't solve the problem. I also check if the MTU of the wireguard interface is 1500 with ip a and the slirp4netns MTU with the process tree of the podman container to be sure that I set it correctly. Should be everywhere 1500.

That's an idea. Maybe I can narrow down my problem to wireguard+podman+DNS-server or wireguard+podman+ dnsmasq

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.