Docker daemon DNS resolver problem (dockerized pihole)

Hi,

I'm using the official pihole docker image (in combination with cloudflared and dhcphelper).
Everything is working fine: my clients in the "normal" network are getting IPs and correct DNS settings assigned name resolving also works fine here.
The problem: in my other docker containers external (non docker) DNS resolving does not work.

I mapped the port 53 tcp+udp to the docker host machine and added "127.0.0.1" to the daemon.json.
Additionally I added the (static) ip of the docker host as dns entry in the docker-compose files of the other containers. (resolve.conf in the containers unchanged "127.0.0.11").

As I checked the pihole query log I could see the requests there. So the docker daemon did forward them correctly to pihole. And pihole shows no resolve problems (everything green). But the reply seems to never reach the docker daemon:

[resolver] read from DNS server failed, read udp 172.18.0.5:45844->192.168.31.5:53: i/o timeout"

I spend two days on searching for a solution for this problem but I'm out of ideas.

Here are my configs and outputs:

gerneral:
Network 192.168.31.0/24
docker bridge network (self created): docker_main "172.18.0.0/16" -> for all containers to get network access (automatic assigned ips)
docker bridge network (self created): "pihole_pihole" "172.20.0.0/16" -> for communication between pihole and cloudflared (manual assigned ips)

docker host:
ubuntu 20.04 on raspberry pi 4 IP: 192.168.31.5 (static) hostname: xwing
nslookup works:

ubuntu@xwing:/apps/pihole$ nslookup google.com
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
Name:   google.com
Address: 142.250.186.78
Name:   google.com
Address: 2a00:1450:4001:800::200e

pihole:
docker compose:

version: "3.7"

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    networks:
        pihole:
          ipv4_address: "172.20.0.200"
        default: {}
    ports:
      - "53:53/tcp"
      - "53:53/udp"
#     - "192.168.31.5:53:53/udp"
      - "8088:80/tcp"
    expose:
      - "80"
      - "443"
    environment:
      - TZ=Europe/Berlin
      - WEBPASSWORD=PW
      - DNSSEC=True
      - PROXY_LOCATION=pihole
      - VIRTUAL_HOST=pihole.lan
      - VIRTUAL_PROTO=https
      - VIRTUAL_PORT=443
      - VIRTUAL_PORT=80
      - DNS1=172.20.0.201#5053
      - DNSMASQ_LISTENING=all
    # Volumes store your data between container upgrades
    volumes:
       - './data/etc-pihole/:/etc/pihole/'
       - './data/etc-dnsmasq.d/:/etc/dnsmasq.d/'
    dns:
      - 127.0.0.1
      - 172.20.0.201:5053
      - 1.1.1.1
    restart: unless-stopped
    cap_add:
      - NET_ADMIN
    depends_on:
      - cloudflared
      - dhcphelper

  dhcphelper:
    restart: unless-stopped
    container_name: dhcphelper
    network_mode: "host"
    image: homeall/dhcphelper:latest
    environment:
      IP: '172.20.0.200'
    cap_add:
      - NET_ADMIN

  cloudflared:
    image: crazymax/cloudflared:latest
    container_name: cloudflared
    hostname: cloudflared
    networks:
      pihole:
        ipv4_address: "172.20.0.201"
      default: {}
    environment:
      - TZ=Europe/Berlin
      - TUNNEL_DNS_UPSTREAM=https://1.1.1.1/dns-query,https://1.0.0.1/dns-query
    restart: always

networks:
  default:
    external:
      name: docker_main
  pihole:
    ipam:
      driver: default
      config:
        - subnet: 172.20.0.0/24

an other docker container (zwigbee2mqtt):
network is working and we can reach the host with 192.168.31.5 but (external) name resolution is not working (docker internal is working):

/app # ping 192.168.31.5
PING 192.168.31.5 (192.168.31.5): 56 data bytes
64 bytes from 192.168.31.5: seq=0 ttl=64 time=0.409 ms
64 bytes from 192.168.31.5: seq=1 ttl=64 time=0.465 ms
^C
--- 192.168.31.5 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.409/0.437/0.465 ms

/app # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=113 time=27.086 ms
64 bytes from 8.8.8.8: seq=1 ttl=113 time=27.274 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 27.086/27.180/27.274 ms

/app # ping google.com
ping: bad address 'google.com'

/app # ping pihole
PING pihole (172.18.0.4): 56 data bytes
64 bytes from 172.18.0.4: seq=0 ttl=64 time=0.616 ms
64 bytes from 172.18.0.4: seq=1 ttl=64 time=0.536 ms
^C
--- pihole ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.536/0.576/0.616 ms

/app # cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0

docker compose:

version: '3.7'
services:
  zigbee2mqtt:
    container_name: zigbee2mqtt
    image: koenkk/zigbee2mqtt:latest
    dns: 192.168.31.5
    volumes:
      - ./data:/app/data
    devices:
      - /dev/ttyUSB0:/dev/ttyUSB0
    restart: unless-stopped
    environment:
      - TZ=Europe/Berlin

networks:
  default:
    external:
      name: docker_main

I still have no idea why the config above is not working but think I finally found a workaround:
changed dns: 192.168.31.5 to dns: 172.18.0.4 (ip of the pihole container in the docker_main network) -> everything seems to work fine now. :smiley:

Entering the container name here does not work unfortunately.
I added a ipv4_address: "172.18.0.4" in the docker compose of the pihole container - hope that makes it static enough in a network with automatic ip assignment.
Disabling automatic ip assignment for the network docker_main is not an option btw.

It's not - if you take the same workaround make sure you start the pihole container before all the others (after host reboots, or docker daemon restarts, ...)