Docker Swarm and Client Count Issues

[MattMan]

Hi, I have been using Pihole on Docker with great success for about 6 months and decided to graduate to swarms. Pihole is my first attempt.

I have successfully created a service/swarm and the two containers are working as expected in relation to providing DNS to the network, fail over etc.

One difference now on the Pihole GUI is the "clients" are only showing as "2" (localhost and 10.255.0.2) any other device does not even show up as an IP address. I have at least 34 devices on my network when viewed with the standard Docker setup so this is purely a swarm issue that I have probably created.
I suspect that the swarm is NATing the real network addresses in some way so they only appear from that one IP.
Can anyone provide a clue as to why this might be the case and how to resolve it?

Here is my run command:
docker service create --name pihole
--dns=127.0.0.1 --dns=1.1.1.1
-p 53:53/udp
-p 443:443/tcp
-p 53:53/tcp
-p 32774:80/tcp
--mount type=volume,src=pihole,dst=/etc/pihole
--mount type=volume,src=dnsmasq.d,dst=/etc/dnsmasq.d
--replicas 2
--env TZ=Australia/Melbourne
--env ServerIP=192.168.4.168
--env DNS1=192.168.4.1
--env DNS2=192.168.4.1
--env WEBPASSWORD=pihole
--env IPv6=False
--mode replicated
pihole/pihole:latest

thanks,

[diginc]

When your LAN IPs don't show up correctly in docker bridge mode you can have better luck getting that working with Bridge or MACVLan modes as described on this page: Docker DHCP and Network Modes - Pi-hole documentation

If these modes don't work for Swarm then I'm not sure exactly what can be done.

[DistractionRectangle]

diginc's solution works for singular pihole containers, swarm mode throws a serious wrench into the works.

The solution for swarms of piholes is fairly involved if you aren't familiar with setting up a transparent reverse proxy, requiring:

• configuring a reverse proxy server with ports bound in host mode
• modifying the pi-hole container; specifically, baking in NAT rules.

If you're on docker for windows, there is no solution.


Anyways...

This runs afoul of moby/moby/25526.

The short of it is this, if you want to use swarm mode you're best bet is setting up a reverse proxy (with ports bound in host mode) and set up the reverse proxy and pihole containers for Direct Server Return. Nginx has a decent tutorial about this here.

I'd suggest following the guide with docker-compose up/down (such that you can use the build directive to bake in the required packages and entrypoint commands) and once you have that working migrate the changes to the pihole image for use with swarm.

[D-o-c]

Dears

if I understood correctly from this discussion and from others on the web, i should be able to see all the clients if I expose the DNS ports with Host mode (pihole is forced to run on a single swarm node) in this way:

 ports:
      - published: 53
        target: 53
        protocol: tcp
        mode: host
      - published: 53
        target: 53
        protocol: udp
        mode: host
      - published: 67
        target: 67
        protocol: udp
        mode: ingress
      - published: 8053
        target: 80
        protocol: tcp
        mode: ingress

Unfortunately, if I expose ports in this way the dns service does not work anymore: I can see the port exposed on the container:

pi@raspy3:~ $ docker port 3be0321961a6
53/tcp -> 0.0.0.0:53
53/udp -> 0.0.0.0:53

but i cannot see them with NETSTAT:

pi@raspy3:~ $ netstat -atu | grep LISTEN
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN
tcp6       0      0 [::]:8053               [::]:*                  LISTEN
tcp6       0      0 [::]:domain             [::]:*                  LISTEN
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN
tcp6       0      0 [::]:https              [::]:*                  LISTEN
tcp6       0      0 [::]:8000               [::]:*                  LISTEN
tcp6       0      0 [::]:9000               [::]:*                  LISTEN
tcp6       0      0 [::]:2377               [::]:*                  LISTEN
tcp6       0      0 [::]:7946               [::]:*                  LISTEN
tcp6       0      0 [::]:http               [::]:*                  LISTEN

Could you help me understanding what I am loosing, please?

Thanks

[Bucking_Horn]

@D-o-c

if I understood correctly from this discussion and from others on the web, i should be able to see all the clients if I expose the DNS ports with Host mode

I don't think so.

Personally not using Docker nor swarm, but I get the following from just reading this discussion:

@DistractionRectangle

swarm mode throws a serious wrench into the works
(...) requiring:

• configuring a reverse proxy server with ports bound in host mode
• modifying the pi-hole container; specifically, baking in NAT rules.

If you’re on docker for windows, there is no solution.

As your configuration does not include any hints at you setting up a modified Pi-hole container or a reverse proxy, I can only assume that's what you are missing.

[DistractionRectangle]

@Bucking_Horn

As your configuration does not include any hints at you setting up a modified Pi-hole container or a reverse proxy, I can only assume that's what you are missing.

A reverse proxy is not strictly necessary to get client IPs//details; it's necessary when you want more than one pihole NAT'd behind a single IP to see the original client IP. For a single node swarm, this is simply an issue with setting up host mode.

Swarm has it's own caveats with host mode, as github issue I linked prior outlines. For a single node swarm, you'll find the workarounds listed in that issue handy - unfortunately, network config via yml just isn't supported via swarm, so your up/down process won't be as clean. A reverse proxy might be a nice to have as you could manage it separate from the swarm and simply attach the swarm managed containers to the same network as the reverse proxy.

[deHakkelaar]

@D-o-c

but i cannot see them with NETSTAT:

pi@raspy3:~ $ netstat -atu | grep LISTEN
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN
tcp6       0      0 [::]:8053               [::]:*                  LISTEN
tcp6       0      0 [::]:domain             [::]:*                  LISTEN

Something is listening on the "domain" port:

pi@noads:~ $ cat /etc/services
[..]
domain          53/tcp                          # Domain Name Server
domain          53/udp

Try below to see who:

sudo netstat -nltup

[_popeofmope]

I ran into this issue as well and was able to resolve it with MACVLAN routing. I walkthrough the setup in my blog post, Docker Swarm + MACVLAN

/cc @MattMan

[D-o-c]

solved with a flash of genious last night! It was only an issue with the "Interface listening behavior" setting. I had to choose "listed on all interface, permit all origins" (I think when I expose something on host in swarm it changes the interface in the container).
Anyway i will give a go also to the _popeofmope solution (I cannot yet mention you).

With the new version I have an issue with DHCP but that's another topic

Thanks to all

[_popeofmope]

@D-o-c, I made a post in the FAQ/How-to, feel free to post and questions or feedback over there

High Availability Pi-hole via Docker Swarm and GlusterFS

I came across a few posts that deal with issues surrounding multiple Pi-hole instances that my high availability implementation seem to resolve. Instead of hijacking the posts, which I already tried to do, I figured I'd create a new one for my solution. I use Docker Swarm to replicate the Pi-hole service and MACVLAN routing to ensure proper communication between client and Pi-hole services. To keep settings, lists, etc in sync, I use bind mounts backed by GlusterFS. The solution is working wel…