DHCP Server does not answer to broadcasts

Stefan_Strableg · December 1, 2023, 5:01pm

The issue I am facing:

I am trying to set up PiHole as a DHCP Server. Its is working perfectly fine as DNS.
The issue is that PiHole doesnt answer to DHCP Discover Packets and clients do not get an IP.

Details about my system:

Im running PiHole on Debian 12. My Routers DHCP is disabled.
PiHoles DHCP is configured to hand out IPs from 192.168.1.200 to 192.168.1.253, just as the router was.

Ive tried using my WiFi extender as an DHCP server whitch works fine.
Ive also used nmap on my PC:

nmap -sU --script=dhcp-discover 192.168.1.101 -p 67 --system-dns
Starting Nmap 7.94 ( https://nmap.org ) at 2023-12-01 17:44 Mitteleuropõische Zeit
Nmap scan report for 192.168.1.101
Host is up (0.0020s latency).

PORT   STATE         SERVICE
67/udp open|filtered dhcps
MAC Address: BC:24:11:12:AC:BF (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 5.47 seconds

On my PiHole, ss shows that port 67 is being listened to by PiHole:

Netid State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
udp   UNCONN 0      0            0.0.0.0:67        0.0.0.0:*    users:(("pihole-FTL",pid=44585,fd=4))

As you can see in my debug report, PiHole also claims its discoverable as a DHCP server.

This is how it looks when using my Extender as a DHCP Server (192.168.1.250):

And this is how it looks when using PiHole (No answer to DHCP DISCOVER):

Heres my debug token: https://tricorder.pi-hole.net/uoCzlooe/

Bucking_Horn · December 1, 2023, 5:14pm

There's a couple of things Pi-hole can do to help you with analysing the issue.

You could verify whether Pi-hole's DHCP server is replying as expected by running the following command from your Pi-hole machine:

sudo pihole-FTL dhcp-discover

In addition, Pi-hole would log DHCPDISCOVER requests as received.
You could monitor and search your pihole.log for those requests:

sudo grep DHCP /var/log/pihole/pihole.log

If no DHCPDISCOVER is listed, that would mean that your client's respective broadcasts never reached your Pi-hole. Note that those broadcasts are same-link only, so if you run a segmented network (VLANs, L3 switches, routers, ...), you'd need one DHCP server per link or a DHCP relay.

Stefan_Strableg · December 1, 2023, 5:43pm

Thank you for your fast answer!

Just as in my debug session, pihole-FTL dhcp-discover prints this:

pihole-FTL dhcp-discover
Scanning all your interfaces for DHCP servers
Timeout: 10 seconds

* Received 300 bytes from eth0:192.168.1.101
  Offered IP address: 192.168.1.216
  Server IP address: 192.168.1.101
  Relay-agent IP address: N/A
  BOOTP server: (empty)
  BOOTP file: (empty)
  DHCP options:
   Message type: DHCPOFFER (2)
   server-identifier: 192.168.1.101
   lease-time: 86400 ( 1d )
   renewal-time: 43200 ( 12h )
   rebinding-time: 75600 ( 21h )
   netmask: 255.255.255.0
   broadcast: 192.168.1.255
   dns-server: 192.168.1.101
   domain-name: "lan"
   router: 192.168.1.1
   --- end of options ---

DHCP packets received on interface eth0: 1

After attempting a connection with my Phone and PC the log still looks like this:

sudo grep DHCP /var/log/pihole/pihole.log
Dec  1 17:32:28 dnsmasq[50203]: compile time options: IPv6 GNU-getopt no-DBus no-UBus no-i18n IDN DHCP DHCPv6 Lua TFTP no-conntrack ipset no-nftset auth cryptohash DNSSEC loop-detect inotify dumpfile
Dec  1 17:32:28 dnsmasq-dhcp[50203]: DHCP, IP range 192.168.1.200 -- 192.168.1.253, lease time 1d

What does same-link only mean? I simply have a router with all my devices connected via the same subnet 192.168.1.x.

deHakkelaar · December 1, 2023, 7:57pm

Its not related to subnets.
Client DHCP discover broadcasts are restricted to whats called the broadcast domain:

Any WiFi extender not in bridge mode will form a boundary for broadcast packets.
Also the network setup for your Proxmox host (which you did not mention?) could form a boundary.

Did you check if a firewall is active on the Pi-hole host, or elsewhere, that might be blocking/dropping?
On which host was the packet sniffing done?
Could you describe your network topology/layout?
And if Pi-hole is running as a Proxmox guest, how is it configured to connect to your LAN (bridge, NAT, routed)?
As I'm not familiar with Proxmox, what "network modes" do you have for your guests?

Ps. above nmap is in unicast instead of broadcast.
Below for broadcast:

sudo nmap --script broadcast-dhcp-discover

But I prefer to let pihole-FTL do the discovery for it can detect if there are multiple DHCP services active in the connected network segment which nmap can not.

Bucking_Horn · December 1, 2023, 9:19pm

Good spotting!

@Stefan_Strableg, your debug log indeed shows your Pi-hole to run on a virtual interface eth0@if8, and its MAC address would imply that it would be managed by some Proxmox virtualisation software.
It is quite likely that Proxmox would isolate a container into a network of its own.

Do you run your Pi-hole in some kind of Proxmox VM?
If so, how did you configure its network?

Stefan_Strableg · December 3, 2023, 5:21pm

I indeed am using Proxmox, which i forgot to mention. Im sorry for that.
PiHole is running on a Debian 12 Proxmox container.

The firewall on my PiHole container is disabled, and its network settings are configured like this:

vmbr0 is the default bridge between the VMs and the network card.

The packet sniffing was done on my PC. My Network Topology looks like this:

In my understanding the vmbr0 bridge should not be an obstacle.

Nmap does not detect any DHCP servers when run from my PC. Pihole-FTL does discover its own DHCP server when run on my PiHole machine.

deHakkelaar · December 3, 2023, 8:54pm

Had a quick read on Proxmox and it seems to be just another Linux stack that we can work with.
But running a DHCP server as a guest seems to need extra attention as it looks like Proxmox default does SNAT for its guests without extra steps.
Which would form a barrier for broadcast packets.
Am not sure though about the SNAT as I dont have Proxmos!

What do below show when run on the Proxmox host itself (might want to redact MAC's):

ip l

ip -br -4 a

ip -4 r

sudo brctl show

sudo nft list ruleset

sudo iptables -nvL -t nat

And do you have another Proxmox x86 Linux guest thats connected to this same vmbr0 bridge?
Does this one get a response when run below in this guest?

wget https://github.com/pi-hole/FTL/releases/download/v5.23/pihole-FTL-musl-linux-x86_64 -O pihole-FTL

chmod +x pihole-FTL

sudo ./pihole-FTL dhcp-discover

Stefan_Strableg · December 4, 2023, 7:18am

Heres the output of the commands on my PiHole Server (MACs redacted):

ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether ##:##:##:##:##:## brd ff:ff:ff:ff:ff:ff link-netnsid 0

ip -br -4 a
lo               UNKNOWN        127.0.0.1/8
eth0@if12        UP             192.168.1.101/24

ip -4 r
default via 192.168.1.1 dev eth0 onlink
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.101

brctl show was not installed by default and also gives no output after installing it.

table
        chain input {
                type filter hook input priority filter; policy accept;
        }

        chain forward {
                type filter hook forward priority filter; policy accept;
        }

        chain output {
                type filter hook output priority filter; policy accept;
        }
}

iptables also was not installed so i assume its not relevant, but it gives this output after installing:

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

I do have another guest on vmbr0 (running WireGuard VPN), and on it pihole-FTL seems to reckognise PiHoles DHCP server. My Routers DHCP Server is enabled again right now, so ignore the first DHCP offer:

Scanning all your interfaces for DHCP servers 
Timeout: 10 seconds                           
                                              
* Received 300 bytes from eth0:192.168.1.1    
  Offered IP address: 192.168.1.100           
  Server IP address: N/A                      
  Relay-agent IP address: N/A                 
  BOOTP server: (empty)                       
  BOOTP file: (empty)                         
  DHCP options:                               
   Message type: DHCPOFFER (2)                
   server-identifier: 192.168.1.1             
   lease-time: 86400 ( 1d )                   
   netmask: 255.255.255.0                     
   router: 192.168.1.1                        
   dns-server: 192.168.1.1                    
   --- end of options ---                     
                                              
                                              
* Received 300 bytes from eth0:192.168.1.101  
  Offered IP address: 192.168.1.202           
  Server IP address: 192.168.1.101            
  Relay-agent IP address: N/A                 
  BOOTP server: (empty)                       
  BOOTP file: (empty)                         
  DHCP options:                               
   Message type: DHCPOFFER (2)                
   server-identifier: 192.168.1.101           
   lease-time: 86400 ( 1d )                   
   renewal-time: 43200 ( 12h )                
   rebinding-time: 75600 ( 21h )              
   netmask: 255.255.255.0                     
   broadcast: 192.168.1.255                   
   dns-server: 192.168.1.101                  
   domain-name: "lan"                         
   router: 192.168.1.1                        
   --- end of options ---                     
                                              
DHCP packets received on interface eth0: 2

So im assuming this means there is some kind of boundary between the Proxmox and the rest of the network. But i can't figure out why or where.

Im not quite familiar with (S)NAT and just read the Wikipedia article, but i dont understand how it can form a barrier for broadcast packets.

deHakkelaar · December 4, 2023, 9:21am

They were suppose to be run on the "Proxmox host itself".
Doesnt that Proxmox host/hypervisor/VMM have a shell or SSH access?

And dont have to install anything if the command isn't installed!

deHakkelaar · December 4, 2023, 9:36am

Oh the broadcast packet doesnt have a source IP address (0.0.0.0).
See the sniffing screenshots?
Its ARP based and you cant NAT ARP bc for NAT you need IP addresses in the table to translate.
In a nutshell

Stefan_Strableg · December 4, 2023, 12:49pm

Well, i should learn to read. Yes, Proxmox does have Shell access.
Heres what the host says:

ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether ##:##:##:##:##:6e brd ff:ff:ff:ff:ff:ff
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether ##:##:##:##:##:6e brd ff:ff:ff:ff:ff:ff
4: veth100i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UP mode DEFAULT group default qlen 1000
    link/ether ##:##:##:##:##:87 brd ff:ff:ff:ff:ff:ff link-netnsid 0
5: fwbr100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether ##:##:##:##:##:fa brd ff:ff:ff:ff:ff:ff
6: fwpr100p0@fwln100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether ##:##:##:##:##:07 brd ff:ff:ff:ff:ff:ff
7: fwln100i0@fwpr100p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UP mode DEFAULT group default qlen 1000
    link/ether ##:##:##:##:##:fa brd ff:ff:ff:ff:ff:ff
13: veth101i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether ##:##:##:##:##:71 brd ff:ff:ff:ff:ff:ff link-netnsid 1

ip -br -4 a
lo               UNKNOWN        127.0.0.1/8
vmbr0            UP             192.168.1.69/24

ip -4 r
default via 192.168.1.1 dev vmbr0 proto kernel onlink
192.168.1.0/24 dev vmbr0 proto kernel scope link src 192.168.1.69

brctl show
bridge name     bridge id               STP enabled     interfaces
fwbr100i0       8000.7266d92418fa       no              fwln100i0
                                                        veth100i0
vmbr0           8000.001999f7db6e       no              enp0s25
                                                        fwpr100p0
                                                        veth101i0

nft list ruleset doesnt display anything.

iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Okay, that makes sense.
Just for clarification: It is not supposed to have an IP yet, is it? Since it does not have an IP address it sends a DHCP DISCOVER packet and waits for an offer from a DHCP server, which it identifies by the Transaction ID it sent in the DISCOVER packet, right?

Im quite a bit lost at this point. What can I do to make PiHoles DHCP Server run on my Proxmox container?

Stefan_Strableg · December 4, 2023, 1:12pm

I just confirmed by running tshark on my Proxmox host: Packets with no source IP are not received, only my Routers DHCP OFFER and DHCP ACK and not my Phones DHCP DISCOVER and DHCP REQUEST. But I dont understand why? What does the Proxmox host need the Source IP for? And for what is ARP used for here?

deHakkelaar · December 4, 2023, 11:45pm

Sorry, ARP has little to do with the DHCP discover broadcast.
I meant to say that its MAC based communication when the client hasn't acquired an IP yet.
NAT can only translate IP's from one network to another and not MAC's, thus forming a boundary.
I dont do this every day

But I dont see any NAT rules applied so thats not it ... I believe.

Correct.

You seem to have two bridges: fwbr100i0 & vmbr0.
My guess is that veth100i0 & veth101i0 are the backend interfaces for the containers.
How many containers do you run?

And can you figure out which veth... interface belongs to which guest?
You might be able to figure out by bringing the Pi-hole guest down and check if one of those interfaces (or maybe another) has disappeared?
If its the veth100i0 interface that belongs to the Pi-hole guest, this fwbr100i0 bridge that its attached to appears to be an isolated network/bridge that needs some other way to connect to your LAN.

If its the veth100i0 interface for Pi-hole, for diagnosing you could detach it from the fwbr100i0 bridge with below:

sudo brctl delif fwbr100i0 veth100i0

And attach it to the vmbr0 bridge with below:

sudo brctl addif vmbr0 veth100i0

And test if the DHCP broadcasts are coming through?
For that you could tail the Pi-hole logs live with below (EDIT: with Pi-hole DHCP enabled):

sudo tail -F /var/log/pihole/pihole.log | grep dnsmasq-dhcp

Or as I'm not familiar with tshark, install tcpdump with (EDIT: Pi-hole DHCP doesnt have to be enabled for this):

sudo apt install tcpdump

And sniff with:

sudo tcpdump -nti any udp port 67

To undo:

sudo brctl delif vmbr0 veth100i0

sudo brctl addif fwbr100i0 veth100i0

EDIT: Oh are you sure the firewall on the Proxmox host isnt blocking UDP 67-68?
Could you post output for below run on that Proxmox host?

sudo iptables -nvL

deHakkelaar · December 5, 2023, 12:15am

Oh just so you know it should work with a bridge setup, below on my Xen hypervisor host with three guests:

dehakkelaar@xen02:~$ sudo brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr0          8000.2e5efe010c32       no              eth0
                                                        vif5.0
                                                        vif6.0
                                                        vif9.0

And below on my Pi-hole guest:

dehakkelaar@ph6b:~$ sudo tcpdump -nti any udp port 67
[..]
eth0  B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 0c:2f:b0:xx:xx:xx, length 308

Stefan_Strableg · December 6, 2023, 2:43pm

I have two containers 100 (WireGuard, veth100i0) and 101 (PiHole, veth101i0).

fwbr100i0 appears to be the Firewall for Container 100.

This shows that the forewall for Container 101 is actually disabled, since if i enable it and run brctl show a fwbr101i0 firewall bridge appears:

bridge name     bridge id               STP enabled     interfaces
fwbr100i0       8000.7266d92418fa       no              fwln100i0
                                                        veth100i0
fwbr101i0       8000.be2965b4191e       no              fwln101i0
                                                        veth101i0
vmbr0           8000.001999f7db6e       no              enp0s25
                                                        fwpr100p0
                                                        fwpr101p0

I have no experience using Proxmox's firewall (pve-firewall),
but running pve-firewall status on the host prints:

Status: disabled/running

And at /etc/pve/firewall/101.fw i found this file:

[OPTIONS]

policy_in: ACCEPT

So the hosts firewall should also be disabled.

Heres the command you asked for run on the host:

iptables -nvL
Chain INPUT (policy ACCEPT 316K packets, 130M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 170K packets, 93M bytes)
 pkts bytes target     prot opt in     out     source               destination

Running tcpdump -nti any udp port 67 gives the same result as tshark, which is that only the the routers DHCP OFFER and DHCP ACK are received on the host (with the Router is running as DHCP Server).

Why does not even the Proxmox host receive the DISCOVER and REQUEST packets?

deHakkelaar · December 7, 2023, 1:41am

Packets 1 and 3 below are the client DISCOVER and REQUEST using broadcast (the "B" before the 0.0.0.0 source IP)

Packets 2 and 4 below are the server OFFER and ACKNOWLEDGE in unicast (IP to IP not broadcast).

pi@ph5b:~ $ sudo tcpdump -nti any udp port 67
[..]
eth0  B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:16:3e:xx:xx:xx, length 300
eth0  Out IP 10.0.0.4.67 > 10.0.0.146.68: BOOTP/DHCP, Reply, length 312
eth0  B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:16:3e:xx:xx:xx, length 300
eth0  Out IP 10.0.0.4.67 > 10.0.0.146.68: BOOTP/DHCP, Reply, length 319

So seems to appear something in Proxmox is bugging broadcasts.
I have no more ideas and am unfamiliar with Proxmox.
Could ask on the Proxmox support forum?

Ps. I found below howto so it looks possible:

Pi-hole can also be configured as a DHCP server, something I suggest you enable as well, as it will most like be more powerful than your present Internet gateway DHCP server.

https://blog.habitats.tech/howto-install-pi-hole-in-proxmox-ve-possibly-the-best-ad-blocker-and-privacy-protector

deHakkelaar · December 7, 2023, 3:27am

Last thought to check if the problem lies in your network setup or on the Proxmox host
Do you have a Linux x86_64 laptop/desktop? EDIT: everyone should
You could try connect it straight into the Proxmox host Ethernet port without a modem/router/switch inbetween.
That way if you assign a static IP on the lap/desktop, you could do the tcpdump/tshark (via SSH) and run pihole-FTL dhcp-discover (on lap/desktop) at the same time.
I assume the Proxmox host has a static IP configured already?

Stefan_Strableg · December 19, 2023, 11:37am

Im busy with other projects at the moment, if I find a solution i will post it here.

Im gonna try plugging my laptop straight into the Proxmox host and if the problem actually lies on the Proxmox host i will continue in their forum.

Thank you for your help!

deHakkelaar · December 19, 2023, 10:54pm

Would love to hear the outcome ... for next time

Another thought that came to mind is the "promiscuous mode" requirement for the bridged interface enp0s25.

promiscuity 1 from below output:

dehakkelaar@xen02:~$ ip -d link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master xenbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 54:b2:03:xx:xx:xx brd ff:ff:ff:ff:ff:ff promiscuity 1  allmulti 1 minmtu 68 maxmtu 9000
[..]

To flip on if not already on:

sudo ip link set enp0s25 promisc on

system · January 9, 2024, 10:55pm

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.