Just found the real reason this wouldn't work. The machine's /etc/resolv.conf had the following entries
nameserver 192.168.x.x
nameserver 1.1.1.1
options single-request
The first entry is the local address of the machine itself. This came about because there's something on the machine that's fetching the DNS entries from the router and setting it in the conf. The router requires the config for all the hosts to use the correct DNS. Since this machine is one of the hosts (it runs things other than pihole), it gets this DNS that is causing an issue. Manually editing the nameserver line out before starting pihole fixes the problem.
I probably have to automate this in my deployment process meaning that the start script makes sure it removes the namespace before starting pihole and enables it after.
Update: A cursory reading tells me the dhclient automatically updates /etc/resolv.conf. If I edit the file before I start pihole, it will come up. dhclient will then update the conf so all requests from the machine go through pihole (which runs inside a container on the machine).