I have pihole docker container running on a rasperry pi 3 that also hosts a Caddy container to act as my reverse proxy.
When caddy tried to obtain certificates i noticed it was unable to query the LetsEncrypt servers:
{"level":"error","ts":1622658397.9726129,"logger":"tls.obtain","msg":"will retry","error":"[quest.silman.network] Obtain: [quest.silman.network] creating new order: provisioning client: performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:55090->127.0.0.11:53: i/o timeout (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":30.53253729,"max_duration":2592000}
{"level":"warn","ts":1622658467.9983745,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:48364->127.0.0.11:53: i/o timeout"}
...
{"level":"info","ts":1622658824.0147946,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["home.silman.network","*.lan.silman.network","portainer.silman.network","foundry.silman.network","music.silman.network","watch.silman.network","quest.silman.network"]}
{"level":"info","ts":1622658824.032467,"logger":"tls","msg":"cleaned up storage units"}
{"level":"warn","ts":1622658834.0256608,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [*.lan.silman.network]: making OCSP request: Post \"http://r3.o.lencr.org\": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:53246->127.0.0.11:53: i/o timeout"}
{"level":"warn","ts":1622658844.0353408,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [home.silman.network]: making OCSP request: Post \"http://r3.o.lencr.org\": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:52310->127.0.0.11:53: i/o timeout"}
{"level":"warn","ts":1622658854.0544538,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [portainer.silman.network]: making OCSP request: Post \"http://r3.o.lencr.org\": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:45250->127.0.0.11:53: i/o timeout"}
{"level":"warn","ts":1622658864.0741205,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [foundry.silman.network]: making OCSP request: Post \"http://r3.o.lencr.org\": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:59742->127.0.0.11:53: i/o timeout"}
I edited /etc/resolv.conf and changed the nameservers for just the raspberry pi itself to 8.8.8.8 and 4.4.4.4 and after that the caddy container was able to obtain certs.
What is going on here? Why does making PiHole it's own nameserver block outbound DNS queries - thus making it unable to obtain certs from Lets Encrypt?