Pi-hole forwarding queries to cloudflared but no internet on clients

The issue I am facing:
No internet
Details about my system:
Pi-Hole with Cloudflared set up on port 5053 as per documentation. In query log I can see that Pi-Hole is forwarding queries to 127.0.0.1#5053 but no internet on clients.
dig query-

<<>> DiG 9.16.1-Ubuntu <<>> 127.0.0.1 -p 5053 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49330
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 1ee60ae5a6c4057a (echoed)
;; QUESTION SECTION:
;google.com.                    IN      A
;; ANSWER SECTION:
google.com.             276     IN      A       142.250.66.238
;; Query time: 40 msec
;; SERVER: 127.0.0.1#5053(127.0.0.1)
;; WHEN: Wed Jan 25 00:09:07 UTC 2023
;; MSG SIZE  rcvd: 77
Jan 25 00:21:54 dnsmasq[3525]: query[A] www.google.com from 192.168.1.128
Jan 25 00:21:54 dnsmasq[3525]: forwarded www.google.com to 127.0.0.1#5053

What I have changed since installing Pi-hole:
NA

debug log removed by moderator and made private

Your example dig command went directly to your Cloudflared instance.

Repeat the same dig. but this time just to the loopback IP without the port, which should send it to Pi-hole.

like this?

dig 127.0.0.1 google.com

; <<>> DiG 9.16.1-Ubuntu <<>> 127.0.0.1 google.com
;; global options: +cmd
;; connection timed out; no servers could be reached

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36535
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             2       IN      A       0.0.0.0

;; Query time: 32 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Jan 25 01:08:35 UTC 2023
;; MSG SIZE  rcvd: 55

The DNS server at the loopback received it, and the domain was blocked. It is unusual that the domain google.com would be blocked by Pi-hole, though.

It's showing that for every domain. If I check Google in upstream DNS in pi-hole clients get internet but this dig query using 127.0.0.1 still shows 0.0.0.0. Is there something else I might have done wrong?
Here is status of cloudflared if that helps-
cloudflared DNS over HTTPS proxy
Loaded: loaded (/etc/systemd/system/cloudflared.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-01-25 02:50:40 UTC; 9s ago
Main PID: 42960 (cloudflared)
Tasks: 8 (limit: 4611)
Memory: 14.3M
CGroup: /system.slice/cloudflared.service
└─42960 /usr/local/bin/cloudflared proxy-dns --port 5053 --upstream https://1.1.1.1/dns-query --upstream https://1.0.0.1/dns-query

Jan 25 02:50:40 orion systemd[1]: Started cloudflared DNS over HTTPS proxy.
Jan 25 02:50:40 orion cloudflared[42960]: 2023-01-25T02:50:40Z INF Adding DNS upstream url=https://1.1.1.1/dns-query
Jan 25 02:50:40 orion cloudflared[42960]: 2023-01-25T02:50:40Z INF Adding DNS upstream url=https://1.0.0.1/dns-query
Jan 25 02:50:40 orion cloudflared[42960]: 2023-01-25T02:50:40Z INF Starting DNS over HTTPS proxy server address=dns://localhost:5053
Jan 25 02:50:40 orion cloudflared[42960]: 2023-01-25T02:50:40Z INF Starting metrics server on 127.0.0.1:35125/metrics

Run that dig command again and observe the reply of 0.0.0.0. Then look at Pi-hole's Query Log and find the entry for google.com. It should be showing pink coloured and in the Status column will say why it is blocked.

If it says "Blocked (regex blacklist)" then it looks like you have added a rule but it is malformed and is catching everything. If you click that status text it will show you the rule.

If it says "Blocked (gravity)" then google.com is on a blocklist, which is unusual and you probably don't want that. You can find which list by searching for google.com in Tools > Search Adlists > enter domain name > Search exact match.

Or it may say something else in which case please advise.

2023-01-26 00:27:10 A www.google.com 192.168.1.30 OK (sent to localhost#5053) N/A

Are there any entries that say "OK (answered by localhost#5053)" for anything?

When you dig using cloudflared directly you get a response, so that seems to be working.

When you dig using Pi-hole the dnsmasq log and query log shows that it gets passed to cloudflared on the correct port. So that also seems to be working. But dig gives you both a "connection timed out; no servers could be reached" error and a NOERROR answer of 0.0.0.0, implying that it did reach a server which returned an all-zero blocked response.

Not sure I understand what's going on.

No entries that say "OK (answered by localhost#5053)" for anything
Could it be something to do with resolv.conf file or systemd-resolved as I've installed on ubuntu inside docker? GitHub - pi-hole/docker-pi-hole: Pi-hole in a docker container

How did you start your container?

  • Using a compose file? (Please post the file);
  • Typing a docker command? (Please post the full command)
  • Using a web interface (like Portainer)?

I'm using CasaOS which has one click deployment of Pi-Hole
Here is a screen shot of container settings-

Figured it out. Maybe the official documentation can be updated for people who want similar set up as mine. Since pi-hole is running in docker & cloudflared daemon on the host system it wasn't working. Solution was to add --address 0.0.0.0 at the end of configuration file so it becomes
CLOUDFLARED_OPTS=--port 5053 --upstream https://1.1.1.1/dns-query --upstream https://1.0.0.1/dns-query --address 0.0.0.0
Restart cloudflared with service restart cloudflared and set the IP of the host as upstream dns your-host-ip#5053