Conditional Forwarding issues w/ tailscale

youngt2 · February 28, 2023, 5:07pm

I have been trying to get reverse lookups to work with my tailscale server.
Currently I have tailscale installed on the same device as my pi-hole, pi-hole running in docker.

Local pi-hole IP: 192.168.9.3
pi-hole tailscale IP: 100.77.70.40
other tailscalepc IP: 100.95.5.25
tailscale dns name: tailxxxxx.ts.net

I have added a custom .conf file with the following lines:

server=/tailxxxxx.ts.net/100.100.100.100
rev-server=100.64.0.0/10,100.100.100.100

This seems to work for normal lookups:

dig tailscalepc.tailxxxxx.ts.net
...
;; ANSWER SECTION:
tailscalepc.tailxxxxx.ts.net. 600 IN    A       100.95.5.25

;; Query time: 8 msec
;; SERVER: 192.168.9.3#53(192.168.9.3)
...

The reponse is correct and pi-hole shows querying quad-100 for this request:

However, I cannot seem to get the reverse lookup to work.

dig -x 100.95.5.25
...
;; AUTHORITY SECTION:
95.100.in-addr.arpa.    10800   IN      SOA     localhost. nobody.invalid. 1 3600 1200 604800 10800

;; Query time: 12 msec
;; SERVER: 192.168.9.3#53(192.168.9.3)

I guess this is because pi-hole does not query quad-100 as my rev-server entry would imply it should (172.18.0.1#5335 is local unbound).

I can do the reverse lookup if I dig to quad 100 directly from the pi-hole device:

dig -x 100.95.5.25 @100.100.100.100
...
;; ANSWER SECTION:
25.5.95.100.in-addr.arpa. 600   IN      PTR     tailscalepc.tailxxxxx.ts.net.

;; Query time: 0 msec
;; SERVER: 100.100.100.100#53(100.100.100.100) (UDP)

Am I missing something obvious?

Just had a thought that maybe pi in docker cannot route to quad-100 properly so it tries the default server to lookup instead. But running the same dig -x 100.95.5.25 @100.100.100.100 from within docker exec pihole I get a successful lookup

DanSchaper · February 28, 2023, 6:45pm

What is your docker-compose.yml or your docker run command? Are you using the Environment Variables noted at GitHub - pi-hole/docker-pi-hole: Pi-hole in a docker container?

How have you mounted that custom.conf file? What path is it mounted in to?

What does the pihole.log show when you try to run the PTR lookup? You can either directly tail the log file or use the admin web interface tools to tail the log in real time.

Bucking_Horn · February 28, 2023, 7:22pm

Just to complement Dan's questions:
I have no difficulties with reverse lookups for my dockered Pi-hole when mimicking your configuration (via rev-server=100.64.0.0/10,192.168.1.200, where .200 is a bare metal Pi-hole): I can see reverse lookups directed at the one dockered Pi-hole show up in the other bare-metal Pi-hole as expected.

EDIT: For prosperity, I'm runnning a Docker Tag 2022.10.

youngt2 · February 28, 2023, 7:36pm

To be clear I am trying to add my tailscale network as an additional rev_server, sorry I did not specify that. I already pass my local lan in env variables for docker and have seen no issue with those reverse lookups.
I am using docker-compose. The pi-hole portion of my compose:

  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    hostname: pi3
    #network_mode: host
    networks:
      - piserver
    ports:
      - 53:53/tcp
      - 53:53/udp
      #- 8888:80/tcp
    environment:
      TZ: $TZ
      WEBPASSWORD: Password1
      FTLCONF_LOCAL_IPV4: 192.168.9.3
      PIHOLE_DNS_: 172.18.0.1#5335
      REV_SERVER: true
      REV_SERVER_DOMAIN: home
      REV_SERVER_TARGET: 192.168.8.1
      REV_SERVER_CIDR: 192.168.0.0/16
      VIRTUAL_HOST: pihole.myreverseproxy.com
      WEBTHEME: "default-auto"
      #INTERFACE: eth0
      #DNSMASQ_LISTENING: all
      CUSTOM_CACHE_SIZE: 0
      FTLCONF_MOZZILLA_CANARY: true
      FTLCONF_BLOCK_ICLOUD_PR: true
      SKIPGRAVITYONBOOT: 1
    volumes:
      - ./pihole/etc-pihole:/etc/pihole
      - ./pihole/etc-dnsmasq:/etc/dnsmasq.d
    restart: unless-stopped
    labels:
      - traefik.enable=true
      - traefik.http.routers.pihole.rule=Host(`pihole.$DOMAINNAME`)
      - traefik.http.routers.pihole.entrypoints=websecure
      - traefik.http.routers.pihole.service=pihole-svc
      - traefik.http.services.pihole-svc.loadbalancer.server.port=80

I mounted a file 98-custom-options.conf to the /etc/dnsmasq.d via the second docker mount seen above.
When viewing the request in the pihole.log I see only:

Feb 28 14:26:57: query[PTR] 25.5.95.100.in-addr.arpa from 192.168.8.10
Feb 28 14:26:57: forwarded 25.5.95.100.in-addr.arpa to 172.18.0.1#5335
Feb 28 14:26:57: reply 100.95.5.25 is NXDOMAIN

I've tried putting the info in my extra config as a docker variable directly (instead of the local lan), so replacing the variables above with:

      REV_SERVER_DOMAIN: tailxxxxx.ts.net
      REV_SERVER_TARGET: 100.100.100.100
      REV_SERVER_CIDR: 100.64.0.0/10

but pi-hole still doesn't seem to use quad-100 when using a reverse lookup.

Feb 28 14:35:51: query[PTR] 25.5.95.100.in-addr.arpa from 192.168.8.10
Feb 28 14:35:51: forwarded 25.5.95.100.in-addr.arpa to 172.18.0.1#5335
Feb 28 14:35:51: reply 100.95.5.25 is NXDOMAIN

I was hoping this would be an easy solution to show the tailscale client names in pi-hole as it works for the LAN perfectly.

DanSchaper · February 28, 2023, 8:18pm

I'm seeing the same unwanted behavior with a similar setup.

@DL6ER I think there's an issue in the forwarding code, I remember there being a discussion previously about non-natural CIDR blocks (one that isn't 8/16/24) but this seems to not work when using 100.64.0.0/8 as well.)

root@09184bcf19b7:/etc/pihole# cat /etc/dnsmasq.d/22-custom.conf
server=/tailxxxxx.ts.net/100.100.100.100
rev-server=100.64.0.0/10,100.100.100.100

[2023-02-28 12:05:28.718 620M] **** new UDP IPv4 query[PTR] query "25.5.95.100.in-addr.arpa" from eth0/192.168.88.185#59988 (ID 126, FTL 111916, src/dnsmasq/forward.c:1766)
[2023-02-28 12:05:28.718 620M] 25.5.95.100.in-addr.arpa is not known
[2023-02-28 12:05:28.722 620M] DNS cache: 192.168.88.185/25.5.95.100.in-addr.arpa is not blocked
[2023-02-28 12:05:28.722 620M] **** forwarded 25.5.95.100.in-addr.arpa to 192.168.88.254#53 (ID 126, src/dnsmasq/forward.c:554)
[2023-02-28 12:05:28.722 620M] **** got upstream reply from 192.168.88.254#53: 25.5.95.100.in-addr.arpa is (NXDOMAIN) (ID 126, src/dnsmasq/rfc1035.c:702)
[2023-02-28 12:05:28.722 620M] Set reply to NXDOMAIN (2) in src/dnsmasq_interface.c:2146
[2023-02-28 12:05:28.723 620M] Skipping detection of external blocking IP for ID 126 as query is PTR

But using the old style server format works.

root@09184bcf19b7:/etc/pihole# cat /etc/dnsmasq.d/22-custom.conf
server=/tailxxxxx.ts.net/100.100.100.100
rev-server=100.64.0.0/10,100.100.100.100

server=/95.100.in-addr.arpa/100.100.100.100

2023-02-28 12:17:11.432 1134M] **** new UDP IPv4 query[PTR] query "25.5.95.100.in-addr.arpa" from eth0/192.168.88.185#57889 (ID 65, FTL 112208, src/dnsmasq/forward.c:1766)
[2023-02-28 12:17:11.433 1134M] 25.5.95.100.in-addr.arpa is not known
[2023-02-28 12:17:11.434 1134M] Checking if "25.5.95.100.in-addr.arpa" is in gravity: no
[2023-02-28 12:17:11.436 1134M] DNS cache: 192.168.88.185/25.5.95.100.in-addr.arpa is not blocked
[2023-02-28 12:17:11.437 1134M] **** forwarded 25.5.95.100.in-addr.arpa to 100.100.100.100#53 (ID 65, src/dnsmasq/forward.c:554)
[2023-02-28 12:17:11.459 1134M] **** new UDP IPv4 query[PTR] query "25.5.95.100.in-addr.arpa" from eth0/192.168.88.185#57889 (ID 66, FTL 112209, src/dnsmasq/forward.c:1766)
[2023-02-28 12:17:11.459 1134M] 25.5.95.100.in-addr.arpa is known as not to be blocked

youngt2 · February 28, 2023, 8:49pm

I hadn't thought to try different rev-servers. I can confirm you method works!

On the plus side, I am able to get the larger /8 block to work.

server=/tailxxxxx.ts.net/100.100.100.100
rev-server=100.0.0.0/8,100.100.100.100

Feb 28 15:40:22: query[PTR] 25.5.95.100.in-addr.arpa from 192.168.8.10
Feb 28 15:40:22: forwarded 25.5.95.100.in-addr.arpa to 100.100.100.100
Feb 28 15:40:22: reply 100.95.5.25 is tailscalepc.tailxxxxx.ts.net

Bucking_Horn · February 28, 2023, 9:08pm

You want to be cautious about actually using that 100.0.0.0/8.

Tailscale's 100.64.0.0/10 is repurposing an IP address range reserved for carrier-
grade NAT, so it's safe to use (potentially unless you actually are behind an ISP's carrier-grade NAT?).

Your enlarged range would comprise public IPs as well, which may or may not cause issues when forwarding to Tailscale's virtual DNS server at 100.100.100.100.

According to dnsmasq docs on rev-server, the previous class limitation has gone:

Allowed prefix lengths are 1-32 (IPv4) and 1-128 (IPv6). If the prefix length is omitted, dnsmasq substitutes either 32 (IPv4) or 128 (IPv6).

Edit: This has been added with dnsmasq 2.87.

It's strange that you can reproduce the issue while I can't, Dan.
I'm using an older Pi-hole image (2022.10), though.

youngt2 · March 1, 2023, 3:08am

Makes sense. This is more of a convenience thing anyway, so I can wait and see if there is a fix. I guess in the mean time I could do lots of smaller blocks-

100.64.0.0/16
100.65.0.0/16
...
...
100.127.0.0/16

Or maybe just add the /16s which tailscale has assigned me an IP

Should I make an issue on github? I did find this PR Include CIDR notation for IPv4 non-natural blocks. by dschaper · Pull Request #1151 · pi-hole/pi-hole · GitHub from a long time ago

DanSchaper · March 2, 2023, 7:17pm

That particular PR is unrelated to this issue.

Please open a new Issue on GitHub - pi-hole/pi-hole: A black hole for Internet advertisements (might end up on the FTL repo though, not sure yet.) Reference this topic as well, you can include a link in your Issue.

DL6ER · March 2, 2023, 7:43pm

This is 100.64.0.0 - 100.127.255.255 so 100.95.5.25 is well within.

This issue should rather been brought up at dnsmasq-discuss@lists.thekelleys.org.uk rather than our Git repositories. You'd eventually be directed this way in the end.

You can see right at the top of /var/log/pihole/pihole.log that this option is not doing what we expect it to do

Mar  2 20:41:42 dnsmasq[2003181]: using nameserver 100.100.100.100#53 for domain 64.100.in-addr.arpa (no DNSSEC)

as only 100.64.0.0/16 is considered when 100.64.0.0/10 was requested.

youngt2 · March 3, 2023, 4:16pm

I tried the older image you mentioned @Bucking_Horn, and the addtional dnsmasq config file with rev-server=100.64.0.0/10,100.100.100.100 works! The pihole.log output is as expected.

Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain tail1de2d.ts.net
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 64.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 65.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 66.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 67.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 68.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 69.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 70.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 71.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 72.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 73.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 74.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 75.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 76.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 77.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 78.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 79.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 80.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 81.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 82.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 83.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 84.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 85.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 86.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 87.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 88.100.in-addr.arpa
Mar  3 10:06:36 dnsmasq[359]: more servers are defined but not logged

Each needed /24 block is added individually.

I tried again with 2022.11 and it did not work. Looks like 2022.10 has the dnsmasq version where the update occurred

Update embedded dnsmasq to v2.87 by @DL6ER in pi-hole/FTL#1449

So that was great luck you happened to be using it... 2022.11 uses the next tagged dnsmasq version

Update embedded dnsmasq to v2.88test3 by @DL6ER in pi-hole/FTL#1469

I tried a quick dnsmasq config file and using rev-server=100.64.0.0/10,100.100.100.100 shows the same output as we saw with it only using 100.64.0.0/16

dnsmasq: started, version 2.89 cachesize 150
dnsmasq: compile time options: IPv6 GNU-getopt no-DBus no-UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset no-nftset auth no-cryptohash no-DNSSEC loop-detect inotify dumpfile
dnsmasq: using nameserver 100.100.100.100#53 for domain 64.100.in-addr.arpa
dnsmasq: reading /etc/resolv.conf
dnsmasq: using nameserver 100.100.100.100#53 for domain 64.100.in-addr.arpa

So maybe the problem is in dnsmasq, but I barely know what I'm doing

Edit: I did the same test with dnsmasq 2.87 and got the same output, so perhaps I'm not doing it right...

DL6ER · March 3, 2023, 4:26pm

I'll try to allocate some debugging time this evening and come back to you with that I find.

youngt2 · March 3, 2023, 4:49pm

For future reference

github.com/pi-hole/pi-hole

conditional forwarding with non-natural CIDR block

opened 04:49PM - 03 Mar 23 UTC

youngt2

### Versions - Pi-hole: v5.15.5 - AdminLTE: v5.18.4 - FTL: v5.21 ### …Platform - OS and version: Ubuntu 22.04.2 LTS - Platform: Docker ### Expected behavior Using a conditional forwarding server of 100.64.0.0/10 should add all the corresponding /16 domains ``` 100.64.0.0/16 100.65.0.0/16 ... ... 100.127.0.0/16 ``` ### Actual behavior / bug dnsmasq only outputs for 100.64.0.0/16 `dnsmasq[2003181]: using nameserver 100.100.100.100#53 for domain 64.100.in-addr.arpa` ### Steps to reproduce Add an additional file to `/etc/dnsmasq.d` directory `98-custom-options.conf` Include `rev-server=100.64.0.0/10` Start pihole. ## Debug Token No specific debug token, but issue is outlined in detail here: https://discourse.pi-hole.net/t/conditional-forwarding-issues-w-tailscale/61522 ### Additional context This issue was resolved in the dnsmasq program with [revision 2.87](https://thekelleys.org.uk/dnsmasq/CHANGELOG#:~:text=SEVFAIL%20rcode%20instead.-,version%202.87,-Allow%20arbitrary%20prefix). Docker tag 2022.10 has the dnsmasq version where the update occurred >Update embedded dnsmasq to v2.87 by [@DL6ER](https://github.com/DL6ER) in [pi-hole/FTL#1449](https://github.com/pi-hole/FTL/pull/1449) The addtional dnsmasq config file with rev-server=100.64.0.0/10,100.100.100.100 using docker 2022.10 works as expected. The pihole.log output is as expected using this image. ``` Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain tail1de2d.ts.net Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 64.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 65.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 66.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 67.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 68.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 69.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 70.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 71.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 72.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 73.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 74.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 75.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 76.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 77.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 78.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 79.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 80.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 81.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 82.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 83.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 84.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 85.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 86.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 87.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: using nameserver 100.100.100.100#53 for domain 88.100.in-addr.arpa Mar 3 10:06:36 dnsmasq[359]: more servers are defined but not logged ``` Each needed /24 block is added individually. 2022.11 uses the next tagged dnsmasq version and does not work. It goes back to the wrong reverse server as shown in the actual behavior above. >Update embedded dnsmasq to v2.88test3 by [@DL6ER](https://github.com/DL6ER) in [pi-hole/FTL#1469](https://github.com/pi-hole/FTL/pull/1469) This may be a dnsmasq bug, not pihole, but additional testing that I'm not sure how to do would be needed.

DL6ER · March 3, 2023, 5:09pm

I found the bug, it was introduced in

The fix is simple

github.com/pi-hole/dnsmasq

Fix --rev-server option.

pi-hole:master ← pi-hole:fix/rev-server

opened 05:08PM - 03 Mar 23 UTC

DL6ER

+6 -0

The `--rev-server` option is broken in the currently released version of `dnsmas…q` for any non-dividable-by-8 CIDR subnets. It was broken in 1db9943c6879c160a5fbef885d5ceadd3668b74d when resolving upstream servers by name was extended to `--rev-server` without accounting for the fact that re-using one and the same upstream server for each of the `x.y.z.in-addr.arpa` is actually a wanted feature.

I will send this to the dnsmasq mailing list for inclusion upstream.