Expected Behaviour:
Remote Wireguard client is able to resolve IPs of machines on same LAN as Wireguard server.
Test case: connecting to the pihole admin console remotely via:
http://192.168.1.92/admin
Actual Behaviour:
Android Client able to connect to pihole admin page via:
http://192.168.1.92/admin
Surface Client unable to connect to pihole admin page via:
http://192.168.1.92/admin
If I reload the page and check the network viewer on firefox:
The
GET request to the above address yields NS_ERROR_CONNECTION_REFUSED in the transferred column.
The surface is however routing it's traffic via Wireguard to the Pihole (see troubleshooting below for evidence of this).
I am stumped as to why my phone is able to resolve my home LAN IPs but my Surface (running Pop-OS) is not.
Details
Server
Hardware: Raspberry Pi 3B
Network connection: Ethernet - Passive switch - Gateway
OS: Raspbian Lite
Pihole: installed via the script on Pihole's website.
Unbound: installed via apt, configured as per pihole docs with DNSSEC enabled.
Wireguard: installed and configured via PiVPN
I have not explicitly performed (ie. done myself) the steps detailed on PiHole Wireguard documentation to make local IPs resolvable (here)
BUT my phone is able to resolve local IPs.
Why/how? I'm not sure.
The changes to /etc/sysctl.d/99-sysctl.conf and nftables definitely have not been made on the Wireguard machine.
Client 1 - Android Phone (aka. Nord)
Android: 11
OS: Oxygen OS 11.0.14
Wireguard: Installed via F-droid
Web browser: Firefox mobile
Client 2 - Microsoft Surface Pro 6 (aka. Surface)
OS: Pop-OS 22.04 LTS
Kernel: surface linux kernel
Web browser: Firefox
Network
IP Glossary
192.168.1.254 Gateway
192.168.1.92 Goose
------------------------------------
192.168.1.92:5123 wg0 |
10.140.235.2 wg0-Nord |
10.140.235.3 wg0-Surface |
192.168.1.92:53 pihole-FTL (DNS) |
192.168.1.92:57 pihole-FTL (DHCP) |
|
127.0.0.1:5335 Ubound interface |
------------------------------------
Network Map
Network Map
Surface:51820
|
Internet
|
192.168.1.254:51820---
Gateway |
| | wg0 - Wireguard
Passive Switch |
| |
192.168.1.92:51820----
Goose |
192.168.1.92:53
|
127.0.0.1:5335
Unbound
Troubleshooting
Firefox - Surface
- DNS over HTTPS is turned off on both clients
- Toggling HTTPS-Only does not change behaviour
- Running DNSLeak test shows only my LAN gateway's external IP on both clients.
Wireguard config - Surface
[Interface]
PrivateKey = <my private key>
Address = 10.140.235.3/24,fd11:5ee:bad:c0de::3/64
DNS = 192.168.1.92 // LAN IP of PiHole
[Peer]
PublicKey = <redacted>
PresharedKey = <redacted>
Endpoint = <DDNS for LAN gateway>
AllowedIPs = 0.0.0.0/0, ::0/0
resolve.conf - Surface
nameserver 127.0.0.53
options edns0 trust-ad
search <subdomain of my workplace>
I initially included this here thinking it would be helpful only to discover the nameserver set to a local loopback. Which is apparently due to the fact that DNS is handled internally by Ubuntu/Debian/Pop-OS? by a different service (specifically systemd-resolved - see troubleshooting below).
Commandline - Surface
- Running
digagainst a test site with the wireguard tunnel active at a location external to my home LAN.
stenolepis@pop-os:~$ dig arstechnica.com
; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> arstechnica.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57348
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;arstechnica.com. IN A
;; ANSWER SECTION:
arstechnica.com. 60 IN A 3.136.132.220
arstechnica.com. 60 IN A 3.12.61.22
;; Query time: 67 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon Aug 28 12:47:56 PDT 2023
;; MSG SIZE rcvd: 76
127.0.0.53#53 is the correct LAN address for Unbound.
- Running
traceroutefrom the Surface Client at a location external to my LAN.
stenolepis@pop-os:~$ traceroute arstehnica.com
traceroute to arstehnica.com (103.224.212.231), 30 hops max, 60 byte packets
1 pi.hole (10.140.235.1) 20.100 ms 20.004 ms 19.913 ms
2 192.168.1.254 (192.168.1.254) 21.487 ms 21.447 ms 21.358 ms
3 10.31.36.1 (10.31.36.1) 22.498 ms 22.491 ms 22.437 ms
4 154.11.12.198 (154.11.12.198) 31.906 ms 31.888 ms 154.11.10.159 (154.11.10.159) 25.907 ms
5 ae3.zayo.mpr1.yvr2.ca.zip.zayo.com (64.125.15.12) 26.918 ms 26.802 ms 26.783 ms
6 ae15.cs2.sea1.us.zip.zayo.com (64.125.30.92) 65.665 ms 82.291 ms 82.237 ms
7 * * *
8 * * *
9 ae1.mcs2.lax112.us.eth.zayo.com (64.125.28.237) 305.755 ms 305.663 ms 305.627 ms
^C
First hop is the pihole via wireguard (10.140.235.1 being the wireguard interface address).
Second hop is my gateway.
- Checking what the system is using for DNS via
resolvectl status
stenolepis@pop-os:~$ resolvectl status
Global
Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
Link 2 (wlp1s0)
Current Scopes: DNS
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS
DNSSEC=no/unsupported
Current DNS Server: 192.168.1.1
DNS Servers: 192.168.1.1
DNS Domain: <sub domain of my work>
Link 7 (wg0)
Current Scopes: DNS
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS
DNSSEC=no/unsupported
Current DNS Server: 192.168.1.92
DNS Servers: 192.168.1.92
DNS Domain: ~.
The address of the DNS server is the LAN address of the Pihole.
DNSSEC=no/unsupported is a little troubling.
- Checking that
systemd-resolvedis indeed what is running
stenolepis@pop-os:~$ systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2023-08-28 12:20:30 PDT; 3h 53min ago
Docs: man:systemd-resolved.service(8)
man:org.freedesktop.resolve1(5)
https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
Main PID: 17137 (systemd-resolve)
Status: "Processing requests..."
Tasks: 1 (limit: 9295)
Memory: 5.0M
CPU: 1.773s
CGroup: /system.slice/systemd-resolved.service
└─17137 /lib/systemd/systemd-resolved
Aug 28 14:43:29 pop-os systemd-resolved[17137]: wlp1s0: Bus client set search domain list to: <subdomain belonging to my work>
Aug 28 14:43:29 pop-os systemd-resolved[17137]: wlp1s0: Bus client set default route setting: yes
Aug 28 14:43:29 pop-os systemd-resolved[17137]: wlp1s0: Bus client set DNS server list to: 192.168.1.1
Aug 28 15:13:07 pop-os systemd-resolved[17137]: wlp1s0: Bus client reset search domain list.
Aug 28 15:13:07 pop-os systemd-resolved[17137]: wlp1s0: Bus client set default route setting: no
Aug 28 15:13:07 pop-os systemd-resolved[17137]: wlp1s0: Bus client reset DNS server list.
Aug 28 16:11:53 pop-os systemd-resolved[17137]: Clock change detected. Flushing caches.
Aug 28 16:12:16 pop-os systemd-resolved[17137]: wlp1s0: Bus client set search domain list to: <subdomain belonging to my work>
Aug 28 16:12:16 pop-os systemd-resolved[17137]: wlp1s0: Bus client set default route setting: yes
Aug 28 16:12:16 pop-os systemd-resolved[17137]: wlp1s0: Bus client set DNS server list to: 192.168.1.1
Pihole admin console (checked via phone)
- Traffic from the Surface Client at the Wireguard interface IP assigned to it is active.
Comments
So my Surface is routing it's webtraffic through Wireguard to the pihole which passes DNS to unbound during regular browsing but I can't access the pihole's admin console via it's LAN IP.
This seems to me like the problem is on the Surface but I can't put my finger on what is going wrong.
However, why I am able to resolve local IPs at all on my phone is puzzling as it seems like I missed some crucial steps.
