Website blocked, Found in 1 list "n" which does not exist

Help Community Help
1

Expected Behavior:
Access should not be blocked to www.msftconnecttest.com due to whitelisting (both exacts and regex's). Even if it were, the actual list it is located in should be listed and I should be able to whitelist.

Actual Behavior:
www.msftconnecttest.com is blocked. It is said to be found in 1 of 4 lists. No list is given. Whitelisting it on the block page causes a blue bar to load at the top with no text (as if I whitelisted it). The website is cached and successfully sent over to the device.

Debug Token:
https://tricorder.pi-hole.net/gqusy5pao8

I've confirmed that the whitelist is correct.

I am only using one list, via https://dbl.oisd.nl. The above URLs are not listed in the list.

I posted this question in the community help as I set up cloudflared awhile ago which is working perfect. After having been affected by the Windows 10 NCSI bug where it claims no internet is available even though you do, I wanted to point that service to check my Pi-hole. So I referred to this question (Windows 10 shows Limited internet access but it's connected) where the users jpgpi250 and eejeel discuss on how to point NCSI to check with pi-hole.

I did the following:
1.I created /etc/dnsmasq.d/02-custom_MSncsi.conf.
addn-hosts=/etc/custom_MSncsi.list

2.I created /etc/custom_MSncsi.list.

192.168.1.157 www.msftncsi.com   <-- #.#.#.157 is my pi-hole.
192.168.1.157 ipv6.msftncsi.com
192.168.1.157 www.msftconnecttest.com
192.168.1.157 ipv6.msftconnecttest.com

3.I created /var/www/html/connecttest.txt.
Microsoft Connect Test

4.I created /var/www/html/ncsi.txt.
Microsoft NCSI

5.Restart the DNS server.

Does anyone have any ideas? I am a Linux newbie and am picking it up as I go along. It is working, mostly, as devices don't say "No Internet Access" anymore once I've pointed them to 192.168.1.157. It's just the darn webpage opens up as if I didn't, rinse and repeat. Excuse the extra spaces in URLs above as I need to get around the limit of 5.

2

Some extra images I wasn't able to provide in OP due to URL limit.

For Actual Behavior

image001
image0011263×329 23.3 KB

Whitelisting of URLs

Capture
Capture896×317 13.1 KB

Cached item sent to machine successfully

Capture1
Capture1869×107 5.65 KB

3

(You may avoid URLs being converted to links (and hitting the link limit thus) by applying </> Preformatted text from the menu. I've edited your post accordingly.) :wink:

If your search has indeed been for the full URL, it would always return zero hits.
As a domain blocker, Pi-hole is aware of domains only, not URLs.
Blocking lists for Pi-hole have to conform to HOSTS format, containing only domains or domain to IP associations.

You may use Pi-hole's search utility via Tools | Query Lists to find out which blocklists contain a given domain.

From your description, it is not entirely clear what your issue is.

What webpage are you referring to?

According to your debug log, your network has IPv6 connectivity.
Did you cater for IPv6 as well?

Try to avoid using Pi-hole's public IPv6 address when doing so. Pi-hole is not meant to be publically available, and with a public IPv6, both your IPv6 prefix and the interface identifier are subject to change (the former by your ISP, the latter by IPv6 Privacy Extensions and the likes), and Pi-hole requires a stable address.

Instead of providing mock answers, you could also consider to disable active probing on Windows altogether. Note that machines with multiple network interfaces may sometimes also fail on Windows passive network probing (for further information on Windows active/passive probing, see Windows 10 wifi - internet connectivity keeps dropping ).

4

( You may avoid URLs being converted to links (and hitting the link limit thus) by applying </> Preformatted text from the menu. I've edited your post accordingly. )

Thanks!

If your search has indeed been for the full URL, it would always return zero hits.
As a domain blocker, Pi-hole is aware of domains only, not URLs.
Blocking lists for Pi-hole have to conform to HOSTS format, containing only domains or domain to IP associations.
You may use Pi-hole's search utility via Tools | Query Lists to find out which blocklists contain a given domain.

Sorry for not providing a snippit of my search. Performing a partial search for "msft" shows the domain should not be blocked. I did just edit the exact whitelist for www.msftconnecttest.com to msftconnecttest.com, but I assumed that would've already had been covered by my regex in row 1. I'll see if this helps at all...

 Match found in exact whitelist
   msftconnecttest.com
 Match found in https://dbl.oisd.nl:
   5msft1pbpn.centade.com 
   metrics.dc.ad.msft.net 
   msft-billing.blogspot.com 
   msft-mrm.freewheel.tv 
   msft-updates.com 
   msft.demdex.net 
   msft1pbpn.centade.com 
   msftbilling.com 
   msftbillingsupport.wordpress.com 
   msftmaintenancepage.marketo.com 
   msftoutlook.com 
   tmsftp.osny2p7a0k.com 
   ww16.xmrmsft.com 
   www.msft-billing.blogspot.com 
   www.msft-updates.com 
   www.msftbilling.com 
   www.msftoutlook.com 
   www.xmrmsft.com 
   xmrmsft.com 
   zpvmsftdi8tufgj3awetransfer.000webhostapp.com

From your description, it is not entirely clear what your issue is.

The issue is the "Website Blocked" page that appears whenever the Windows machine queries NCSI over at www.msftconnectest.com. As provided above, it is not located in any block list. I've never seen the behavior of a blocked website followed by something like "found in 1 of 4 lists" but no list is provided, so I can't figure out why this is occurring with Pi-hole. The funny part is whitelisting it there does nothing either as it doesn't exist.

What webpage are you referring to?

www.microsoftconnecttest.com

Did you cater for IPv6 as well?
Try to avoid using Pi-hole's public IPv6 address when doing so. Pi-hole is not meant to be publically available, and with a public IPv6, both your IPv6 prefix and the interface identifier are subject to change (the former by your ISP, the latter by IPv6 Privacy Extensions and the likes), and Pi-hole requires a stable address.

I was reading up on that. I recall a few years back IPv6 domain blocking was broken with Pi-hole. I've seen some user scripts scripts since that help rebind the Pi to your current IPv6 from the ISP. I do have my router set to only serve the Pi as the DNS and nothing else. I use my router as the DHCP server, not the Pi. Could I safely disable IPv6 on the Pi then? Or what would you suggest?

Instead of providing mock answers, you could also consider to disable active probing on Windows altogether. Note that machines with multiple network interfaces may sometimes also fail on Windows passive network probing (for further information on Windows active/passive probing, see Windows 10 wifi - internet connectivity keeps dropping ).

I tried that a few months back before I swept it under the rug for a little while. Every Windows device in my home network ended up reporting No Internet Connectivity with probing off regardless of what was done, so I dropped that option.

5

I'll attempt to explain a bit more in detail why that's not quite an issue.

After all, you are trying to mock some MS websites from within a webserver configured for Pi-hole.

Pi-hole's block page only appears when you try to access http://www.msftconnectest.com through a browser.

Windows probing will not access that URL, but rather try to read some resource content, e.g. from http://www.msftconnecttest.com/connecttest.txt, which should succeed according to your posted configuration.
This would also imply that DNS resoluton is working as intended, i.e. Pi-hole isn't blocking that domain.

When you manually access http://www.msftconnectest.com through a browser, your web server will fail to serve a document for it, since that hasn't been configured.
It will hence generate a 404 error, which is associated with Pi-hole's blocking page.

To mitigate this, you'd have to adopt your webserver for the mock domains you've introduced (or just don't open that top level URL in your browser).

So your above observation is to be expected with your configuration.

In addition, your debug log shows that your router distributes itself as DNS server along with Pi-hole:

*** [ DIAGNOSING ]: Discovering active DHCP servers (takes 10 seconds)
   
   * Received 312 bytes from enxb827eb2f1280:192.168.1.1
     Offered IP address: 192.168.1.157
     Server IP address: 192.168.1.1
     DHCP options:
      Message type: DHCPOFFER (2)
      dns-server: 192.168.1.157
      dns-server: 192.168.1.1
      router: 192.168.1.1
      --- end of options ---

This means that your DHCP clients will bypass Pi-hole via your router as they see fit.
This could also affect Windows NCSI probing if you edited your Windows registry to contain Pi-hole's IP address as suggested by some of the posts you linked (though you don't mention that in your original post).

In any case, Pi-hole has to be your only DNS server for blocking to work reliably.

8

Thanks for the thorough explanation. I think I was hung up on the Pi-hole as the issue only started when a device is using that.

It's slowly coming back to me why I did this. I had thought at the time if I created the mock pages with content that would fix the problem of the browser being opened on login due to ActiveProbing. Obviously, it didn't. Part of the problem is no device on my network can resolve/communicate with dns.msftncsi.com, which was somehow caused by Microsoft back when I first encountered the problem as far back as Windows 10 v1803.

So I removed any and all whitelistings I made for those domains. The issue persisted, but instead the block page showed:

"This site is found in 1 of 4 lists: 
[n]: .wildcard"

Still no go. Tried messing with the pointers in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NlaSvc\Parameters\Internet hoping I could get some different behavior. That was also nill.

I then tried adding DNS A records within Pi-hole for:

dns.msftncsi.com	     192.168.1.157	
www.msftconnecttest.com	 192.168.1.157	
www.msftncsi.com	     192.168.1.157

And it seems that has finally done the trick :crossed_fingers:

In addition, your debug log shows that your router distributes itself as DNS server along with Pi-hole:

*** [ DIAGNOSING ]: Discovering active DHCP servers (takes 10 seconds)
   
   * Received 312 bytes from enxb827eb2f1280:192.168.1.1
     Offered IP address: 192.168.1.157
     Server IP address: 192.168.1.1
     DHCP options:
      Message type: DHCPOFFER (2)
      dns-server: 192.168.1.157
      dns-server: 192.168.1.1
      router: 192.168.1.1
      --- end of options ---

This means that your DHCP clients will bypass Pi-hole via your router as they see fit.
This could also affect Windows NCSI probing if you edited your Windows registry to contain Pi-hole's IP address as suggested by some of the posts you linked (though you don't mention that in your original post).
In any case, Pi-hole has to be your only DNS server for blocking to work reliably.

Fun fact I learned recently is ASUS routers using ASUSWRT also acts as a DNS server (or at least a caching device or similar). However, you can force the WAN and LAN DNS IPs that are handed out, both of which I specified to my pi-hole. I know devices are going where I want because devices that are hardbound like a Roku or Google devices are dealing with blocks I set up and what not. However, I still see that as a problem since it's querying both devices--but not seeing any issues so don't feel like bothering with that now :sweat_smile:

The issue occurred back on my Linksys WRT3200ACM (Linksys is absolute trash now btw) that I had replaced. To my knowledge that had no DNS features like the RT-AC86U.

Thanks for the help! Always glad to have someone to bounce ideas around with.

9

That's a valid configuration.

Just be careful to avoid closing a DNS loop by enabling Conditional Forwarding or by configuring your router as an usptream DNS server for Pi-hole in that scenario. Also, keep in mind that it's harder to attribute stray DNS traffic to individual clients, as your router will be the client for those.

That conclusion is premature: Devices that use a hardcoded DNS server, be it by name or by IP, will query neither Pi-hole nor your router for DNS, but go straight to their encoded public DNS.
The way to stop those would be redirecting or blocking port 53 in your router (or a dedicated firewall device, should you run one of those).

10

Thanks for catching that. I completely forgot about those issues with a sewage backup that occurred yesterday :cry:

It seems the easiest route for me to fix that may be using AsusWRT-merlin (fork) and using the provided DNS filter to direct port 53 traffic to the Pi-Hole. Or there may be easier methods but this gives me many more options and tools to play with.

11

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.