Pi-Hole not blocking ads

-Then this is what I got in the prompt in the pi:
pi@raspberrypi:~ $ sudo nmap -sU -p67 --script dhcp-discover 192.168.1.1

Starting Nmap 7.40 ( https://nmap.org ) at 2019-02-22 12:33 PST
Nmap scan report for 192.168.1.1
Host is up (0.00062s latency).
PORT STATE SERVICE
67/udp closed dhcps
MAC Address: 18:31:BF:05:F4:30 (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 3.98 seconds

  • Then I got this:
    pi@raspberrypi:~ $ nslookup doubleck.com
    Server: 127.0.0.1
    Address: 127.0.0.1#53

** server can't find doubleck.com: NXDOMAIN

  • I did the ipconfig /all in windows and rebooted it, I am still seeing ads on https://www.speedtest.net/
    I'll go ahead and do it again after posting this.

  • Then finally:
    pi@raspberrypi:~ $ echo '>stats' | nc localhost 4711
    domains_being_blocked 112523
    dns_queries_today 19164
    ads_blocked_today 6181
    ads_percentage_today 32.253181
    unique_domains 4531
    queries_forwarded 7718
    queries_cached 5265
    clients_ever_seen 20
    unique_clients 13
    dns_queries_all_types 19164
    reply_NODATA 113
    reply_NXDOMAIN 1956
    reply_CNAME 4262
    reply_IP 10787
    privacy_level 0
    status enabled
    ---EOM---

And you made a typo:

This looks good.

Did you intend to lookup doubleclick.com ? The domain you looked up does not exist on the internet.

Hi everyone. I just set up my pi zero as a pi-hole (using unbound for DNS). In any case, I was having a similar issue where I had traffic showing up on my pi-hole, but not everything was going through there. I'm also running an Asus router. Turns out the issue for me was the DDNS setting (WAN Page, DDNS tab) was turned on. Switching "Enable the DDNS Client" to "no" (radio button) and clicking apply. I did reboot the router after applying the setting (although I can't say I remember if I did that manually, or if that was triggered by the "apply" button.) After this, all my traffic is flowing through the pi-hole. Do be aware that you may see an additional entry in the pi-hole logs once you do this from dns(dot)msftncsi(dot)com about every 15 seconds. I actually had to enable telnet on my router and connect to it to change the address to stop that check from happening. Apparently you do NOT want to block that entry on your pi-hole (at least, if you have any Windows clients) as Windows clients apparently use that domain for activation or windows update or some such thing, so disabling it on just the router seems to have done the trick.

Hi @GeorgeT and thanks for your input.
But the issue in this thread is that he Asus router also advertises its own DNS service to the clients alongside the DNS server configured in the DHCP DNS section on the router.
As the DNS service on the Asus router isn't Pi-holed, ads leak through.
You could configure the router's upstream WAN DNS setting to point to the Pi-hole IP address but this screws up the statistics on the Pi-hole web GUI as from Pi-hole point of view, some of the queries originate from the router instead of the clients.
What is output on your Pi-hole for below one (might need to install nmap ... see above) ?

sudo nmap -sU -p67 --script dhcp-discover <ASUS_ROUTER_IP_ADDRESS>

Example with 10.0.0.1 being my Asus router and 10.0.0.2 being Pi-hole:

pi@noads:~ $ sudo nmap -sU -p67 --script dhcp-discover 10.0.0.1

Starting Nmap 7.40 ( https://nmap.org ) at 2019-02-23 12:53 CET
Nmap scan report for 10.0.0.1
Host is up (0.00053s latency).
PORT   STATE SERVICE
67/udp open  dhcps
| dhcp-discover:
|   DHCP Message Type: DHCPACK
|   Server Identifier: 10.0.0.1
|   Subnet Mask: 255.255.255.0
|   Broadcast Address: 10.0.0.255
|   WPAD:
|
|   NetBIOS Name Server: 10.0.0.1
|   Domain Name: dehakkelaar.nl
|   Domain Name Server: 10.0.0.2, 10.0.0.1
|_  Router: 10.0.0.1
MAC Address: 50:46:5D:6C:20:00 (Asustek Computer)

Nmap done: 1 IP address (1 host up) scanned in 3.90 seconds

Hi deHakkelaar,

I had configured the router's upstream WAN DNS setting to my pi-hole, and despite that being there and in the DHCP settings, ads were still getting through. In fact, I shut down the pi running pi-hole completely (which should have left me unable to browse the internet) and was still able to browse the web without a problem. It wasn't until I disabled the DDNS client on my router that all my traffic started routing through the pi-hole, and when I shut the pi-hole pi down after turning off the DDNS client, I could no longer browse the web until I brought the pi-hole pi back online (which is the behavior that I expected.)

So when DDNS Client was enabled, my clients connected to the Asus router were still seeing ads, and were able to get online just fine even when the pi-hole was shut down (which was not the behavior I was expecting). This behavior stopped when I disabled DDNS. Nothing could get online unless the pi-hole is running, and once it was running, everything seems to be working correctly. Note that I have kept both the WAN and DHCP configured to use the pi-hole for DNS, and I am running unbound on the pi-hole for DNS resolution. In any case, it seemed that this issue mirrored the issue I was having where ads were still getting through after my pi-hole was set up, through some kind of a 'back-door' DNS on the Asus router. Changing the DDNS Client setting in my Asus router is what appears to have stopped that from happening, so I just wanted to make others aware that it seems the DDNS Client being enabled is the 'other' DNS being advertised by the router, at least that's how it appears to have worked on my end.

As for nmap, I just tried running it and got a command not found error. (sudo: nmap: command not found).

(Edit: Finally finished coffee and breakfast, so I was able to get back on my pi and get nmap installed. Here's the output:)

Starting Nmap 7.40 ( https://nmap.org ) at 2019-02-23 13:44 EST
Nmap scan report for 192.168.(my-routers-ip)
Host is up (-0.15s latency).
PORT STATE SERVICE
67/udp open dhcps
| dhcp-discover:
| DHCP Message Type: DHCPACK
| Server Identifier: 192.168.(my-routers-ip)
| Subnet Mask: 255.255.255.0
| Broadcast Address: 192.168.200.255
| WPAD:
|
| Domain Name Server: 192.168.(my-pihole-pi-address), 192.168.(my-routers-ip)
|_ Router: 192.168.(my-routers-ip)
MAC Address: B0:6E:BF:B2:E0:C0 (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 5.21 seconds

Edit 2: Also worth mentioning: After I made sure everything was working as expected, I did go back to the router's WAN DNS and added 1.1.1.1 as the second DNS to serve as a fallback if the pi is offline. But this was done after confirming everything was running through the pihole unbound dns.

nmap shows your Asus router is behaving exactly the same as ours, pushing two DNS servers instead of only the Pi-hole address.

The thing is, if resolving the pi.hole domain isnt working on any of the DNS servers configured on the clients, ads dont get blocked.

Lookup against Pi-hole 10.0.0.2:

xbian@avr ~ $ nslookup pi.hole 10.0.0.2
Server:    10.0.0.2
Address 1: 10.0.0.2 noads.dehakkelaar.nl

Name:      pi.hole
Address 1: 10.0.0.2 noads.dehakkelaar.nl

Lookup against my Asus router 10.0.0.1 which doesnt have Pi-hole configured for upstream WAN DNS:

xbian@avr ~ $ nslookup pi.hole 10.0.0.1
Server:    10.0.0.1
Address 1: 10.0.0.1

nslookup: can't resolve 'pi.hole'

Lookup against Google's public DNS server 8.8.8.8:

xbian@avr ~ $ nslookup pi.hole 8.8.8.8
Server:    8.8.8.8
Address 1: 8.8.8.8 google-public-dns-a.google.com

nslookup: can't resolve 'pi.hole'

Ok, now I see what you're saying. I'm going to try my best to describe what I've been experiencing here (as the pi, Linux and pi-hole are all new to me!) :slight_smile: I thought that perhaps I was seeing the router's ip for a DNS server was because of the secondary address (1.1.1.1) I had added. So I went into the router's WAN settings and removed DNS2 (1.1.1.1) leaving just the pihole as DNS1. The pihole is also set to the DNS setting in LAN DHCP Server page. After doing this, running the nmap command gave me the exact same results, showing both the Pi and the router's ip addresses for DNS servers. However, if I shut down the pi running pihole, nothing gets resolved. No clients are able to get online at all. I then had to turn the pihole back on (as I couldn't access my router's admin page over wifi while the pihole pi was down, I'm guessing because even though I use the ip address of the router in my browser, it still seems to try to resolve it as router dot asus or somesuch).

Once I got back into the interface, I turned the DDNS client back on, and once again shut down the pi running pihole. None of my clients lost connectivity. Everything connected fine (though ads were showing up, obviously.) Disabling DDNS client, and again, I couldn't get back into the router's admin page until I brought the pihole back up. Also worth mentioning is that when I first disabled the DDNS client, even the router's 'phone home' routine (the dns dot msftncsi dot com) was showing up on my pihole every 15 seconds or so, so even the internal stuff on the router is no longer using it's internal DNS with the DDNS client disabled. I had to telnet into the router and disable the address for the 'phone home' in nvram to stop the constant connections. I did not block that site on the pihole because doing so apparently causes Windows clients to think they're no longer online, but that's another topic.

Needless to say, once I turned off the DDNS Client, everything seemed to be working as intended. Everything was running through the pihole, and if I shut it down, nothing connected. That led to me to believe that the DNS services were being turned off on the router. Then you had me run nmap! LOL!

So at this point, I believe that with the DDNS Client disabled the Asus is still broadcasting itself as a DNS server, but it doesn't appear to be able to resolve anything. If the DDNS client is enabled or if I have 1.1.1.1 set as the second DNS on the WAN settings, all my traffic can go through the router to get online without issue, regardless of if the pi is up and running. So it seems that disabling the DDNS Client is preventing the Asus internal DNS from resolving, but obviously, this is less than ideal since it is preventing the router from resolving anything on its own, so I can feel certain that everything is being resolved by the pihole/unbound combo with this configuration, but obviously, if a client makes a request on the router's DNS, it won't resolve and that will lead to time outs and pages failing to load.

So after all this, I have temporarily re-added 1.1.1.1 as the secondary DNS on the router's WAN settings, as this allows me to connect to both the router and the internet in general if the pihole is down. I do understand that if a client uses the 1.1.1.1 DNS, that ads won't be blocked, but right now that doesn't seem to be the biggest concern, as everything seems to be running through the pihole as far as I can see in the logs, and none of the clients are seeing ads while the pihole is up and running. This is only my 3rd day with pihole, so if it takes a couple of days to get all the issues ironed out, then so be it (as long as my wife doesn't lose access while I'm working on stuff! lol) I am considering making a second pihole on my network as a backup so that I could completely remove the 1.1.1.1 entry permanently, but that still wouldn't solve the issue of the Asus router advertising itself as a DNS server.

So now I'm following here instead of contributing because I definitely don't want to leave the Asus as a DNS (even if it's not actually functioning as one with the DDNS client disabled and no secondary DNS), as pages failing to load and loss of connectivity are as bad (if not worse) than the ads I'm trying to stop, but I am hoping there's an alternative to having to switch to the pi for DHCP, as I'm concerned about clients losing connectivity if I'm working on the pi (at least while I'm still new to it.)

Anyhow, thank you for showing me that I still have some stuff that needs addressing and that everything is not working as it should, despite it appearing to. Now to see what can be done. Seems I have more to read and learn! :slight_smile:

Apologies in advance for this long post, but I just wanted to report back what I've learned trying to dig further into this issue, and wanted to be thorough. I'm not sure how to best categorize it (DNS vs DHCP issue), but it essentially seems to boil down to this (for Asus routers running Asus firmware), the router is essentially acting as a DNS "proxy".

What I found out - well lots of stuff, but here's the thing I believe to be the most relevant.

RMerlin (developer of Asuswrt-Merlin custom firmware for Asus routers) posted in another forum (I'm still new here, and don't know if linking to other forums is frowned upon, so I just copied his post from the other page. Also note that this post was from 2013, but it seems to accurately describe how the Asus routers are working today, as far as DNS is concerned.)

Asuswrt has the router run as a DNS proxy (that's the short version of it). That field on the DHCP page will usually contain your router's IP (that's the default when it's left empty), which means your clients will get your router's IP as the DNS, and then the router performs all DNS lookups using the DNS obtained either from your ISP, or manually entered on the WAN page. Having this means that your router can easily act as a caching server for DNS queries for your whole LAN, which can improve performances.

So if you wanted to use custom DNS servers, you would have to enter them on the WAN page, while leaving the DNS entry on the DHCP page untouched.

So between my testing and what I've learned about Asus routers, I believe the following to accurately describe the situation.

  • The router will assign DNS Servers to all clients that obtain their ip address from the router's DHCP server. It will always assign its own ip address as one of the DNS servers.
  • If the LAN - DHCP Server page has a DNS Server address, that address will be assigned as the first DNS to its DHCP clients, but it will still always assign itself as the secondary. The pihole's ip address should be assigned here to make sure it is given as the first DNS server to DHCP clients.
  • The router is apparently acting as a DNS "proxy" and will use the DNS servers that are defined on the WAN - Internet Connection page, so the pihole needs to be defined here to ensure that requests made to the router are resolved by the pihole.

My own testing seems to indicate that if there is only the pihole address in DNS1 on the WAN page, and the pihole is not running, the router cannot resolve addresses because it simply doesn't know any other DNS servers unless DDNS Client is turned on, in which case, the built-in DNS "proxy" seems to fall back to the DDNS client for resolving DNS requests.

So from what I've read, it appears that it shouldn't matter that the router is giving its own ip address as the secondary DNS, since, as long as you've pointed DNS1 on the WAN page to the pihole's address, that is the address the router will use to resolve the requests. If I'm understanding this all correctly, then it seems we shouldn't even have to define the pihole ip address on the LAN - DHCP Server page at all, and clients can apparently use only the router's ip for DNS, and as long as the WAN DNS1 points to the pihole ip and there is no secondary address (and DDNS Client is disabled), then the router is going to use only the pihole to resolve DNS requests, and apparently will even cache them for faster serving to clients. Although I would still personally recommend assigning the pihole DNS to the DHCP page so clients query the pihole directly first, and indirectly (through the router) second.

My earlier tests seem to confirm this behavior, though I haven't yet attempted removing the pihole ip from the LAN - DHCP Server page (and leaving it blank there), and trying to just let the router assign itself as the DNS server to all the clients. But my testing does seem to indicate that if the pihole is the only address the Asus DNS "proxy" knows about, then that is the address it will use for resolving DNS requests.

If I am understanding how this Asus DNS stuff works, then I can simply add another pihole on the network and point the router's WAN DNS2 to the new one and that would provide redundancy if one is down.

I have to assume at this point, that the correct(?) way to configure the Asus routers to ensure that all traffic is going through the pihole is to:

  1. Set the DHCP DNS address on the router to the pihole's ip so clients will get that as their primary DNS. This is to ensure the clients at least try to resolve through the pihole directly first.
  2. Set the WAN DNS1 address to the pihole's ip so clients that weren't able to resolve through the pihole directly will 'fall back' to the router's DNS, which is actually getting its requests resolved by the pihole.
  3. You can leave the WAN DNS2 blank, or assign it to another pihole on the network. I currently have 1.1.1.1 to ensure connectivity if the pihole is down. If left blank and the pihole is not running, you will not be able to get online.
  4. Turn off the DDNS Client (WAN - DDNS page)

Clients should show the pihole's ip address as its first DNS and the router's ip as its second.

Here's what I believe is going on -
A client makes a DNS request, it first asks the pihole for the resolution. If, for whatever reason, the pihole doesn't respond or whatever, the client will fall back to the using router's DNS which is asking the pihole to resolve. This behavior can be observed in the logs, as occasionally, when a client makes a request, 2 identical requests show up, with 1 direct from the client ip and 1 from the router ip.

In other words, (if I've understood this all correctly) this configuration actually gives the client a "second chance" at resolving through the pihole since the router is using pihole to resolve. Then, if neither of those resolve, the router's secondary DNS is used (if an address is provided), and finally, if none of those resolve, the router will try through the DDNS Client (if it's enabled.)

I know there's still a chance something might connect through 1.1.1.1 as long as I have it listed there as WAN DNS2, but with all traffic having essentially 2 paths to the pihole, it doesn't appear that anything is getting through to connected clients from 1.1.1.1, at least not yet, and if I do go ahead with a secondary pihole, then I would be comfortable finally getting rid of that 1.1.1.1 once and for all! :smile:

So, as I've mentioned, I'm new to all this, and now it's time for me to ask, am I understanding this correctly? Or am I completely upside-down on this?

(Edit - ran one more test)
I ran one more test,
$ nslookup pi.hole <router_ip>
and
$ nslookup pi.hole <pihole_ip>

Both resolved correctly.

(Edit 2 - fixed a couple of typos)

(Edit 3 - Additional testing)
I did a 'spam' test of
$ nslookup pi.hole <router_ip>
where I just kept running the command and viewing the results.

With 1.1.1.1 as WAN DNS2, sometimes it resolved, sometimes it didn't, which is the expected behavior as the router bounces back and forth between 1.1.1.1 and the pihole. Removing that entry from DNS2, and the command always resolves.

First lets clear up a common misconception, the terms primary, secondary or tertiary DNS server does not imply that always the primary DNS server will be addressed and in case of failure, fallback occurs to the secondary DNS server etc.
On the OS, its just a list of DNS servers to choose from.
Usually there is some logic behind choosing which DNS server to query based upon for example network latency, number of network hops or just the DNS server thats quickest to respond.

When a client does a DNS query, chances are that it will take the path via the router's WAN DNS setting to get to Pi-hole for resolution.
From Pi-hole perspective, this query does not originate from the client but from the router and thus screwing up statistics on the web GUI.
If lucky, the web GUI will display about halve of the DNS queries coming from the clients an the other half from the router.

There is just no way to configure the Asus router in such a way that you get proper name resolution plus stats.
If you find one, please let me know :wink:

I guess I'm not understanding where the statistics is really an issue. Going through my logs, the router's ip address shows up by itself only for internal systems (for example, there's a trendmicro request that is obviously a part of the router's AI Protection). For all other requests logged on the pihole that use the router's address, there is also a second identical request logged showing the correct ip of the actual originating client. So the only thing that seems slightly off about statistics is the fact that there are sometimes 2 entries for the same request with 1 from the router and 1 from the actual client. Certainly this isn't ideal, but it also doesn't appear to have any real negative consequences as the 'real' requests do appear to be getting logged. I don't know if it's logging both requests because the clients have the pihole as one of their DNS entries directly (since the router is still assigning the pihole to clients due to the DHCP DNS settings), but it does appear that all requests are being logged in a way for me to identify from which client the traffic originated.

As far as the DNS order is concerned, I do get that, which is why I want to set up another pihole for a secondary DNS. If there's 2 piholes listed in the WAN, then there's redundancy in case one or the other is down. I'm guessing that setup would have its own issues with statistics since there's no way of knowing which pihole would be queried by any client at any given point in time, so there would be 2 independently logged systems. In any case, if I have only the pihole listed as the WAN DNS1 with nothing in DNS2, and also set the DHCP DNS to the pihole, then all traffic is being routed through the pihole, regardless of if the request went to the router's ip or the pihole's. But as far as I can see in my logs, every request like that (through the router's ip) has 2 separate requests on the pihole so I seem to be able see the correct origin of every request, regardless of if the client went through the router's DNS proxy or not.

Naturally, with 1.1.1.1 as the secondary DNS on the router's WAN page, this behavior isn't consistent (as requests that get through to 1.1.1.1 won't be logged on the pihole, obviously), but without that as part of the equation, literally the only thing that seems 'off' in the statistics are some "duplicate" entries.

Please forgive me, but I guess I'm not understanding how or where there is an issue, and I am trying hard to determine what that issue could be (beyond 2 entries on some requests), as I do want to make sure I've got everything running correctly without having to switch to the pihole for DCHP (at least for the time being.)

So stats get screwed up.
You probably have your router on top in all the stats on the web GUI.

I might not understand this bit.
The logs show you from what IP the query came from right ?
My Asus router doesnt do any trendmicro queries (even with Pi-hole configured for WAN DNS):

pi@noads:~ $ grep trendmicro /var/log/pihole.log
pi@noads:~ $

In the short period I had Pi-hole configured for WAN DNS, the router weaseled its way up into below stats already:

image

Give it up :smiley:
Go activate Pi-hole DHCP :wink:

Just curious, are you also assigning the pihole's ip in the dhcp page (in addition to the wan page) and is DDNS turned off? I found that even with only the pi's ip address in the WAN page, I was still getting ads until I disabled DDNS. In my testing, if DDNS is still running, the trendmicro and that microsoft one I've mentioned before do not show up on the pihole.

On my end, my top 2 permitted domains shown in pihole's dashboard are both trendmicro (one resolves to rgom10-en dot url dot trendmicro dot com while the other resolves to trendmicro dot com dot edgesuite dot net) originating from the router (I believe these would stop if I disabled the router's AI protection.) None of my clients have anything trendmicro installed on them, so I feel confident those are actually originating on the router itself. The most hits I have is 616 on the first trendmico and 308 on the second one. For my top clients, the first ip is my laptop at 2329, with the router's ip coming in at a close 2nd with 2317 and a total of 12404 requests over the last 24 hours.

Ultimately, it does look like the ideal configuration to bypass the Asus router's DNS 'proxy' would be to use the pihole's dhcp server, but I'll need to see if there's a way for there to be a 'backup' in case the pihole is down. Otherwise, short of manually assigning ip addresses on each client, no clients will have any network connectivity (local or otherwise) until pihole is up and running again.

In all honesty, that's the main reason I'm trying to keep dhcp on the router. I figured if something were to go wrong with the pi during a reboot (configuration error or some such on my end), then at least my local clients could talk to each other at a minimum while I work out whatever I might've broken on the pihole end of things. It's also why I've been keeping 1.1.1.1 available on the DNS list on the router, to be absolutely certain that clients can still connect and use the internet even if pihole isn't running. Which is why I also plan to set up a secondary pihole on my network to use as additional DNS server, so all requests will route through one or the other, and if one is offline, the other will 'pick up the slack' so to speak, and the problem of one pihole being down would be invisible to clients.

I've never looked to even see if there is such a thing as a backup (replication?) dhcp server, but if there were such a thing, and it were usable in pihole, then I would feel so much more confident in using pihole for dhcp. Since I'm already planning on putting in a second pihole for backup/additional dns, if that one could also be a 'fall back' dhcp server so if the main one is down, the secondary could temporarily assign ip addresses until the main one comes back online. But honestly, I don't even know if something like that is even possible or exists. I know most networks don't seem to like having 2 dhcp servers on the same network since overlapping ips could be assigned to different clients, but that's the only thing I know of concerning the use of multiple dhcp servers.

See, I have this problem where I can never leave well enough alone! LOL! So I just know at some point, I'm going to be trying something new, and a side effect will wind up taking the pihole down for however long it takes for me to revert or fix whatever the problem is. I'm concerned that if the pihole is providing all the ip addresses, and pihole or the dhcp service isn't running for whatever reason, then I won't be able to ssh into the pi zero to fix the problem (the zero I'm using for pihole is headless.) So my logic is, as long as something else on the network is providing the ip addresses, then as long as the pi zero boots and connects to the network, I should be able to access it through ssh to fix the problems preventing services or whatever from starting.

For my logs, I'm going to try and attach a screenshot to show how I see requests that come from both the router and client at the exact same time. The top-most router entry (akamai) was also called from my laptop, but that entry was on page 2.

So 2329 lookups from the laptop addressing Pi-hole directly are accounted for.
The other lookups the laptop does against the router IP are obscured in the other 2317 lookups from the router where you dont know where coming from.

This is an entirely different subject
But lots of users have two Pi-hole systems for redundancy.
Default time to live for the DHCP leases is 24 hours so plenty of time to turn off DHCP on one Pi and flip it on on the other.
Maybe some leases might expire during the switch over time but the clients dont give up their IP address that easily.

Ohw, I have WAN DNS settings still at default on my router.
So the DNS servers are supplied by my ISP modem.
And DHCP is switched off bc I use Pi-hole for that.
And I dont use/have DDNS.

I have done this and ads still leaked through

Example ???

What does ipconfig /all have to say now about DHCP and DNS servers ?

An nslookup ?