Pihole stopped seeing clients, blocking adds, suddenly on 9/13

moose_homans · September 15, 2019, 8:26pm

Hello! Welcome to my nightmare.

Please follow the below template, it will help us to help you!

You bet!

Expected Behaviour:

The following set up has been working for about 4 months.

Router: Comcast XFINITY combo-router/modem monstrosity.
Comcast router is running the DHCP server.
Client devices that utilize the pihole have reserved IP addresses on the Comcast DHCP router
Pihole is running on a Raspberry Pi 3b
Pihole's raspberry also has a reserved IP address on the Comcast router
Unfortunately, there is no way to redirect the Comcast router's DNS settings (arg!)
So client devices have their individual DNS settings manual set to point directly at the Pihole's local IP address, and their IP addresses assigned statically.
Clients are Windows 10 devices and iOS devices. Note: some stubborn members of my family refuse to use the pihole, so there are some clients that get their DNS automatically.
Until 9/13 (Friday the 13th) clients with manually assigned DNS were showing up as clients in the pihole query log, and blocking was successful.
I know this is a very sub-optimal setup, but please have mercy. It worked for a long while!

Actual Behaviour:

(TL;DR: Pihole only sees and blocks queries originating from the raspberry pi it is running on, plus localhost. I did a bit of troubleshooting to confirm other clients are in fact pointed at the Pihole.)

I will focus on one known-good Windows 10 client (KGWin10), as iOS provides no real opportunities for troubleshooting:

Query log after 9/13 no longer shows queries arriving from any clients other than the raspberry pi the pihole is running on (and its localhost), including KGWin10.

1a) I was an idiot and performed a reconfig repair before creating backups of my query logs. >.<

I confirmed KGWin10 still had the correct DNS address manually assigned in the Windows network adapter configuration settings.
When the raspberry pi is shut down and after I flush KGWin10's dns resolver cache, KGWin10 can no longer ping domain names (e.g., "ping www.google.com" fails) or resolve domain names in web browsers.
However, when raspberry pi is booted and pihole is running, no activity from KGWin10 is logged in the query log and no sites are blocked. (I visit site like facebook.com and click on sponsored posts. None of that activity shows up in the pihole query log.)
I first tried to get IPv6 settings lined up. May have made a mess of the raspberry's IPv6 routing table.
But I have disabled the IPv6 protocol in the network connection properties of KGWin10 so it is only running IPv4, and am encountering the same behavior.
I have performed a repair and then a reconfig pihole -r, no change.

Debug Token:

https://tricorder.pi-hole.net/a7xljn1jgo

Any advice, other than telling me to give Comcast the boot, is most welcome!

technicalpyro · September 15, 2019, 11:42pm

[✓] accountidinfo.info is 0.0.0.0 via localhost (127.0.0.1)
[✓] accountidinfo.info is 0.0.0.0 via Pi-hole (10.0.0.96)
[✓] doubleclick.com is 172.217.9.78 via a remote, public DNS server (8.8.8.8)

this shows that your pi-hole is resolving things and blocking things correctly

likely your router is handing out a second DNS OR ads are coming through IPv6 not giving correct DNS to clients

moose_homans · September 16, 2019, 3:12am

Can ads come through IPv6 when Internet Protocol Version 6 is disabled on the only active network adapter? (see attached screenshot.)

Likewise, will the network adapter still accept a handed-out DNS when the IPv4 configuration is set to manual with the settings in this screen capture?

EDIT: Here's the ipconfig /all from the PC. The 10.0.0.96 is the raspberry's eth0 local address. It's the only DNS listed, so I am confounded.

If DNS as be handed out without being identified by Windows, is there any manual configuration that will, well, stick?

Thanks!

technicalpyro · September 16, 2019, 3:19am

I'm not sure how it is happening in your network perhaps you have an anti virus you use that forces its own DNS or something else what i can tell you though it is not the pi-hole itself

jfb · September 16, 2019, 3:24am

I don't think the problem is at the Pi-Hole end. Something has changed in the way your router is distributing DNS to clients.

Your Pi-Hole debug log shows that in the 24 hours prior to the time shown in this output, Pi-hole received that traffic (3 clients):

   [2019-09-15 10:28:05.999 1034]  -> Total DNS queries: 170
   [2019-09-15 10:28:05.999 1034]  -> Cached DNS queries: 83
   [2019-09-15 10:28:05.999 1034]  -> Forwarded DNS queries: 78
   [2019-09-15 10:28:05.999 1034]  -> Exactly blocked DNS queries: 9
   [2019-09-15 10:28:05.999 1034]  -> Unknown DNS queries: 0
   [2019-09-15 10:28:05.999 1034]  -> Unique domains: 34
   [2019-09-15 10:28:06.000 1034]  -> Unique clients: 3
   [2019-09-15 10:28:06.000 1034]  -> Known forward destinations: 3

moose_homans · September 16, 2019, 3:40am

Maybe the problem is in my Rasbian install? I ask because I have evidence of Pihole thinking it is blocking a query when, as far as I can tell (which isn't much), it in fact isn't.

This is the audit log of my pihole. I used VNC to ping from the localhost to view(dot)atdmt(dot)com(dot)26847(dot)9201(dot)302br(dot)net, which is a blocked site.

It looks to me like the pihole says it was blocked, but the ping kept returning data.

I am an utter amateur and may be completely misinterpreting this, though.

jfb · September 16, 2019, 3:44am

Pi-Hole can still ping a blocked domain. The test is to dig the domain - if Pi-Hole returns 0.0.0.0 it is blocked. Example:

dig +short flurry.com
0.0.0.0

ping -c3 flurry.com
PING flurry.com (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.151 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.158 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.141 ms
--- flurry.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 86ms
rtt min/avg/max/mdev = 0.141/0.150/0.158/0.007 ms

moose_homans · September 16, 2019, 3:47am

Yep! Alas, those three clients were effectively Me, Myself, and I: localhost (127.0.0.1), raspberrypi (10.0.0.96, the local network address for the raspberry pihole runs on), and another localhost which the pihole is calling " client ::1".

MeMyselfandI

jfb · September 16, 2019, 3:49am

It appears that Pi-Hole is correctly resolving any traffic it receives, but the traffic isn't getting to Pi-Hole. That would indicate a problem in the router or client. From a connected client, run nslookup pi.hole and see what comes back. If Pi-Hole is the DNS server, then pi.hole is the answer, served by the IP of the Pi-Hole. If not, the IP of the server will tell you which DNS server answered the request.

Example of a request from my Mac. The Pi-Hole is at IP 100

nslookup pi.hole
Server: 192.168.0.100
Address: 192.168.0.100#53
Name: pi.hole
Address: 192.168.0.100

moose_homans · September 16, 2019, 3:55am

Ah! Gotcha. Of course, you are correct. When I dig from localhost, I get the 0.0.0.0 I expect.

Pihole is an awesome piece of software IMO. I've been taking all the utility (with minimal effort compared to most of my network DIY projects) I got from it for granted.

moose_homans · September 16, 2019, 4:12am

The plot thickens! Odds of an 1D10T error are rapidly approaching 1.

From KGWin10...

C:\>nslookup pi.hole
Server:  UnKnown
Address:  10.0.0.96

*** UnKnown can't find pi.hole: Non-existent domain

But the pihole's hostname is raspberrypi, and when I nslookup raspberrypi...

C:\>nslookup raspberrypi
Server:  UnKnown
Address:  10.0.0.96

Name:    raspberrypi
Address:  10.0.0.237

...and right now I am logged onto the same instance on pihole from 2 different local IP addresses.

That...can't be a good thing. Let me see what the heck I told my router to do...

moose_homans · September 16, 2019, 2:53pm

I most certainly did not do that.

I had both eth0 and wlan0 active on my raspberry. Probably did not mean much, but I disabled wlan0.

I cleaned things up a bit, changing the IP address of eth0 to 10.0.0.99, but the debug log shows things are still working just peachy. Even better than before, in fact, because now all the pihole's IPv6 tests are passing on top of everything else passing:

https://tricorder.pi-hole.net/4adofgop6f

Nevertheless, on my client, nslookup shows the pihole's address for the DNS server, but the Server Name is UnKnown and the client doesn't recognize pi.hole as a domain. Which is indicative of something on my network being screwy, not pihole, correct?

C:\Users>nslookup pi.hole
Server:  UnKnown
Address:  10.0.0.99

*** UnKnown can't find pi.hole: Non-existent domain

Thanks!

jfb · September 16, 2019, 3:50pm

This is correct.

Bucking_Horn · September 16, 2019, 4:34pm

I was just going to guess that, as it perfectly explains why your Pi-hole was accessible under two distinct IP addresses - you were quicker, and I haven't even finished writing

Some points in your initial description have caught my attention:

As you are setting up your DNS manually on each client, your router can't hand out alternate DNS servers (It could still cause rebind issues, but in that case, you'd likely be unable to access the internet at all - not what you describe you're suffering from).

So this means that your KGWin10 is indeed using your Pi-hole as DNS server when using system level commands.

So when you are browsing the internet, no ads are being blocked and no activities are logged with Pi-hole.

These three observations taken together stir my suspicion that your browser might somehow be the actively involved in this.

Let's try to narrow that down:

on your KGWin10, bring up Pi-holes admin console
switch to the live view of Pi-hole's log under Tools | Tail pihole.log
You'll see an empty window at first that likely will fill up with a few lines every now and then
now open a Command Prompt on your KGWin10 and force some activity:
Let's start with something within your local network, like
ping raspberrypi
Watch the Pi-hole log - this should give you something like

query[A] raspberrypi.local from 10.0.0.16
dnsmasq[5128]: forwarded raspberrypi.local to 10.0.0.1
dnsmasq[5128]: reply raspberrypi.local is 10.0.0.99

And now something on the internet that is blocked by Pi-hole
ping doubleclick.com
should result in

dnsmasq[5128]: query[A] flurry.com from 10.0.0.16
dnsmasq[5128]: /etc/pihole/gravity.list doubleclick.com is 0.0.0.0

Finally, in a new browser window, open a site you know to be blocked, like flurry.com, which should show up in the Pi-hole log

dnsmasq[5128]: query[A] flurry.com from 10.0.0.16
dnsmasq[5128]: /etc/pihole/gravity.list flurry.com is 0.0.0.0

If that last step doesn't bring up any activities in Pi-Hole log, your browser is diverting DNS queries, cutting off your Pi-hole.
That would bring up questions as: What browser are you using? Have you probably switched to a new browser on Sep 13th, or did your browser receive an update then?

If you are seeing no log entries appearing for any of these activities at all, I am out of explanations.

moose_homans · September 16, 2019, 8:27pm

[quote="Bucking_Horn, post:15, topic:23587"]
Let’s try to narrow that down:[/quote]

Righto!

on your KGWin10, bring up Pi-holes admin console

switch to the live view of Pi-hole’s log under Tools | Tail pihole.log. You’ll see an empty window at first that likely will fill up with a few lines every now and then

(minus the filling up lines so far)

now open a Command Prompt on your KGWin10 and force some activity.

Let’s start with something within your local network, like
ping raspberrypi

C:\Users\>ping raspberrypi

Pinging raspberrypi.local [10.0.0.99] with 32 bytes of data:
Reply from 10.0.0.99: bytes=32 time=5ms TTL=64
Reply from 10.0.0.99: bytes=32 time=5ms TTL=64
Reply from 10.0.0.99: bytes=32 time=7ms TTL=64
Reply from 10.0.0.99: bytes=32 time=7ms TTL=64

Ping statistics for 10.0.0.99:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 5ms, Maximum = 7ms, Average = 6ms

Watch the Pi-hole log

...

I got doodly-squat. Same for ping doubleclick.com or ping www.google.com or ping flurry.com

If you are seeing no log entries appearing for any of these activities at all, I am out of explanations.

To prove to myself I wasn't crazy, I used ping raspberrypi from the raspberry pi itself. It did not register in the log. However, ping pi.hole did:

Sep 16 15:18:55 dnsmasq[989]: query[A] pi.hole from 127.0.0.1
Sep 16 15:18:55 dnsmasq[989]: /etc/pihole/local.list pi.hole is 10.0.0.99
Sep 16 15:18:55 dnsmasq[989]: query[AAAA] pi.hole from 127.0.0.1
Sep 16 15:18:55 dnsmasq[989]: /etc/pihole/local.list pi.hole is 2601:446:c400:d766:5e8e:bebf:1563:e28
Sep 16 15:18:55 dnsmasq[989]: query[PTR] 8.2.e.0.3.6.5.1.f.b.e.b.e.8.e.5.6.6.7.d.0.0.4.c.6.4.4.0.1.0.6.2.ip6.arpa from 127.0.0.1
Sep 16 15:18:55 dnsmasq[989]: /etc/pihole/local.list 2601:446:c400:d766:5e8e:bebf:1563:e28 is raspberrypi
Sep 16 15:18:55 dnsmasq[989]: query[PTR] 8.2.e.0.3.6.5.1.f.b.e.b.e.8.e.5.6.6.7.d.0.0.4.c.6.4.4.0.1.0.6.2.ip6.arpa from 127.0.0.1
Sep 16 15:18:55 dnsmasq[989]: /etc/pihole/local.list 2601:446:c400:d766:5e8e:bebf:1563:e28 is raspberrypi

and ping doubelclick.com

Sep 16 15:20:23 dnsmasq[989]: query[A] doubleclick.com from 127.0.0.1
Sep 16 15:20:23 dnsmasq[989]: /etc/pihole/gravity.list doubleclick.com is 0.0.0.0
Sep 16 15:20:23 dnsmasq[989]: query[AAAA] doubleclick.com from 127.0.0.1
Sep 16 15:20:23 dnsmasq[989]: /etc/pihole/gravity.list doubleclick.com is 0.0.0.0

Other things I've done:

Noticed that the raspberrypi's clock was off and set it to local time.
Moved raspberrypi from a switching hub to plugging directly into the Comcast router.
Installed cygwin on KGWin10 just so I could run dig commands and get frustrated.

Thanks much for the advice, everyone! I think I'll demand a "Dad breaks the internet" day sometime this week and set up a known good router just to see if I can narrow it down to the Xfinity router.

Bucking_Horn · September 16, 2019, 8:50pm

My test steps assume you execute them on a freshly booted machine - my fault, I should have thought of that

This leaves us with one last straw of hope - your Pi-hole might not have seen requests from KGWin10 because that already knew the IP addresses.

You don't need a restart to try, just close all running instances of your browser and flush the systems DNS-Cache using ipconfig /flushdns at the Command Prompt.

moose_homans · September 17, 2019, 2:22am

I've been flushing my dns cache like a...thing that flushes caches all the time.

moose_homans · September 18, 2019, 4:13am

Hello!

Re-imaging via n00bs and a fresh install of pihole did the trick! The root cause remains a mystery, as I had not touched the raspberrypi other than using the pihole web interface for a good 3 months before it went wackadoodle.

Thanks to everyone who gave me advice!

system · October 9, 2019, 4:13am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.