Seeing an unrecognized domain in top clients

Expected Behaviour:

I should see only hostnames and domains that are in /etc/hosts on the pihole installation.

Actual Behaviour:

I see some hostnames that are in /etc/hosts but I also see one that I don't recognize and it's not anywhere in my configuration: gigabitethernet6-0.dca2c-fcor-rt2.netsrv.digex.net

Debug Token:

4kn7z2gc39

digex.net is a verizon owned domain.

Most likely you have a device that "calls home" ... A lot !

That would make sense if this was in the "top domains" list. Rather, it's showing in the "top clients" list, meaning that somehow, this client on the LAN is getting an external IP address and PiHole is resolving that external IP to a hostname for my viewing pleasure.

I thought perhaps my router is using the PiHole for DNS, which it shouldn't be, but my router's external IP is different.

The ping time to gigabitethernet6-0.dca2c-fcor-rt2.netsrv.digex.net is ~10 ms from my wired-in workstation. Traceroutes don't complete but are clearly routed outside of my home network onto Verizon and then to Alter.net, where they die:

$ traceroute gigabitethernet6-0.dca2c-fcor-rt2.netsrv.digex.net
traceroute to gigabitethernet6-0.dca2c-fcor-rt2.netsrv.digex.net (164.109.3.177), 64 hops max, 52 byte packets
 1  10.9.9.1 (10.9.9.1)  0.972 ms  0.405 ms  0.345 ms
 2  lo0-100.pitbpa-vfttp-311.verizon-gni.net (100.6.43.1)  0.951 ms  1.146 ms  1.108 ms
 3  b3311.pitbpa-lcr-21.verizon-gni.net (100.41.220.116)  6.025 ms
    b3311.pitbpa-lcr-22.verizon-gni.net (100.41.220.118)  8.233 ms
    b3311.pitbpa-lcr-21.verizon-gni.net (100.41.220.116)  6.434 ms
 4  * * *
 5  0.ae2.gw2.dca6.alter.net (140.222.233.99)  11.017 ms  10.935 ms
    0.ae1.gw2.dca6.alter.net (140.222.233.97)  10.492 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
…
41 * * *

if you hoover the mouse on the entry, or even click on it, it will filter the queries for that particular entry.

Under client you should see the IP that is requesting it.

Is that an internal IP ?

I just looked at your debug log and see some PTRs for 164.109.3.177 so indeed it seems that external clients are querying your Pi-hole.

How about your firewall (maybe the one in your router)? Did you forward any ports so that external servers can access your Pi-hole?

I was going the long way :slight_smile: That would have been my next question ...

Perhaps you set Pi-hole as your WAN DNS server, and it is showing the external hostnames of your devices?

if you hoover the mouse on the entry, or even click on it, it will filter the queries for that particular entry.

When I hover, I see the expected name:

pihole_wtf

When I click, I'm presented with the same crazy hostname as the client:

For context, the one I clicked on is just a LIFX lightbulb. Also, I just noticed that the time of the request is ridiculously far future. I can't explain that.

pi@pihole:~ $ date
Wed Apr  4 23:26:17 EDT 2018

Did you forward any ports so that external servers can access your Pi-hole?

I have no port forwards configured except some SSH and MOSH ports to another host on the LAN, not the pihole.

Perhaps you set Pi-hole as your WAN DNS server, and it is showing the external hostnames of your devices?

Hilariously, I made this exact misconfiguration when I first set up my new router several months ago. Sadly, it's not the case anymore.

Could this have something to do with my logging setup?

pi@pihole:~ $ ls /mnt/dns-backup | wc -l
538
pi@pihole:~ $ cat logrotate
/var/log/pihole.log {
	su root root
	daily
	rotate 3650
	copytruncate
	compress
	delaycompress
	notifempty
	nomail
	olddir /mnt/dns-backup
	missingok
	dateext
	dateyesterday
	dateformat -%Y%m%d
}

/var/log/pihole-FTL.log {
	su root root
	daily
	rotate 3650
	copytruncate
	compress
	delaycompress
	notifempty
	nomail
	olddir /mnt/dns-backup
	missingok
	dateext
	dateyesterday
	dateformat -%Y%m%d
}

It seems unlikely but that's basically the only customization I've done aside from having to checkout Pi-hole Version vDev (HEAD, v3.2.1-0-ge602008) because I hit some problem with 3.3.0 that caused dnsmasq not to start.

Are the long names IPv6?

Are the long names IPv6?

No, they're IPv4.

I'm no dev here... just my two cents.

Is it possible that when pi-hole wants to have the name of your devices, it sends a dns request and that dns request is sent to an external dns and you get that "gigabitethernet6-0.dca2c-fcor-rt2.netsrv.digex.net" as an answer ? So, every devices in your home will end up being with that hostname... That would be a weird behaviour to send request for internal host and also weird behaviour for a external dns to answer that...

Did you try a few nslooklup of your internal IP from your pi-hole ?

Is it possible that when pi-hole wants to have the name of your devices, it sends a dns request and that dns request is sent to an external dns

It's possible, but really strange. It's already getting the hostname correctly for the link title. I can't reason why the link text would be different. Maybe I'll have to dig into the code.

pi@pihole:~ $ nslookup lifx4
Server:		127.0.0.1
Address:	127.0.0.1#53

Name:	lifx4
Address: 10.9.9.101

This and a sample of the others are correct.

api@pihole:~ $ cat /etc/resolv.conf
# Generated by resolvconf
nameserver 127.0.0.1
pi@pihole:~ $ ps aux | grep dnsmasq
pi       13558  0.0  0.4   4268  2016 pts/0    S+   09:05   0:00 grep --color=auto dnsmasq
dnsmasq  25390  0.0  2.7  16128 12128 ?        S    Apr04   2:16 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -r /var/run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new --local-service --trust-anchor=.,<snip>
pi@pihole:~ $ cat /var/run/dnsmasq/resolv.conf
# Generated by resolvconf
nameserver 10.9.9.1

I also looked through files in /etc/dnsmasq.d and didn't see any shenanigans in there. They all appear to be files managed by PiHole although two of them lack a comment explaining so. Both of these have address and server directives that make sense for the intent given the filename.

Weird! What is the output of http://pi.hole/admin/api.php?getQuerySources in a browser where you logged into your Pi-hole dashboard?

{
  "top_sources": {
    "gigabitethernet6-0.dca2c-fcor-rt2.netsrv.digex.net|nastiest": 13295,
    "gigabitethernet6-0.dca2c-fcor-rt2.netsrv.digex.net|kid": 12710,
    "10.9.9.78": 4855,
    "gigabitethernet6-0.dca2c-fcor-rt2.netsrv.digex.net|vizio-soundbar": 3000,
    "gigabitethernet6-0.dca2c-fcor-rt2.netsrv.digex.net|lifx4": 1560,
    "gigabitethernet6-0.dca2c-fcor-rt2.netsrv.digex.net|lifx1": 1530,
    "gigabitethernet6-0.dca2c-fcor-rt2.netsrv.digex.net|lifx2": 1505,
    "gigabitethernet6-0.dca2c-fcor-rt2.netsrv.digex.net|nest": 1105,
    "gigabitethernet6-0.dca2c-fcor-rt2.netsrv.digex.net|localhost": 245,
    "steamlink|10.9.9.55": 140
  }
}

Okay, that's very odd. Sorry, I have to ask again which FTL version are you running? Could you also post the output of

pihole -v

or

echo ">version" | nc pi.hole 4711
pi@pihole:~ $ pihole -v
  Pi-hole version is v3.2.1 (Latest: v3.3.1)
  AdminLTE version is v3.3 (Latest: v3.3)
  FTL version is v3.0 (Latest: v3.0)

Remember, though, I had to manually checkout v3.2.1 because v3.3.1 has (had?) that bug with dnsmasq not starting. I'll go back to releases when that bug is fixed OR I give up and re-do the installation on the Pi. It's still running Debian Jessie.

Oh, yes. I think there is an inconsistency between AdminLTE v3.3 and older versions of FTL. This is nothing you should worry about. If it annoys to too much, you could also down-grade the web interface to a compatible version, what do you think?

My top concern in raising this is security: did I do something that compromised my DNS or was PiHole upstream compromised? At this point, neither are likely, bordering on impossible, so that's out.

My next concern was misconfiguration: did I do something that was causing this? Unlikely, because, frankly, I've not done anything to the PiHole installation aside from my custom logging setup. So, that's out.

My next goal was reporting to see if anyone else had encountered this and enable devs to help troubleshoot what could have caused it. It looks like we're all at ¯\_(ツ)_/¯ stage with it, so…

At this point, I think my effort on this is best expended simply redeploying a fresh PiHole setup from scratch on a fresh installation of a the latest stable Debian, reintegrating my logging solution, and moving on with life! We'll chalk this one up to a weird occurrence and see what happens next.

No, I'm sorry for not having been explicit with this. I think I know what happens, I'm just not sure why it happens in the first place. If you have your system still running, would you mind posting the output of

echo ">top-clients" | nc pi.hole 4711

so I could see if my assumption was correct?

Looks like it's not connecting? Trying the IP directly also fails.

[colin@ayla ~]$ echo ">top-clients" | nc pi.hole 4711
[colin@ayla ~]$ echo $?
1
[colin@ayla ~]$ nmap -p 4711 pi.hole
Starting Nmap 7.70 ( https://nmap.org ) at 2018-04-17 11:10 EDT
Nmap scan report for pi.hole (10.9.9.2)
Host is up (0.00071s latency).

PORT     STATE  SERVICE
4711/tcp closed trinity-dist

Nmap done: 1 IP address (1 host up) scanned in 0.06 seconds

From the pi, which looks like pihole-FTL is only listening on localhost and not on all interfaces:

pi@pihole:~ $ sudo netstat -tanp | grep 4711
tcp        0      0 127.0.0.1:4711          0.0.0.0:*               LISTEN      1712/pihole-FTL
tcp6       0      0 ::1:4711                :::*                    LISTEN      1712/pihole-FTL
pi@pihole:~ $ echo ">top-clients" | nc 10.9.9.2 4711
pi@pihole:~ $ echo $?
1
pi@pihole:~ $ echo ">top-clients" | nc localhost 4711
0 13295 nastiest nastiest
1 12710 kid kid
2 4855 10.9.9.78
3 3000 vizio-soundbar vizio-soundbar
4 1560 lifx4 lifx4
5 1530 lifx1 lifx1
6 1505 lifx2 lifx2
7 1105 nest nest
8 245 localhost localhost
9 140 10.9.9.55 steamlink
---EOM---