Request: Ability to add clients to known client list from Network page

Yes, this is exactly the issue why they're not already in there. The MAC needs to be unique in the table, but we can likely "just" use the IP address as pseudo-MAC.

The issue is more with the statistics. Given many users have a non-deterministic DHCP server (handing out IP addresses sequentially in the order of appearance of the device), it is a commonly seen situation that a smartphone gets the IP address 192.168.0.2 which was associated to a laptop on the day before. And this just because the device was switched on before the laptop on this day. All the statistics that were counted for the laptop yesterday would now be added to the same "device" because of the same IP address. I don't think I like this...

Note that the Pi-hole's internal DHCP server is in fact deterministic, i.e., tries to assign the same dynamic IP address to the same devices every time. This is also the reason why the first device can receive a pretty high terminal address such as 192.168.0.216 because we computed this IP from the hash of the MAC address of the requesting device. This to make per-client filtering possible even with devices that may do not even support (or honor) static assignments. I, for instance, do not have a single static IP (only exception is the Pi-hole itself) in my network. Everything else is dynamic.

There is not only a lack in validation, the entire logic seems to need a restructuring. Already on my todo list. ETA sometime next week.

Please try if

pihole checkout ftl new/all_clients_network_table

fills your network table with all the clients you are missing from it. Instead of only those known to the kernel (ARP/neigh cache), we now also populate the table with all remaining clients known to FTL. Obviously, there will be no MAC address available here and also the interface information is too difficult to obtain, so they are missing. I will have to invest a little bit of work into hiding the mock hardware addresses in the web interface but it should already work now.

Note that this change does not implement validation for the input fields in any way. This has to come separately (and later).

2 Likes

Awesome!

I'm not sure if it is a FTL's refresh/cache issue (as the network table is not populate immediately) but I have a client that is now twice in the list.

No ideas why it is this client and not all others too....

  Current Pi-hole version is v4.3.5-444-gf617ed2
  Current AdminLTE version is v4.3.2-436-g56de14a
  Current FTL version is vDev-ae3418a

I assume the device has not (yet) been used today, right? Hence, it was in FTL's memory but not in the ARP cache (the entry expired over night) and FTL added it to the table as something the ARP cache didn't know about. This is maybe expected side effect of what we do now.

I pushed a small modification skipping the addition of clients with zero (active) queries. This may relieve this special situation. However, we cannot know if we see one and the same thing without the MAC address.

Please update FTL to the latest version and remove the entry you're seeing there with

sudo sqlite3 /etc/pihole/pihole-FTL.db "DELETE FROM network WHERE hwaddr = 'ip-10.0.1.84';"

No need to notify FTL of this change, it will update the network table a few minutes later.

Right :slight_smile:

But there are others that haven't queried pihole today and haven't got a duplicated entry.

Why are you using the arp cache as comparison instead of pihole's network table? This would circumvent the problem with clients not seen for a while (no info in the ARP cache) because they are still in the network overview? I guess there is a drawback I don't see right now... :smiley:

This reminds me that I've seen a feature request somewhere to be able to delete single entries from the network table via web UI :wink:

Because we don't know if a device that was not seen in weeks with some IP address 192.168.0.3 is still the same as it was a long time ago. Most users have routers with sequential IP address associating DHCP servers, i.e., the IP a device gets solely depends on the order of switching the devices on at this day. This makes the entire identify devices by their IP addresses idea pretty useless and is why I am still cautious with this approach.

Yes, this is because the last query of this device was three days ago, i.e., it isn't present any more in Pi-hole's memory.


edit: I pushed another small change that should finally eliminate the double addition of devices with the same IP address. This should resolve most of my concerns.

I hear you, but I don't think we want to push the release of v5.0 much longer. There has to be something left for v5.1 :wink:

Clearly understand. This is why it should be used with static DHCP leases.

Hehe :wink:

My general take on networking tell me: Never use static DHCP leases. It's also why Pi-hole's integrated DHCP server is a deterministic one. But that's another story.

I went back and forth about having this as an option for power users (disabled by default), but then I think it wouldn't make any difference as "normal" users wouldn't have any devices in their networks which are not also known to the Pi-hole (except lo, a.k.a. 127.0.0.1 or ::1 on the Pi-hole server itself).

If there is once enough time I would really like to hear the reason for that.

My guess is: 99% of the users won't notice any differences (except of 127.0.0.1) so you could also skip the option and enable it by default. The ones that use VLAN/subnetting will likely be happy to see their device and will use the function anyway if they know about it. You would save them a step to enable the configuration.

Yes, this is my plan.

Networking is also about simplicity. KISS is a good principle to keep things clean. Other reasons:

  • It is much easier to remember addresses than IPs when they are meaningful.

  • Using names like chromecast is much easier than trying to remember that it was 10.0.30.254 after not having used this in months. In your case, it is not only the last octet to be remembered, it is even more so prone to errors. DNS is a central service and should be relied on enough to assume it is stable.

  • Switching behavior to host names already now will make the transition to dual-stacked (at at some point, hopefully, IPv6-only) systems a lot easier.

  • The IP address can change without affecting the end user in case you run internal services and want to move them from one to another device. This may be less important on home-style networks, but, as you know, one IP can identify a single server and yet have many names associated to it. One server can host multiple "virtual hosts" and only using a host name, they can be distinguished.

There are probably some more but this is what came to my mind without thinking too much about it. And typing on the phone is not one of my strengths, either, so I'll just stop here. I think this request is resolved once we merge this into the beta code. Anything else we're missing here (except the necessary web changes to hide mock-addresses)?

Made an account to say I just stumbled onto this and it worked great for me.

I recently migrated to a docker container bridged setup of pihole, unbound and dhcphelper relay running together. Once that was up I noticed the network table page populated with just the IP of the dhcp relay interface.

Running the above command populated around 20 hostnames and their details for the exception of obviously the MAC address.

Thank you! Looking forward to it being added officially in upcoming releases!

1 Like

Something went off again. Not sure when it happend, my guess would be when the device 10.0.1.84 was turned on first time today. It is the same device we had the issues earlier today. 10.0.1.1. is the Gatway/DHCP server.

nanopi@nanopi:~$ pihole -v
  Pi-hole version is v4.3.5-444-gf617ed2 (Latest: v4.4)
  AdminLTE version is v4.3.2-436-g56de14a (Latest: v4.3.3)
  FTL version is vDev-560685a (Latest: v4.3.1)

Not that I see. I have one more idea but this can wait for v5.1 (will there be a beta as well?)

Thanks for your explanation and thoughts.

I raised the idea of doing a beta every time, let's see if this becomes reality. The likeliness is high. The delay from v5.0 to v5.1 should be much smaller than the last update from v4.3 (which is almost one year old). We're planing more for like two months in between each point release to ship features to users more quickly but v5.0 was such a massive change and improvement in so many directions that it tool a very long time to mature.

What is the output of

sqlite3 /etc/pihole/pihole-FTL.db "SELECT * FROM network_addresses;"

? We can work from there.

Just noticed, another device appeared twice.

nanopi@nanopi:~$ sqlite3 /etc/pihole/pihole-FTL.db "SELECT * FROM network_addresses;"
10|10.0.1.35|1581332400
5|10.0.1.4|1583391600
11|10.0.1.84|1583564400
4|10.0.1.1|1583571600
2|10.0.1.190|1583575200
6|10.0.1.64|1583575200
8|10.0.1.3|1583575200
12|10.0.1.6|1583575200
1|10.0.1.136|1583575200
9|10.0.1.215|1583575200
3|10.0.1.2|1583575200
7|10.0.1.182|1583575200
13|10.0.40.3|1583578800
14|10.0.30.254|1583578800
15|127.0.0.1|1583578800
16|10.0.1.84|1583578800
16|10.0.1.1|1583614800
17|10.0.40.2|1583679600
18|10.0.1.190|1583701200
19|10.0.1.64|1583730000
  Pi-hole version is v4.3.5-445-g22ce5c0 (Latest: v4.4)
  AdminLTE version is v4.3.2-438-gcb0eb3e (Latest: v4.3.3)
  FTL version is vDev-560685a (Latest: v4.3.1)

edit: more are coming

For the first screenshot, I cannot reliably tell what happened as the timestamps are not visible here.

For the second screenshot, the situation was as follows:

  1. The device was known since Feb 5th with its real MAC address.
  2. On March 9th, this device was known to FTL as have made a lot of queries within the last 24 hours, however, it was not known any more to the Pi-hole through the ARP cache as the ARP entry either expired or you rebooted the device while the other was already offline. Hence, FTL had no other chance than adding a new device for it as it shall not assume that devices with the same IP are identical.

edit I'm not sure how to code around this without assuming that devices can be uniquely identified by their IP addresses. For a minute, I was thinking we suppress these extra additions for a running FTL instance by memorizing the database ID of this device so we can update even when the ARP data is gone. Even when this knowledge will not survive a restart, it would help. However, this would have another consequence, namely that devices changing their IP address while FTL is running, will be attributed incorrectly to another entry.

New idea: Instead of adding a new mock device, we can check who was the last device using a given IP address. Then add to this one. Only if an IP address is altogether new, we create a mock device. Even when this would likely resolve what you see, it will still mean we make assumptions somehow and the network page will not be fully precise (as it was before). What do you think?

edit 2: Alternative, maybe better idea: We ignore non-mock devices for which the IP address is already in the database but no ARP entry is found.

I'm not sure if I have understood the whole process completely (how long does FTL store entries in its memory - 24h? When do they beginn to count? How often does it update the network table? Every 60 minutes?) but I see that it's difficult without assuming IPs are unique :-/
I think there is no way around to assume devices use the same IP when they are not on the same subnet as there is no unique identifier (like MAC). This is the limitation if you want to add devices from outside of the local subnet to the network table.

I have two ideas - but you may not like them because they increase complexity.
(I assume that for every request from the local subnet there is a ARP entry for at least the time until the cache expires)

  1. Would it consume to much time/resources to update the network table "live" (so not using FTL's memory)? For every IP request FTL gets, it looks in the ARP cache for the corresponding MAC and attributes the request to that MAC. So there should no discordance between FTL and ARP for devices on the local subnet as every device sending requests should be still in the ARP cache.
    IPs not found in the ARP cache must be from other subnets as the cache is probed "live" and there should be no devices in the same subnet without an ARP entry when this device is performing a DNS request. Create a mock-MAC for those IP not found and add it to the network table.

  2. If you don't want to make it "live" you could take "snapshots" of the ARP cache periodically and determine retrospectively which device (IP) had which MAC when it did a certain request and/or if had an entry in the ARP cache at a particular time at all. Then you could again assume for a request at at particular time without an ARP entry that it came from outside of the local subnet and a mock-MAC should be created.

Yes

From the moment a query is received on.

Once a minute.

What you suggest is already what we so. We do not do it "live" but when we update the database (so once every minute). This shouldn't be an issue.

This was my assumption as well. However, "real" clients being added with mock-MAC despite the fact that they are on the local subnet seem to suggest a different situation. Furthermore, I haven't seen any mock devices so far in my network (I only run local devices on the same subnet at the moment).
The only added device is 127.0.0.1 for me.


edit: Hiding the mock hardware addresses on the dashboard happens with

pihole checkout web new/all_clients_network_table
1 Like

Mhhh can't decide between this and tweak/domain_regex_input.
To many new/exciting changes :slight_smile: