Started this topic to provide feedback to the new feature allowing clients to be defined by MAC address.
I switched to the branches (core still master) by
pihole checkout ftl new/mac_clients
pihole checkout web new/mac_clients
And flushed network table afterwards.
Important
First thing I noticed in Group management/Clients Known Clients now only contain clients with known MAC, no IPs are listed anymore (I have devices on other Vlans so IP would still be handy).
Created a new group with no adlists associated ("Bypass") and added client defined by its MAC. nslookup flurry.com was still blocked for that client. Removing the client and re-adding by it's IP, doing nslookup flurry.com again and IPs for that domain were resolved.
From the log
[2020-05-13 22:30:31.758 15256] Querying gravity database for client 10.0.1.3 (getting best match)
[2020-05-13 22:30:31.759 15256] Found database hardware address 10.0.1.3 => f0:9f:c2:ce:09:31
[2020-05-13 22:30:31.759 15256] Querying gravity database for client f0:9f:c2:ce:09:31 (hardware address)
[2020-05-13 22:30:31.759 15256] Gravity database: Client f0:9f:c2:ce:09:31 not found. Using default group.
Is the match case-sensitive? The client was added from the drop-down menu in all uppercase MAC.
Thanks for your testing, I appreciate it. More pair of eyes certainly help to find and fix such glitches, especially when adding a new fundamental feature like this one requiring a lot of changes under the hood.
This was a displaying quirk on the web frontend, please try again if you see the IP addresses you're expecting. Note that only recently (24h) seen IP addresses will appear.
Yes, this was the case. I guess it's anyhow safe to assume that the case my differ. As there is no different in upper and lower case, I changed the comparison to be case-insensitive.
I already configured my entire network (more than 50 IoT devices) using this and everything works as expected and very smoothly. Keep up the good work!
Could you add IPs for the entries a MAC is known (IP-MAC (hostname)). How does the drop-down behave if multiple IPs are associated with one MAC (can't test myself, doesn't have this situation)
Group assignment works now
[2020-05-14 09:12:03.530 11458] Querying gravity database for client 10.0.1.3 (getting best match)
[2020-05-14 09:12:03.531 11458] Found database hardware address 10.0.1.3 => f0:9f:c2:ce:09:31
[2020-05-14 09:12:03.531 11458] Querying gravity database for client f0:9f:c2:ce:09:31 (hardware address)
[2020-05-14 09:12:03.532 11458] Querying gravity database for client 10.0.1.3 (getting groups)
[2020-05-14 09:12:03.532 11458] Gravity database: Client f0:9f:c2:ce:09:31 found. Using groups [1].
A few questions for my understanding:
How often does FTL associate IP->MAC? Every time it gets a request from an IP without explicite group assignment?
What are the benefits from using network table for MAC lookups instead of neigh? Speed? Latter should be more up to date and removes need to remove old IP entries every 24h.
What happens if my non-deterministic DHCP assigns a new IP to a device within the 24h time range and the network table is not updated already?
How are conflict group assignments handled when the IP is associated with an different group than the MAC? IP takes priority and MAC assignment is skipped?
P.S. Is it possible to skip the update of the apt cache every time I update/change the core branch? EDITTo answer my question: no
There used to be some code in there that checked if it had been updated in the last x(24 maybe?) hours and if so, didn't update. But that may have been removed at some point. I'll look into it and edit this comment if I find anything useful, so that the thread doesn't veer off topic
I just added support for multiple host names per client to the networking table. You'll have to update both branches. Nothing else (except for a bit of patience) is needed.
Note that I already added the vendor associated to the MAC address into the select menu. They may help to identify devices.
(note that the most recently host name will be shown first in case there are multiple).
Currently, it doesn't care. It shows IP addresses only for devices FTL found no associated MAC addresses for (VPN connected devices, etc.).
When a query comes in for the first time from a previously unknown client
Once for all clients after any change has been made to any of the per-group settings.
The network table already has all the information at hand. It is a deterministic source also containing information about devices outside the scope of ip neigh (said VPN devices, etc.). Also, opening a pipe to ip neigh is rather inefficient so we do this only in the thread dedicated to this job.
It will fall into the default group. There is even the chance that devices are not found in the ip neigh cache immediately on their first query. This is an issue. We have to come up with a solution for this (maybe try to assign a client once again to groups one minute after it first popped up).
Yes. MAC addresses are only used in case there is no IP match.
Added the IP addresses. Again, if there are multiple (recent) ones, they are all shown where front to back related to most recent to least recently seen. This will also allow you to use the search field for finding clients by their IP addresses.
Thanks for the long explanation. With the addition of the addresses it's much easier to find the devices.
Few suggestions:
Change the order of properties to MAC, address, hostname, vendor. Vendor as second is quite prominent and might only add little info..
Remove address from devices with unknown MAC. This information is duplicated.
Change the "length" of the drop-down as it is touching the "Show X entries" drop-down. I think it looks a bit strange. Longer or shorter - both seem fine.
I disagree. In fact, I think address gives the least information. In a properly set up networking, it is dynamically assigned by the DHCP server and you do not have to care about it (you should use a host name, like server.home). We could maybe get the host name first, however, mind that the number of host names can be arbitrarily large. Let's try this.
The vendor information typically tells you, at least, which manufacturer to expect. If this is Motorola, I know it can only be my phone. For my desktop computer (ASRock), this is already less obvious.
Ah, yes, I missed to remove them.
This was actually quite difficult to obtain, however, we have it now (I set the maximum height to 400 pixels).
Mhh, maybe it is because in my small environment I know all the devices personally
Let's ask @Coro - using it with >50 devices might give some insight to.
How are the devices ordered at the moment? Given the two groups "MAC" and "IP" separately I would prefer to see devices in each group in ascending oder.
Thanks for enlarging the drop-down. Somehow the "background" doesn't fill the entire drop-down.
Yes, I agree with what was said. I do not care about the IP address, that is the business of my DHCP server. All devices have proper host names, like 001.lightbulb.home so I can identify them. I configured them in the smart home controller and everything is perfectly nice. I checked in the network table and see that each light bulb has multiple IPv6 addresses. Adding the address in front would make finding clients by just looking at the items much harder for me. I like host name in front.
I compared this against my network table and it seems to be ordered by "first seen". The oldest devices (known for very long time) appear at the top. New ones appear at the very bottom. Maybe we can invert this (most recently added devices = new ones) at the top?
I don't think sorting alphabetically would make sense. Separating them into IP and MAC - maybe. However, I'd rather like to see the "most recent device at the top" implementation. Only this will actually make it a useful feature. A new device may be MAC, however, it may also be IP-only, so grouping them into MAC/IP and them sorting most-recent-top will not work as I still have to scroll down very long until I can find the IP section.
It works for me. I am using current Firefox on Xubuntu. Which browser are you using @yubiuser? Maybe it is a Chrome/Internet Explorer issue?
Just a weird idea: can you extend the code to specify devices by hostname? Then users would be able to use client-> groups also via VPN/VLAN etc. with non-deterministic DHCPs.....
I'll check this out when I'm back home (takes a few days).
Well, it's not that weird at all. I was toying with the idea of doing this. However, it is a quite fundamental change (again) and extra code that would need to be written (again). I'll be busy for some time, but I will keep thinking about it. Currently, I see no harm in this idea. It just gives users even more flexibility. They can decide if they want to use it - or not.
Well, after having consulted my pillow, I recall why I haven't added it, so far. It will not work as you may expect it to. I will (superficially) go into the technical details involved here.
Example 1 - The client is already present in the network for a longer time before doing its first lookup
When there is an entry in the network table with MAC, IP and host name, everything would work as expected. We can look up the IP address of the incoming request, relate this to a MAC address and check if one of the host names associated with this MAC is a configured client. If so, take the groups defined here.
However, I think this scenario will happen only rarely.
Example 2 - The client immediately performs a lookup when connecting to the network
There is may be no network table entry, so far. Chances are high that there isn't even a (complete) ARP/neigh entry when the device freshly joined the network, so FTL can only see its IP address.
Neither MAC nor host name will be available at this point and I don't think there's something we can really do about this. Hence, if a client is configured through MAC or host names, we will miss the configuration at this point and (wrongfully) attribute it to the default group.
We can "compensate" for the miss by triggering a re-grouping inside FTL when we are able to gather information about the client. However, this still means that a client would be able to do lookups it should not do (because usually blocked by, e.g., gravity) for some (short but maybe up to one minute) period of time. I don't think this is good.
And, before you ask, we can neither do a lookup for the host name when we first see a client appearing. The reason for this is that we are already processing an incoming query (said client is asking the DNS server something). FTLis now processing and trying to find out what to do with this client (to to reply may depend on the assigned groups). Unfortunately, we cannot run concurrent queries when we're still in this incoming query. Hence we cannot make lookups for hostnames while we're still in the loop of processing a running query (the DNS infrastructure is locked).
As you said, this is the same limitation for MAC and hostname - which doesn't speak against implementing the hostname approach when MAC is implemented.
For both pihole needs to
As the network table is updated every minute you could trigger a "re-groupping" immediately afterwards. This would also compensate for
I agree from a technical point of view - from a user's point of view I don't care. When I had no pihole all devices could contact whatever they wanted. I would rather have this trade-off (small amount of time wrongly assigned clients) than to miss this feature. I know you want pihole to be perfect - I guess for most it is sufficient when it is really good.
