Created a new group with no adlists associated ("Bypass") and added client defined by its MAC. nslookup flurry.com was still blocked for that client. Removing the client and re-adding by it's IP, doing nslookup flurry.com again and IPs for that domain were resolved.
From the log
[2020-05-13 22:30:31.758 15256] Querying gravity database for client 10.0.1.3 (getting best match)
[2020-05-13 22:30:31.759 15256] Found database hardware address 10.0.1.3 => f0:9f:c2:ce:09:31
[2020-05-13 22:30:31.759 15256] Querying gravity database for client f0:9f:c2:ce:09:31 (hardware address)
[2020-05-13 22:30:31.759 15256] Gravity database: Client f0:9f:c2:ce:09:31 not found. Using default group.
Is the match case-sensitive? The client was added from the drop-down menu in all uppercase MAC.
Thanks for your testing, I appreciate it. More pair of eyes certainly help to find and fix such glitches, especially when adding a new fundamental feature like this one requiring a lot of changes under the hood.
This was a displaying quirk on the web frontend, please try again if you see the IP addresses you're expecting. Note that only recently (24h) seen IP addresses will appear.
Yes, this was the case. I guess it's anyhow safe to assume that the case my differ. As there is no different in upper and lower case, I changed the comparison to be case-insensitive.
There used to be some code in there that checked if it had been updated in the last x(24 maybe?) hours and if so, didn't update. But that may have been removed at some point. I'll look into it and edit this comment if I find anything useful, so that the thread doesn't veer off topic
I just added support for multiple host names per client to the networking table. You'll have to update both branches. Nothing else (except for a bit of patience) is needed.
Note that I already added the vendor associated to the MAC address into the select menu. They may help to identify devices.
(note that the most recently host name will be shown first in case there are multiple).
Currently, it doesn't care. It shows IP addresses only for devices FTL found no associated MAC addresses for (VPN connected devices, etc.).
When a query comes in for the first time from a previously unknown client
Once for all clients after any change has been made to any of the per-group settings.
The network table already has all the information at hand. It is a deterministic source also containing information about devices outside the scope of ip neigh (said VPN devices, etc.). Also, opening a pipe to ip neigh is rather inefficient so we do this only in the thread dedicated to this job.
It will fall into the default group. There is even the chance that devices are not found in the ip neigh cache immediately on their first query. This is an issue. We have to come up with a solution for this (maybe try to assign a client once again to groups one minute after it first popped up).
Yes. MAC addresses are only used in case there is no IP match.
Added the IP addresses. Again, if there are multiple (recent) ones, they are all shown where front to back related to most recent to least recently seen. This will also allow you to use the search field for finding clients by their IP addresses.
I disagree. In fact, I think address gives the least information. In a properly set up networking, it is dynamically assigned by the DHCP server and you do not have to care about it (you should use a host name, like server.home). We could maybe get the host name first, however, mind that the number of host names can be arbitrarily large. Let's try this.
The vendor information typically tells you, at least, which manufacturer to expect. If this is Motorola, I know it can only be my phone. For my desktop computer (ASRock), this is already less obvious.
Ah, yes, I missed to remove them.
This was actually quite difficult to obtain, however, we have it now (I set the maximum height to 400 pixels).
Yes, I agree with what was said. I do not care about the IP address, that is the business of my DHCP server. All devices have proper host names, like 001.lightbulb.home so I can identify them. I configured them in the smart home controller and everything is perfectly nice. I checked in the network table and see that each light bulb has multiple IPv6 addresses. Adding the address in front would make finding clients by just looking at the items much harder for me. I like host name in front.
I compared this against my network table and it seems to be ordered by "first seen". The oldest devices (known for very long time) appear at the top. New ones appear at the very bottom. Maybe we can invert this (most recently added devices = new ones) at the top?
I don't think sorting alphabetically would make sense. Separating them into IP and MAC - maybe. However, I'd rather like to see the "most recent device at the top" implementation. Only this will actually make it a useful feature. A new device may be MAC, however, it may also be IP-only, so grouping them into MAC/IP and them sorting most-recent-top will not work as I still have to scroll down very long until I can find the IP section.
It works for me. I am using current Firefox on Xubuntu. Which browser are you using @yubiuser? Maybe it is a Chrome/Internet Explorer issue?
I'll check this out when I'm back home (takes a few days).
Well, it's not that weird at all. I was toying with the idea of doing this. However, it is a quite fundamental change (again) and extra code that would need to be written (again). I'll be busy for some time, but I will keep thinking about it. Currently, I see no harm in this idea. It just gives users even more flexibility. They can decide if they want to use it - or not.
Well, after having consulted my pillow, I recall why I haven't added it, so far. It will not work as you may expect it to. I will (superficially) go into the technical details involved here.
Example 1 - The client is already present in the network for a longer time before doing its first lookup
When there is an entry in the network table with MAC, IP and host name, everything would work as expected. We can look up the IP address of the incoming request, relate this to a MAC address and check if one of the host names associated with this MAC is a configured client. If so, take the groups defined here.
However, I think this scenario will happen only rarely.
Example 2 - The client immediately performs a lookup when connecting to the network
There is may be no network table entry, so far. Chances are high that there isn't even a (complete) ARP/neigh entry when the device freshly joined the network, so FTL can only see its IP address.
Neither MAC nor host name will be available at this point and I don't think there's something we can really do about this. Hence, if a client is configured through MAC or host names, we will miss the configuration at this point and (wrongfully) attribute it to the default group.
We can "compensate" for the miss by triggering a re-grouping inside FTL when we are able to gather information about the client. However, this still means that a client would be able to do lookups it should not do (because usually blocked by, e.g., gravity) for some (short but maybe up to one minute) period of time. I don't think this is good.
And, before you ask, we can neither do a lookup for the host name when we first see a client appearing. The reason for this is that we are already processing an incoming query (said client is asking the DNS server something). FTLis now processing and trying to find out what to do with this client (to to reply may depend on the assigned groups). Unfortunately, we cannot run concurrent queries when we're still in this incoming query. Hence we cannot make lookups for hostnames while we're still in the loop of processing a running query (the DNS infrastructure is locked).
So let me summarize: We have clients on the local subnet which are actively using pihole and despite it checks the ARP cache every minute there are situations where the cache has no entry for that device? That’s strange…
Hence, if a client is configured through MAC or host names, we will miss the configuration at this point and (wrongfully) attribute it to the default group.
As you said, this is the same limitation for MAC and hostname - which doesn't speak against implementing the hostname approach when MAC is implemented.
For both pihole needs to
"compensate" for the miss by triggering a re-grouping inside FTL when we are able to gather information about the client.
As the network table is updated every minute you could trigger a "re-groupping" immediately afterwards. This would also compensate for
What happens if my non-deterministic DHCP assigns a new IP to a device within the 24h time range and the network table is not updated already?
However, this still means that a client would be able to do lookups it should not do (because usually blocked by, e.g., gravity) for some (short but maybe up to one minute) period of time. I don't think this is good.
I agree from a technical point of view - from a user's point of view I don't care. When I had no pihole all devices could contact whatever they wanted. I would rather have this trade-off (small amount of time wrongly assigned clients) than to miss this feature. I know you want pihole to be perfect - I guess for most it is sufficient when it is really good.