Is this still the case?
The network table had to be reset as we're now explicitly linking host names to IP addresses and this data was not available from the existing data. However, the host names should be re-populated rather soon (at least for clients known to FTL). I will work on getting host names for clients not directing queries at FTL. However, this is more lower priority as it is just the icing on the cake whereas it will not be trivial to add this concerning all the datalocks which need to be carefully aligned for this.
Furthermore, we remove records which are older than 24 hours (we flag them as "expired"). Do you think this is bad/needs tweaking?
You can customize the time after which IP addresses and host names are removed through pihole-FTL.conf using MAXNETAGE=24.0 to specify 24 hours. You can also set 72.0 for three days, etc.
Fixed.
I also changed the debug flag you have to set to true from DEBUG_DATABASE=true to DEBUG_CLIENTS=true. This will be a lot less chattier as you're likely not interested in the remaining database noise. I edited my post above to show this new flag.
I think this is a good starting point. 24h is a common lease time.
Flushing network table was successful but gave an error.
There was a problem applying your settings.
Debugging information:
PHP error (2): implode(): Invalid arguments passed in /var/www/html/admin/scripts/pi-hole/php/savesettings.php:728
Table header needs to be adjusted. Maybe simply "Client"
Played with the hostname input. I know you implemented a basic filter, but adding // crashed FTL.
pi@pihole:~ $ pihole checkout development
Please note that changing branches severely alters your Pi-hole subsystems
Features that work on the master branch, may not on a development branch
This feature is NOT supported unless a Pi-hole developer explicitly asks!
Have you read and understood this? [y/N] y
[i] Requested option "development" is not available
or run what @technicalpyro suggested but you'll need to re-checkout web and FTL afterwards, so the first command is preferred.
Well, interesting, this killed the system-provided function inet_pton. I was expecting this to be robust against invalid input but this doesn't seem to hold true. I will implement a check for IP addresses before calling this kernel function. You triggered an edge-case with an empty CIDR. Thanks for trying exactly this. It is now fixed. Please try further things to break it
The idea is to do two things:
Try to obtain a host name for clients newly added to the table which are unknown to FTL
Re-resolve all host names from all addresses once an hour.
No. 2 wasn't working because I wrapped everything into a single transaction and forgot to actually commit the changes to the database. This is now fixed as well. Please see if they populate after some time (as said, it may take up to the next full hour).
Can confirm. Took some time, but now I see all names for clients where Local DNS Records exist.
What I get now might be normal debug output.
[2020-05-19 19:00:00.677 17215] Res: no more rows available
[2020-05-19 19:00:00.678 17215] Res: no more rows available
[2020-05-19 19:00:00.679 17215] Res: no more rows available
[2020-05-19 19:00:00.679 17215] Res: no more rows available
[2020-05-19 19:00:00.680 17215] Res: no more rows available
[2020-05-19 19:00:00.681 17215] Res: no more rows available
[2020-05-19 19:00:00.682 17215] Res: no more rows available
[2020-05-19 19:00:00.683 17215] Res: no more rows available
[2020-05-19 19:00:00.684 17215] Res: no more rows available
[2020-05-19 19:00:00.685 17215] Res: no more rows available
[2020-05-19 19:00:00.688 17215] Res: no more rows available
[2020-05-19 19:00:00.689 17215] Res: no more rows available
[2020-05-19 19:00:00.690 17215] Res: no more rows available
[2020-05-19 19:00:00.691 17215] Res: no more rows available
[2020-05-19 19:00:00.692 17215] Res: no more rows available
[2020-05-19 19:00:00.693 17215] Res: no more rows available
You should now also see interface names being populated for devices added to the network table for devices more than one hop away. There are three restrictions for when they can be obtained:
The said device has do at least one query after you update to the most recent FTL
The query must be made over UDP (interface information isn't available for TCP queries at the moment)
You may not use the option bind-interfaces.
I'll put some more thoughts in relaxing 2 & 3, however, getting the interface for them is way trickier than one would have thought.
edit Restriction no. 2 is solved, no. 3 will likely stay.
192.168.4.3 is my phone connected through Wireguard. We get both the interface as well as the host name. The latter is defined through local DNS records. It shows the interface wg0 as my config file is /etc/wireguard/wg0.conf. If you'd name your wireguard interface differently, then you may see other things here (like home or work).
Note: For OpenVPN connections you'd see tun0 etc.
We also get interface and host name of 127.0.0.1.
The third device actually exists physically. Its record was complete even before my changes.
I know, I can already hear it ... Client selection by interface...
There we go, per-Interface is now supported as well. To keep interface names separate from host names, we will prepend interface names with a colon like :eth0 in order to rule out accidental matches.
The client column does now support the following for client selection:
Type
Example
IP address
192.168.2.12
IP subnet
192.168.2.0/24
MAC address
AA:BB:CC:DD:EE:FF
host name
localhost
interface
:eth0
edit The first match (from top to bottom) will be used to define the groups of a client.
I think we're on a good way here and definitely need more testing of all the new possibilities.
Your pace is awesome. Soon there will be no features left to be implemented....
It get fairly complicated with all the possibilities now. What is the order clients will be assigned? IP, MAC, Hostname, Interface. The moment one property fits all the others are skipped? Although the explanation is long already this might be worth putting there.
What is missing now is the "interface" in the explanation and tooltip hint.
And the "warning" only says "IP" and "MAC".
I can already here it... support wildcards......and vendor
So I have a device connected from a different source using Pihole as dns server on a VPN. I assume I would probably see my interface as eth0 since the interface is not directly connected to the pi.
Yes. The table I put from top to bottom. The first hit counts. IP is more important than MAC address which is more important than hostname which is more important than interface.
Yeah, I did not update the web interface at all for this. Still do be done.
wildcards: Already implemented with IP/CIDR. I don't think wildcards will be needed for MACs, host names or interfaces.
vendor: Why would I add rules for a specific vendor to be blocked? I see no reasoning here. If you want to block specific, e.g., Samsung domains to prevent Samsung devices from talking home ... you should probably just block them for all devices.
edit I think what I added in here can be very useful. If we go ahead and allow more and more, then users will use it even when they should not. Because a lot of users tend to overcomplicate things when there is no need to. It takes a lot of experience to see the simple solution, sometimes.
I will edit the priority into my table above.
Another, rather small update to this experimental branch: As the ip address is a UNIQUE field in the network_addresses table, new assignments of a previously seen IP address will automatically move an already known old address to the new device. This makes removing old entries somewhat less important. However, the new possibility to specify clients by host names may make it more important to be able to remember devices not seen for longer so they can get recognized sooner.
Hence, we disable the automated removal for now, but leave the FTL config option MAXNETAGE so users can re-enable this if they feel a need for it (setting it to a positive value will re-enable automatic removal of entries).
We'll have to see how this goes.
Automated rescanning of client groups a certain time after their first query (if their host names are only known a few minutes later, etc.) is still a to-do.
