Request: Ability to add clients to known client list from Network page

General Beta 5.0
22

I raised the idea of doing a beta every time, let's see if this becomes reality. The likeliness is high. The delay from v5.0 to v5.1 should be much smaller than the last update from v4.3 (which is almost one year old). We're planing more for like two months in between each point release to ship features to users more quickly but v5.0 was such a massive change and improvement in so many directions that it tool a very long time to mature.

What is the output of

sqlite3 /etc/pihole/pihole-FTL.db "SELECT * FROM network_addresses;"

? We can work from there.

23

Just noticed, another device appeared twice.

Screenshot_20200309-085606~2
Screenshot_20200309-085606~21280×232 16.6 KB 

nanopi@nanopi:~$ sqlite3 /etc/pihole/pihole-FTL.db "SELECT * FROM network_addresses;"
10|10.0.1.35|1581332400
5|10.0.1.4|1583391600
11|10.0.1.84|1583564400
4|10.0.1.1|1583571600
2|10.0.1.190|1583575200
6|10.0.1.64|1583575200
8|10.0.1.3|1583575200
12|10.0.1.6|1583575200
1|10.0.1.136|1583575200
9|10.0.1.215|1583575200
3|10.0.1.2|1583575200
7|10.0.1.182|1583575200
13|10.0.40.3|1583578800
14|10.0.30.254|1583578800
15|127.0.0.1|1583578800
16|10.0.1.84|1583578800
16|10.0.1.1|1583614800
17|10.0.40.2|1583679600
18|10.0.1.190|1583701200
19|10.0.1.64|1583730000

  Pi-hole version is v4.3.5-445-g22ce5c0 (Latest: v4.4)
  AdminLTE version is v4.3.2-438-gcb0eb3e (Latest: v4.3.3)
  FTL version is vDev-560685a (Latest: v4.3.1)

edit: more are coming

Bildschirmfoto zu 2020-03-09 19-37-56
Bildschirmfoto zu 2020-03-09 19-37-56980×132 13.6 KB

25

For the first screenshot, I cannot reliably tell what happened as the timestamps are not visible here.

For the second screenshot, the situation was as follows:

  1. The device was known since Feb 5th with its real MAC address.
  2. On March 9th, this device was known to FTL as have made a lot of queries within the last 24 hours, however, it was not known any more to the Pi-hole through the ARP cache as the ARP entry either expired or you rebooted the device while the other was already offline. Hence, FTL had no other chance than adding a new device for it as it shall not assume that devices with the same IP are identical.

edit I'm not sure how to code around this without assuming that devices can be uniquely identified by their IP addresses. For a minute, I was thinking we suppress these extra additions for a running FTL instance by memorizing the database ID of this device so we can update even when the ARP data is gone. Even when this knowledge will not survive a restart, it would help. However, this would have another consequence, namely that devices changing their IP address while FTL is running, will be attributed incorrectly to another entry.

New idea: Instead of adding a new mock device, we can check who was the last device using a given IP address. Then add to this one. Only if an IP address is altogether new, we create a mock device. Even when this would likely resolve what you see, it will still mean we make assumptions somehow and the network page will not be fully precise (as it was before). What do you think?

edit 2: Alternative, maybe better idea: We ignore non-mock devices for which the IP address is already in the database but no ARP entry is found.

26

I'm not sure if I have understood the whole process completely (how long does FTL store entries in its memory - 24h? When do they beginn to count? How often does it update the network table? Every 60 minutes?) but I see that it's difficult without assuming IPs are unique :-/
I think there is no way around to assume devices use the same IP when they are not on the same subnet as there is no unique identifier (like MAC). This is the limitation if you want to add devices from outside of the local subnet to the network table.

I have two ideas - but you may not like them because they increase complexity.
(I assume that for every request from the local subnet there is a ARP entry for at least the time until the cache expires)

  1. Would it consume to much time/resources to update the network table "live" (so not using FTL's memory)? For every IP request FTL gets, it looks in the ARP cache for the corresponding MAC and attributes the request to that MAC. So there should no discordance between FTL and ARP for devices on the local subnet as every device sending requests should be still in the ARP cache.
    IPs not found in the ARP cache must be from other subnets as the cache is probed "live" and there should be no devices in the same subnet without an ARP entry when this device is performing a DNS request. Create a mock-MAC for those IP not found and add it to the network table.

  2. If you don't want to make it "live" you could take "snapshots" of the ARP cache periodically and determine retrospectively which device (IP) had which MAC when it did a certain request and/or if had an entry in the ARP cache at a particular time at all. Then you could again assume for a request at at particular time without an ARP entry that it came from outside of the local subnet and a mock-MAC should be created.

27

Yes

From the moment a query is received on.

Once a minute.

What you suggest is already what we so. We do not do it "live" but when we update the database (so once every minute). This shouldn't be an issue.

This was my assumption as well. However, "real" clients being added with mock-MAC despite the fact that they are on the local subnet seem to suggest a different situation. Furthermore, I haven't seen any mock devices so far in my network (I only run local devices on the same subnet at the moment).
The only added device is 127.0.0.1 for me.

edit: Hiding the mock hardware addresses on the dashboard happens with

pihole checkout web new/all_clients_network_table
1 Like
28

Mhhh can't decide between this and tweak/domain_regex_input.
To many new/exciting changes :slight_smile:

29

Once we reach consensus on one of them, they will hit release/v5.0 and be included in the other branches as well.

30

I see mock devices being added despite the "real" devices are known as well now. We will have to assume some uniqueness of IP addresses, at least within the most recent 12 hours so something similar to avoid them being added.

Issue is that there are really times when a device is not present in the ARP table despite being online and active.

31

So let me summarize: We have clients on the local subnet which are actively using pihole and despite it checks the ARP cache every minute there are situations where the cache has no entry for that device? That's strange....

What about my second suggestion: Taking snapshots of ARP's cache?

You could try to go upwards with the time in small steps, hoping to avoid a 12h assumption. Start with check the snapshot at t-1 minute as well (maybe it is just a ARP hickup and the device was still present 1 minute ago). The earlier we find a device the more likely it is really the "same device".

32

Yes.

The complexity of this is quite large. I really don't think we should code around possible ARP cache defects/hiccups. I pushed my most recent changes that first look up an IP address before adding a new device. There was some small bug with the lastSeen field of IP addresses not being updated which I fixed as well.

33

Going to update, flush network table and see what happens.

A simple way would be to send a ping to a device for which their is no entry in ARP cache, wait 1 sec and look again in the cache - if nothing is there create a mock-address.

34

I do not really want to send pings around every minute ... it somehow doesn't seem clean and can we even be sure that this will work? What if the device has just been disconnected from the network and won't respond to pings any more?

2 Likes
35

Fully agree. This page/tool was meant to be informative as a hint to what may be wrong. It was not meant to start on the slope of network management.

36

Or ping -b <network broadcast address> and get everyone to respond. But that's something you should do manually and not as a part of this pages display.

37

It's been three hours since the update and network flush and network overview is still empty.
Maybe your false-positive filter is a little bit too good :slight_smile:

nanopi@nanopi:~$ sqlite3 /etc/pihole/pihole-FTL.db "select * from network;"
nanopi@nanopi:~$ pihole -v
  Pi-hole version is v4.3.5-445-g22ce5c0 (Latest: v4.4)
  AdminLTE version is v4.3.2-438-ga99bfea (Latest: v4.3.3)
  FTL version is vDev-1caf634 (Latest: v4.3.1)

edit
After 12h still empty. Restarted the pihole device.

edit2

Maybe totally unrelated to the restart, but FTL crashed a few minutes later
Last thing I changed before the crash occurred was to add a new entry to Custom DNS

[2020-03-15 09:00:00.161 5831] ---------------------------->  FTL crashed!  <----------------------------
[2020-03-15 09:00:00.161 5831] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-03-15 09:00:00.161 5831] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-03-15 09:00:00.162 5831] and include in your report already the following details:
[2020-03-15 09:00:00.162 5831] FTL has been running for 339 seconds
[2020-03-15 09:00:00.162 5831] FTL branch: new/all_clients_network_table
[2020-03-15 09:00:00.162 5831] FTL version: vDev-1caf634
[2020-03-15 09:00:00.162 5831] FTL commit: 1caf634
[2020-03-15 09:00:00.162 5831] FTL date: 2020-03-14 12:27:56 +0000
[2020-03-15 09:00:00.162 5831] FTL user: started as pihole, ended as pihole
[2020-03-15 09:00:00.162 5831] Compiled for aarch64 (compiled on CI) using aarch64-linux-gnu-gcc (Debian 6.3.0-18) 6.3.0 20170516
[2020-03-15 09:00:00.162 5831] Received signal: Segmentation fault
[2020-03-15 09:00:00.162 5831]      at address: 0x264
[2020-03-15 09:00:00.162 5831]      with code: SEGV_MAPERR (Address not mapped to object)
[2020-03-15 09:00:00.163 5831] Backtrace:
[2020-03-15 09:00:00.163 5831] B[0000]: 0xaaaac3f07774, /usr/bin/pihole-FTL(+0x31774) [0xaaaac3f07774]
[2020-03-15 09:00:00.163 5831] B[0001]: 0xffffb42b25b8, linux-vdso.so.1(__kernel_rt_sigreturn+0) [0xffffb42b25b8]
[2020-03-15 09:00:00.163 5831] B[0002]: 0xaaaac3efdbb4, /usr/bin/pihole-FTL(parse_neighbor_cache+0x51c) [0xaaaac3efdbb4]
[2020-03-15 09:00:00.163 5831] B[0003]: 0xaaaac3f01018, /usr/bin/pihole-FTL(DB_thread+0x138) [0xaaaac3f01018]
[2020-03-15 09:00:00.163 5831] B[0004]: 0xffffb42447e4, /lib/aarch64-linux-gnu/libpthread.so.0(+0x77e4) [0xffffb42447e4]
[2020-03-15 09:00:00.163 5831] B[0005]: 0xffffb419aadc, /lib/aarch64-linux-gnu/libc.so.6(+0xcfadc) [0xffffb419aadc]
[2020-03-15 09:00:00.163 5831] ------ Listing content of directory /dev/shm ------
[2020-03-15 09:00:00.163 5831] File Mode User:Group  Filesize Filename
[2020-03-15 09:00:00.163 5831] rwxrwxrwx root:root 260 .
[2020-03-15 09:00:00.164 5831] rwxr-xr-x root:root 3K ..
[2020-03-15 09:00:00.164 5831] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-03-15 09:00:00.164 5831] rw------- pihole:pihole 4K FTL-dns-cache
[2020-03-15 09:00:00.164 5831] rw------- pihole:pihole 12K FTL-overTime
[2020-03-15 09:00:00.164 5831] rw------- pihole:pihole 524K FTL-queries
[2020-03-15 09:00:00.164 5831] rw------- pihole:pihole 4K FTL-upstreams
[2020-03-15 09:00:00.164 5831] rw------- pihole:pihole 340K FTL-clients
[2020-03-15 09:00:00.165 5831] rw------- pihole:pihole 98K FTL-domains
[2020-03-15 09:00:00.165 5831] rw------- pihole:pihole 16K FTL-strings
[2020-03-15 09:00:00.165 5831] rw------- pihole:pihole 12 FTL-settings
[2020-03-15 09:00:00.165 5831] rw------- pihole:pihole 120 FTL-counters
[2020-03-15 09:00:00.165 5831] rw------- pihole:pihole 56 FTL-lock
[2020-03-15 09:00:00.165 5831] ---------------------------------------------------
[2020-03-15 09:00:00.165 5831] Thank you for helping us to improve our FTL engine!
[2020-03-15 09:00:00.165 5831] FTL terminated!

Crashes happen at the full hour now

[2020-03-15 09:07:44.194 6488] Compiled 1 whitelist and 23 blacklist regex filters in 13.5 msec
[2020-03-15 10:00:00.181 6488] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-03-15 10:00:00.181 6488] ---------------------------->  FTL crashed!  <----------------------------
[2020-03-15 10:00:00.181 6488] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-03-15 10:00:00.181 6488] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-03-15 10:00:00.181 6488] and include in your report already the following details:
[2020-03-15 10:00:00.181 6488] FTL has been running for 3136 seconds
[2020-03-15 10:00:00.181 6488] FTL branch: new/all_clients_network_table
[2020-03-15 10:00:00.181 6488] FTL version: vDev-1caf634
[2020-03-15 10:00:00.181 6488] FTL commit: 1caf634
[2020-03-15 10:00:00.181 6488] FTL date: 2020-03-14 12:27:56 +0000
[2020-03-15 10:00:00.182 6488] FTL user: started as pihole, ended as pihole
[2020-03-15 10:00:00.182 6488] Compiled for aarch64 (compiled on CI) using aarch64-linux-gnu-gcc (Debian 6.3.0-18) 6.3.0 20170516
[2020-03-15 10:00:00.182 6488] Received signal: Segmentation fault
[2020-03-15 10:00:00.182 6488]      at address: 0x264
[2020-03-15 10:00:00.182 6488]      with code: SEGV_MAPERR (Address not mapped to object)
[2020-03-15 10:00:00.182 6488] Backtrace:
[2020-03-15 10:00:00.183 6488] B[0000]: 0xaaaad372a774, /usr/bin/pihole-FTL(+0x31774) [0xaaaad372a774]
[2020-03-15 10:00:00.183 6488] B[0001]: 0xffff85c965b8, linux-vdso.so.1(__kernel_rt_sigreturn+0) [0xffff85c965b8]
[2020-03-15 10:00:00.183 6488] B[0002]: 0xaaaad3720bb4, /usr/bin/pihole-FTL(parse_neighbor_cache+0x51c) [0xaaaad3720bb4]
[2020-03-15 10:00:00.183 6488] B[0003]: 0xaaaad3724018, /usr/bin/pihole-FTL(DB_thread+0x138) [0xaaaad3724018]
[2020-03-15 10:00:00.183 6488] B[0004]: 0xffff85c287e4, /lib/aarch64-linux-gnu/libpthread.so.0(+0x77e4) [0xffff85c287e4]
[2020-03-15 10:00:00.183 6488] B[0005]: 0xffff85b7eadc, /lib/aarch64-linux-gnu/libc.so.6(+0xcfadc) [0xffff85b7eadc]
[2020-03-15 10:00:00.183 6488] ------ Listing content of directory /dev/shm ------
[2020-03-15 10:00:00.183 6488] File Mode User:Group  Filesize Filename
[2020-03-15 10:00:00.183 6488] rwxrwxrwx root:root 260 .
[2020-03-15 10:00:00.183 6488] rwxr-xr-x root:root 3K ..
[2020-03-15 10:00:00.183 6488] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-03-15 10:00:00.184 6488] rw------- pihole:pihole 4K FTL-dns-cache
[2020-03-15 10:00:00.184 6488] rw------- pihole:pihole 12K FTL-overTime
[2020-03-15 10:00:00.184 6488] rw------- pihole:pihole 524K FTL-queries
[2020-03-15 10:00:00.184 6488] rw------- pihole:pihole 4K FTL-upstreams
[2020-03-15 10:00:00.184 6488] rw------- pihole:pihole 340K FTL-clients
[2020-03-15 10:00:00.184 6488] rw------- pihole:pihole 98K FTL-domains
[2020-03-15 10:00:00.184 6488] rw------- pihole:pihole 16K FTL-strings
[2020-03-15 10:00:00.185 6488] rw------- pihole:pihole 12 FTL-settings
[2020-03-15 10:00:00.185 6488] rw------- pihole:pihole 120 FTL-counters
[2020-03-15 10:00:00.185 6488] rw------- pihole:pihole 56 FTL-lock
[2020-03-15 10:00:00.185 6488] ---------------------------------------------------
[2020-03-15 10:00:00.185 6488] Thank you for helping us to improve our FTL engine!
[2020-03-15 10:00:00.185 6488] FTL terminated!

https://tricorder.pi-hole.net/5wg0is4o6j

edit 3

After changing back to ftl release/v5.0 nothing has crashed.

38

It shouldn't be. Please add

DEBUG_DATABASE=true
DEBUG_ARP=true

to your /etc/pihole/pihole-FTL.conf (create if it does not exist) and run pihole restartdns
This will add a lot of interesting details to your /var/log/pihole-FTL.log and should help us identify why devices are not being added.

Oh, that's not very nice. Can you run a debugging session as described here?

https://docs.pi-hole.net/ftldns/debugging/

39 
[2020-03-15 20:00:00.005 28278] dbquery: "BEGIN TRANSACTION"
[2020-03-15 20:00:00.012 28278] dbquery: "END TRANSACTION"
[2020-03-15 20:00:00.032 28278] dbquery: "INSERT OR REPLACE INTO ftl (id, value) VALUES ( 1, 1584298786 );"
[2020-03-15 20:00:00.043 28278] dbquery: "UPDATE counters SET value = value + 117 WHERE id = 0;"
[2020-03-15 20:00:00.053 28278] dbquery: "UPDATE counters SET value = value + 24 WHERE id = 1;"
[2020-03-15 20:00:00.060 28278] Notice: Queries stored in FTL_db: 117 (took 57.5 ms, last SQLite ID 2103646)
[2020-03-15 20:00:00.060 28278] dbquery: "DELETE FROM queries WHERE timestamp <= 1552762800"
[2020-03-15 20:00:00.062 28278] Notice: Database size is 134.66 MB, deleted 0 rows
[2020-03-15 20:00:00.063 28278] dbquery: "BEGIN TRANSACTION"
[2020-03-15 20:00:00.073 28278] dbquery: "SELECT id FROM network WHERE hwaddr = '78:8a:20:45:43:a7';"
[2020-03-15 20:00:00.111 28278] dbquery: "INSERT INTO network (hwaddr,interface,firstSeen,lastQuery,numQueries,name,macVendor) VALUES ('78:8a:20:45:43:a7','eth0',1584298800, 1584298672, 4, 'cloudkey', 'Ubiquiti Networks Inc.');"
[2020-03-15 20:00:00.114 28278] dbquery: "INSERT OR REPLACE INTO network_addresses (network_id,ip,lastSeen) VALUES(1,'10.0.1.2',(cast(strftime('%s', 'now') as int)));"
[2020-03-15 20:00:00.115 28278] dbquery: "SELECT id FROM network WHERE hwaddr = '3c:97:0e:13:36:c0';"
[2020-03-15 20:00:00.136 28278] dbquery: "INSERT INTO network (hwaddr,interface,firstSeen,lastQuery,numQueries,name,macVendor) VALUES ('3c:97:0e:13:36:c0','eth0',1584298800, 1584298760, 59, 'thinkpad-lan', 'Wistron InfoComm(Kunshan)Co.,Ltd.');"
[2020-03-15 20:00:00.138 28278] dbquery: "INSERT OR REPLACE INTO network_addresses (network_id,ip,lastSeen) VALUES(2,'10.0.1.136',(cast(strftime('%s', 'now') as int)));"
[2020-03-15 20:00:00.139 28278] dbquery: "SELECT id FROM network WHERE hwaddr = 'f0:9f:c2:1e:8f:e9';"
[2020-03-15 20:00:00.169 28278] dbquery: "INSERT INTO network (hwaddr,interface,firstSeen,lastQuery,numQueries,name,macVendor) VALUES ('f0:9f:c2:1e:8f:e9','eth0',1584298800, 1584297914, 0, 'usg', 'Ubiquiti Networks Inc.');"
[2020-03-15 20:00:00.169 28278] dbquery: "INSERT OR REPLACE INTO network_addresses (network_id,ip,lastSeen) VALUES(3,'10.0.1.1',(cast(strftime('%s', 'now') as int)));"
[2020-03-15 20:00:00.170 28278] dbquery: "SELECT id FROM network WHERE hwaddr = 'd4:38:9c:01:ac:6c';"
[2020-03-15 20:00:00.199 28278] dbquery: "INSERT INTO network (hwaddr,interface,firstSeen,lastQuery,numQueries,name,macVendor) VALUES ('d4:38:9c:01:ac:6c','eth0',1584298800, 1584298540, 5, 'sony-xz1-compact', 'Sony Mobile Communications Inc');"
[2020-03-15 20:00:00.199 28278] dbquery: "INSERT OR REPLACE INTO network_addresses (network_id,ip,lastSeen) VALUES(4,'10.0.1.190',(cast(strftime('%s', 'now') as int)));"
[2020-03-15 20:00:00.200 28278] dbquery: "SELECT id FROM network WHERE hwaddr = 'a4:d9:31:60:b1:e3';"
[2020-03-15 20:00:00.226 28278] dbquery: "INSERT INTO network (hwaddr,interface,firstSeen,lastQuery,numQueries,name,macVendor) VALUES ('a4:d9:31:60:b1:e3','eth0',1584298800, 1584298414, 14, '', 'Apple, Inc.');"
[2020-03-15 20:00:00.226 28278] dbquery: "INSERT OR REPLACE INTO network_addresses (network_id,ip,lastSeen) VALUES(5,'10.0.1.183',(cast(strftime('%s', 'now') as int)));"
[2020-03-15 20:00:00.227 28278] Network table: 127.0.0.1 NOT known through ARP/neigh cache
[2020-03-15 20:00:00.227 28278] dbquery: "SELECT network_id FROM network_addresses WHERE ip = '127.0.0.1' AND timestamp > (cast(strftime('%s', 'now') as int)-86400) ORDER BY lastSeen DESC;"
[2020-03-15 20:00:00.227 28278] SQLite3 message: no such column: timestamp in "SELECT network_id FROM network_addresses WHERE ip = '127.0.0.1' AND timestamp > (cast(strftime('%s', 'now') as int)-86400) ORDER BY lastSeen DESC;" (1)
[2020-03-15 20:00:00.227 28278] db_query_int(SELECT network_id FROM network_addresses WHERE ip = '127.0.0.1' AND timestamp > (cast(strftime('%s', 'now') as int)-86400) ORDER BY lastSeen DESC;) - SQL error prepare (1): no such column: timestamp
[2020-03-15 20:00:00.227 28278] check_database(1): Disabling database connection due to error
[2020-03-15 20:00:00.228 28278] db_query_int("SELECT id FROM network WHERE hwaddr = 'ip-127.0.0.1';") called but database is not available!
[2020-03-15 20:00:00.228 28278] dbquery("COMMIT") called but database is not available!
[2020-03-15 20:00:00.228 28278] ERROR: parse_neighbor_cache() failed!

Will run debugging tomorrow.

40

Okay, thanks, I've already found the bug you were seeing, so no need to run a debugging session tomorrow.

An update of FTL should resolve this for you (the binaries are already built and uploaded by the CI)!

1 Like
41

Immediately started working again. So far no duplicates.

1 Like