Force update of IP / hostname / clientname relationship

Help Community Help
1

On several Windows 10 computers, the privacy extensions are enabled, resulting in ever changing IPv6 addresses (restart, reboot, …) I do NOT want to disable the privacy extensions.

I've managed to detect and add (automatically) the (ever changing) IPv6 address and hostname, both in a hostsfile (dnsmasq functionality, doesn't require a restart, using "hostsdir=" syntax ) and as a client (group management, requires /usr/local/bin/pihole restartdns reload-lists).

For some reason, the change is immediately usable (and visible) in the long-term data / query log (search for hostname), but takes some time to be applied to the query log or dashboard (shows the IPv6 address, changes after some time into hostname).

Is there a way to force pihole-FTL to update this relationship, so that the changes are also immediately available in the query log and the dashboard?

2

Wouldn't restartdns reload do what you require already?

For me, it takes about one to five seconds for the display to show hostnames instead of IPs. Is that different and/or too slow for you?

Also (haven't tried that), maybe explicit reverse lookups for the IPv6 addresses you detected may coerce Pi-hole into acknowledgeing them quicker.

I'd also be interested how your automatic detection works.
Does ist rely on passive observations, or would it involve additional client-side software?

6

I don't want to use restartdns, as it takes to long for dns resolution to become available again. I'm using the dnsmasq hostdir= syntax, as soon as a new IPv6 / hostname is entered in the additional hosts file, because it doesn't require any action, you just see (immediately) the following messages in the log file:

Aug  2 22:49:56 dnsmasq[22852]: inotify, new or changed file /etc/pfsense/neighbour
Aug  2 22:49:56 dnsmasq[22852]: read /etc/pfsense/neighbour - 17 addresses

When using this, there is no service interruption, read the dnsmasq man for more info. nslookup and / or dig commands, executed on any workstation immediately show the correct result.

I have a 4 port router, running pfsense. This is the only device, that knows it all, other devices on the LAN ports, including pihole, only know limited (to the physical network) information, when requesting arp (arp -a) or neighbours (ip neigh).
In order to get all the IPv6 info, I remotely execute the command ndp -na from the pi on the router (ssh), on regular intervals (cron). The script, running the remote command, than processes the output, creating a new hostsfile entry (automatically picked up by the original dnsmasq code, embedded in pihole_FTL) and a client entry (sqlite3 command). To activate the client entry, the reload-lists command is used (no noticeable service interruption).
The output (ndp -a) looks like this (IP addresses changed):

2a02:1810:4d02:xxxx:c5a5:yyyy:79e1:zzzz b8:27:xx:yy:38:c7 igb2 23h59m58s S
2a02:1810:4d02:xxxx:20e:yyyyy:fecf:zzzz 00:0e:xx:yy:f3:cf igb2 permanent R
fe80::20e:c4ff:fecf:f3ce%igb1        00:0e:c4:xx:yy:ce   igb1 permanent R

and contains both the IPv6 address and the MAC address. Using a static list (MAC / hostname), it's fairly easy to make the correct hosts and client entries.

As a result, all necessary info is available for pihole-FTL, unfortunately, only the long-term data visualizes the data immediately, the query log and the dashboard require additional time, before the IP address is replaced with the hostnames.

7

Sorry - I obviously missed typing reload, I've corrected that in my original post.

My thought was that beacuse in contrast to reload-lists, reload will flush the DNS cache as well, that may be the reason why you don't see a replacement.

As I said, I see the dashboard displaying correct values after only a few seconds.
How long does that take for you?

As to your

I am not familiar with FreeBSD at all, but I guess ndp is its equivalent for Linux's ip neigh, as they both would handle Neighbour Discovery Protocol. That would mean your router's information is link-local only as well.

Your solution may not work in the presence of L3 switching hardware in your network. For devices behind such a piece of equipment, NDP will report several IP addresses from different devices by that L3 switch's same MAC address.

8

DNS resolution is unavailable for more than 15 seconds, which is a pain in the *** for other users / devices

As stated before (using hostdir= and reload-lists), nslookup and dig immediately show the correct result on any workstation, I wonder why pihole (or at least some portions of the web interface) don't pick up the changes immediately.

When a new IPv6 address becomes active, and the user has a blocking problem, I force run the script, this to update the hosts file and the client entries immediately. It than takes minutes (more than 5, sometimes up to 20) for the query log and the dashboard to pick up the changes. This is especially annoying when having to solve a blocking problem for a new client (changed IPv6 temporary address). If multiple devices have recently started up, the query log shows a lot of IPv6 entries, this makes it more difficult to identify the user / device that has a problem.

Not a problem in my environment, I've tested my solution / script with several devices, for which I have entered a static configuration file (MAC / hostname), works perfectly, only the delay in the web interface remains a problem.

I wonder if the web interface pages handle the (address to hostname) resolution differently...

9

Have you had a look in the log which step takes so long? Maybe there is room for improvement?

10

I cannot recreate your observations:
DNS resolution does not stop for me, and dashboard refreshes with the correct information within a few seconds.
Differences in my setup: I do IP addresss updates manually (of course), I do not use IPv6, and I use addn-hosts and restartdns reload to include definitions.

EDIT: For comparison: I ran my tests on the latest vanilla Pi-hole (no custom checkouts), with half a dozen clients on my network.

11

I generated a debug log (not send to pihole), there are no errors in the log.
I checked pihole.log, no obvious errors
@DL6ER: I checked the pihole-FTL.log (using ftl new/tre-regex), this log has entries I don't like:

[2020-08-03 10:17:46.708 27839M] SQLite3 message: bind on a busy prepared statement: [SELECT * FROM queries WHERE timestamp >= 1596356266] (21)
[2020-08-03 10:17:46.708 27839M] SQLite3 message: misuse at line 83855 of [3bfa9cc97d] (21)
[2020-08-03 10:17:46.713 27839M] Resizing "/FTL-strings" from 8192 to 12288
[2020-08-03 10:17:46.718 27839M] Resizing "/FTL-queries" from 229376 to 458752
[2020-08-03 10:17:46.742 27839M] SQLite3 message: bind on a busy prepared statement: [SELECT * FROM queries WHERE timestamp >= 1596356266] (21)
[2020-08-03 10:17:46.743 27839M] SQLite3 message: misuse at line 83855 of [3bfa9cc97d] (21)

A lot more log entries like this, the list goes on...

I checked all my scripts (filesearch) for the string SELECT * FROM queries WHERE timestamp >=, nothing found, so it doesn't appear to be something I'm initiating, I also ran the majority of my scripts that execute sqlite3 statements, this to verify if it may be some thing I'm causing, no new entries in the pihole-FTL log...

if your not using IPv6, with the windows 10 privacy extensions, you will never have this problem. adding an IPv4 hostname (to custom.list or any other hosts file) and or a client to the database is a one time operation.

The reason(s) for using IPv6, and adding the temporary addresses to hosts and database:

  • IPv6 is here to stay, you better get onboard...
  • More recent programs, I'm using, appear to prefer IPv6 over IPv4, when available.
  • DL6ER once said: If you're not using IPv6, you're missing out on a big part of the internet...
  • I want all pihole clients listed by name, this to ensure no external devices are using my network (when an IP address without a client name is listed in the client list ( telnet 127.0.0.1 4711 - >top-clients withzero (256)), it immediately triggers my attention -> mail)
12

I am not debating your use of IPv6.
I just don't have it enabled myself, so I add and delete lines for existing IPv4s to mimic what your script does.
I don't think it matters for the refresh issue, but thought I'd mention it nonetheless.

(edited for clarity)

13

I wasn't thinking about error but lags in the startup of FTL, especially while counting/compiling regex entries after a reload.

I'm not sure if you are aware of this topic: DNS resolution unexpectedly stops after upgrade to 5.0 - #38 by yubiuser

If you don' have any debug flag enabled, you would see a delay between those two lines

[2020-07-02 11:41:59.853 1202] Reloading DNS cache

[2020-07-02 11:43:26.199 1202] Compiled 0 whitelist and 0 blacklist regex filters in 5.9 msec

With advanced debug flags it appears that pihole hangs at counting domains in vw_gravity
Querying count of distinct domains in gravity database table vw_gravity

And using the special debug branch it turned out to be here

[2020-07-17 17:23:44.892 28893M] Reading regex from database...
[...]
[2020-07-17 17:24:37.655 28893M] Querying count of distinct domains in gravity database table vw_regex_blacklist: SELECT COUNT(DISTINCT domain) FROM vw_regex_blacklist

It seems to be a tough bug, so far only one person encountered it on master but on new/regex I could reproduce it.

14

pihole restartdns reload:

[2020-08-03 11:12:22.614 27841M] Reloading DNS cache

[2020-08-03 11:12:22.692 27841M] Compiled 2 whitelist and 35 blacklist regex filters for 15 clients in 31.6 msec
15

Ok. Then

must hang somewhere else.

Just curious: does pihole restartdns (without any suffix) fixes the problem with dashboard hostname resolution?

16

tried this 2 times (dashboard and stopwatch on a different computer. Restarted the windows computer (ipv6 with privacy extensions) and forced the script to run. result with restartdns (no options):

  • 20 seconds
  • 34 seconds

why does restartdns (no options) pick up the changes almost immediately , while restartdns (reload-list) takes forever?

edit
@DL6ER the sqlite3 errors in the log, mentioned earlier, appear after pihole restartdns but NOT after pihole restartdns reload-lists (currently using MAXDBDAYS=8 and DBINTERVAL=2 in /etc/pihole/pihole-FTL.conf)
/edit

18

I don't like them either. They are created only in a very specific case on a full restart (never on a reload). But you have already seen this. I agree this should be fixed. There is a small bug where we do not check properly for the busy state of the database during initial database importing but only if there are records with the regex ID of a non-CNAME caused block was read from the database.

Fixed now in

pihole checkout ftl fix/read_should_not_bind

@jpgpi250 Please check if this resolves the issue for you. This branch is based on the latest release/v5.2 branch.

19

thanks again for looking into this, much appreciated.

does fix/read_should_not_bind also contain the code from new/tre-regex or do I have to remove the regex (.*;querytype=!A) before switching?

any idea why query log and dashboard have a very long update time (switch from displaying IP to hostname) when using restartdns reload-lists as opposed to restartdns (long-term data / query log doesn't have the problem)?

20

Just leave it, it will be ignored. I also switched from new/tre-regex to an other branch without disabling or removing it.

21

@jpgpi250 @yubiuser is right, just leave it and ignore the regex warning about invalid syntax (if any).

No, this seems separate, the corresponding debug mode (I think it is DEBUG_RESOLVER=true) should show when FTL tries to resolve IP addresses and what the outcome is.

22
  • fairly easy to get the error when running new/tre-regex, a few restarts and bingo
  • impossible to get the error when running fix/read_should_not_bind, force rotated the log and ran (still running):
while : ;do
	sleep 60
	grep "SQLite3 message" /var/log/pihole/pihole-FTL.log
	pihole restartdns
done

must of restarted over 30 times by now, no errors in the log (yes, my log isn't in the default location), so the fix works. thank you!!!

23

Thank your for testing it so quickly. It is scheduled to still make it into v5.2

24

Appreciate your work, but again, just like the new/mac-clients, the new/super-clients branch doesn't solve my, and I assume other users, problem.

You're always assuming that arp -a and / or ip neigh (the equivalent code in pihole-FTL) is able to figure out the MAC address of the clients. My pi is isolated on a separated physical network, and doesn't see anything but the router/firewall (4 port) interface.

I assume neither branches will solve this problem, unless you claim this to be otherwise.

One of the reasons my dashboard, query log and long term data views are a total mess is, as explained earlier here (but that topic unfortunately turned into a FTL problem and fix) is, again I assume, the missing data in the pihole-FTL database

FTL sees, and ads the clients to the network table, but fails to update the name field, example screenshot below

image
image1712×1654 415 KB

FTL sometimes inserts the hostname into the database, but often fails to enter this info, even if the data is available (separate configuration file, added to dnsmasq). If that field contains a value, the dashboard and the query log display the correct info (hostname) after a while, if the field is (remains) blank, the query log displays the ipv6 address, all different because of the windows 10 privacy extensions ( all the IPs in the screenshot are from a single client, db up and running (new v5.1.2) for almost 2 days)

currently, I'm investigating a safe way to inject the names my self, that is one of the reasons I asked for a version indication in pihole-FTL.db,