Long Term Data should be storing all requests from all clients
Actual Behaviour:
Long term Data only shows data for approx 1 day, then stops storing activity. If I re-boot the pi, it works, but again for only approx 1 day, then stops
Iv'e searches many topics on this and can confirm...
PHP-SQLite3 is installed
the pi has 664 permissions on /etc/pihole/pihole-FTL.db
the pi has write permissions on same
If I use the "Query Log" and click "Show all", then I can see everything, it's just when using "Long Term Data" I get this weirdness.
Illegal characters in some of your device names seem to cause database write errors, e.g. heth's-tablet.
Note that a couple of Internet standards restrict hostnames to only contain letters 'a' through 'z' (in a case-insensitive manner), the digits '0' through '9' and the hyphen/minus character ('-').
As this seems to disable the database connection in return, it may well be the root cause for lack of long-term database entries.
Removing the apostrophe from the name (as in heths-tablet) should fix this.
Pi-hole's web UI does actively safeguard against invalid hostnames.
How did you provide the names for your devices?
Update - I renamed all devices I could find to remove any illegal characters.
My last entry in long term data was 3 days ago, yet in the query log, I see everything upto the last few minutes. Additionally, If I try to run a Long Term Data Query for a long date range (i.e. "This year") I get an "Unexpected error"
Number 3 is now implemented in branch new/all_clients_network_table where we are doing some work on the code around the network table, anyways. The connected PR is
FTL receives a reply from someone (typically the embedded dnsmasq, could be some other source as well)
FTL stores what it got in the database
I think it is okay to store what we received as reply to our PTR earlier? When we receive something "invalid", the replying DNS server seems to be the one to blame.
If the PTR data (payload) is RFC compliant then store it. If the data is invalid then I wouldn't keep it. There's no use for it and the invalid payload may contain something we don't want.
Next iteration: Host names are now validated for...
Length (maximum length is given by MAXHOSTNAMELEN given by the system through asm-generic/param.h, defaulting to 64, can be overwritten during compilation)
If the maximum length is exceeded, FTL truncates the hostname and logs:
[2020-03-23 14:59:00.475 14511] WARN: Hostname of client 192.168.0.219 too long, truncating to 64 chars!
Note: The truncated version of the host name is used
Valid characters (we accept A-Z a-z 0-9 _ - .)
If invalid characters or binary blobs are found, FTL does not import the hostname and logs:
That's in the responsibility of the client If I recall correctly what was in the RFC. I just checked, Firefox correctly does this (try www.allestörungen.de). If you directly fire this query through dig, it will not be converted. However, FTL A.K.A. dnsmasq is still able to handle them and we're only talking about client host names here. This validation does not touch the regular DNS server operation in any way.
This code has been merged into the regular beta code so everyone will benefit. Please re-checkout the beta branches to continue receiving updates and new features:
