Before running pihole -up to update, make sure you’ve read over the release notes below to ensure there’s nothing that would stop you from upgrading. Pi-hole docker containers will be updated in due course… (before anyone asks “Where’s the Docker update??”)
Update: Docker containers now tagged at :v5.4
Core:
Only use ‘Location:’ header for FTL update check on Github API #3987 (@dschaper)
Add new query types SVCB and HTTPS #1010 (@DL6ER)
These queries where shown as OTHER before. Now they can be distinguished on the query log pages and are included as separate items in the daily statistics.
Detect and handle interface changes of clients with the same IP #999 (@DL6ER)
Update SQLite3 to 3.34.0 #1006 (@DL6ER)
We also expose a full-fledged SQLite3 shell interface through pihole-FTL sqlite3
Fixes
Fix for errno not being set by posix_fallocate()#1023 (@DL6ER)
Check for validity of iface pointer before dereferencing it #1018 (@DL6ER)
Don’t show retried queries when filtering for blocked queries #1022 (@Daxtorim)
Do not record NXDOMAIN from DNS cache as “regex blocked” #1014 (@DL6ER)
Added missing NS query type to getQueryTypes()#1007 (@Daxtorim)
Do not return regex idx if privacy level > 0 #1016 (@DL6ER)
Ah interesting, does it mean that the sqlite(3) package installed as dependency is actually obsolete, especially now that pihole-FTL sqlite3/sql CLI is available?
I guess if all calls of sqlite in the source would have been changed to pihole-FTL sqlite3 it could work. I'm not sure about the order things happen in the install script: database actions might take place before pihole-FTL is downloaded and available which would not work if sqlite is not present.
The posts above about symlinking the pihole-FTL -> sqlite3 kinda verify it already, but you're right, for the installer it needs to be reviewed and tested. Another step towards standalone all-in-one Pi-hole binary.
Not really. It's not a hard dependency but adding more complexity to the installer to find out if the package is installed and then further making sure that users don't try to install the repo package isn't in the plans.
We could easily add a symlink but there's no way to prevent that from being overwritten by apt. And calling pihole-FTL sqlite3 would be okay as long as the user didn't have a different version of sqlite3 installed and tried to use that version when interacting on the command line.
For now it's a nice thing to have, you can do lua in the same way. Though it's a lot easier to have a shebang of #! /usr/local/bin/pihole-FTL lua on our scripts and not have to pull in the lua package.
So the sqlite3 command is still used by Pi-hole itself? When replacing those calls with pihole-FTL sql, it would not matter what the user does, if sqlite3 is installed or not. Indeed creating the symlink breaks the package install, so that is only something that users could do manually for convenience or to make the command available for other scripts/programs.
Ah, the PHP module does not require the command, does it?
And we'll never know what sqlite3 is being used when users read things that tell them to run sqlite3 to modify the database. Could be a repo version that is older than pihole-FTL sqlite3 and breaks the database if the schema isn't recognized by the older version or the older version makes changes.
Within a stable repo suite, such breaking changes do not happen, but I got your point that you want to be sure about that. True that while backwards-compatibility (new sqlite3 with old database) is nearly guaranteed to work, the other way round it might not (old sqlite3 with new database), and not all distros have as strict rules as Debian has for merging upstream package versions into their release.
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
When i use pihole-FTL sqlite3 on the command line, I get
Pi-hole FTL v5.4 2021-01-15 19:30:02 +0000
SQLite version 2020-12-01 16:14:00
Enter ".help" for usage hints.
Connected to a transient i3.34.0n-memory database.
Use ".open FILENAME" to reopen on a persistent database.
I assume, by adding the symlink, the system, including scripts, will always use the embedded version of pihole-FTL...
Q: would adding the symlink to the system, thus using the pihole-FTL version of sqlite3 (3.34.0) eliminate the problem I sometimes have, e.g. eliminate the "database is locked" error, when executing (scripted) sqlite3 commands?
Q: what happens if a new package version becomes available in the raspbian repository, do you need to exclude the sqlite3 package from updating?
Yes, that's the plan. Embedding dnsmasq into FTL made our scripts a lot easier as we could remove all the if(version == ...) elif(version == ...) ... stuff. Only later on, we hooked more inanimately into dnsmasq to make possible what Pi-hole can do today (regex, groups, CNAME inspection and all the things).
Embedding SQLIte3 into FTL was also meant to give us precise control about what we can expect for our scripts. Exposing the sqlite3 CLI to the user via various methods (including the symlink drop-in replacement) was just a convenience feature for those who always want up-to-date sqlite3 on their systems without yet another third-party repos or compiling from source.
The main driver for this was the new UPDATE FROM which only became part of sqlite3 last fall. This command will likely never arrive in existing/older repositories. Sometime in 2020 is too bleeding-edge for them (and that's okay!).
It is planned to (read as: I plan to) replace sqlite3 by pihole-FTL sqlite3 in all the scripts, but I haven't found the time to do so.
We should discuss this. My plans are:
Remove sqlite3 as dependency from the installer. New installs will not get it, existing ones won't care (it would also not be removed)
Use pihole-FTL sqlite3 everywhere in the scripts. pihole-FTL will be available before the first gravity run which is where sqlite3 will be used for the first time.
If users what to do something where sqlite3 is needed, they will get command not found. They can then either install the OS package or symlink (if they are aware of this). Those who want to play with the CLI likely already have the binary.
I don't see us making the symlink for user automatically, even not if the binary is not installed. apt overwriting our symlink if they later decide to install the OS package seems correct, whichever decision we make. This can be debated, for sure.
We're all looking towards Pi-hole v6.0 here. I don't expect much further development for the v5.x branches - except frm bug fixes, of course. Pi-hole v6.0 currently uses the embedded PH7 library for interpreting PHP. Pi-hole v6.0 already has zero dependencies on any external web server or PHP (so also not on PHP extensions like php-sqlite3).
Thank's for the detailed explanation, sounds awesome. Have you read Dan's concerns that pre-installed outdated sqlite3 versions might break databases written by the new pihole-FTL embedded version? I have no idea whether there is some kind of minimal required version string in SQLite databases to rule out any incompatibilities in the first place.
What I didn't thing about previously is that even if sqlite3 is a dependency, the installer does not update the package, if it is installed already. And custom compiled binaries usually override the ones from packages via /usr/local/bin (when keeping common make target dir the default PATH order), so keeping sqlite3 as dependency does rarely affect the version that is used.
I guess so, I've seen many comments about breaking changes not being backwards-compatible, e.g., here:
After ADD COLUMN has been run on a database, that database will not be readable by SQLite version 3.1.3 (2005-02-20) and earlier.
so they must have some mechanism to check/control this. Checking the source of sqlite we see some version numbers such as 3 (VFS struct version number) and others (the source of the default build is currently 231,754 lines of code so I didn't check everything).
