Originally published at: https://pi-hole.net/2018/10/15/announcing-our-restful-api-contributions-welcome/
In addition to our next gen Web interface, we have been building a new RESTful API. Until today, this repo has been private, but it is now ready for community feedback and help in maturing it for production.
One reason for making this public now is that we believe that the API is ready for you to look at and experiment with. The API is at the same point in its development as FTL was when we first shared that project with the world, and it benefited from the feedback it got in those early stages.
Another reason for opening up the project now is that many of us are back in school or our jobs have been busier. We're hoping by make the work in progress code available now, some community members would help contribute to it and provide feedback.
Goals for the API
As part of replacing the current PHP web interface, the work that was done by the backend PHP code must be moved to a new subsystem. The new web interface only runs in the browser (unlike PHP), so it can not perform the backend duties that the PHP code performed (such as enabling/disabling blocking). To perform these backend jobs, the new API was created.
The API handles the server-side work necessary to support the web interface. This includes communicating with FTL and retrieving the stats it generates, as well as managing the white/black/regex lists and settings. PHP managed to do these tasks, but often issues arose due to the dynamic nature of PHP and the organic way in which the codebase evolved. The new API needed to be able to handle these tasks without introducing bugs and while providing an extendable codebase which could be easily tested.
In addition to PHP, lighttpd is often mentioned in issues, including how it recently broke the config option we use to provide an optional external config. Because the new Web interface only requires a static file server, the API will host the Web interface. This will give us more control over the web server, make changing the Web interface's port simple, and make Pi-hole easier to package into a Docker container.
Technical Details
The API is written in Rust, a language new to the Pi-hole project. Rust is a safe and fast language which matches well with our goals for the API. It is statically typed and prevents whole categories of errors while being productive and extendable.
The API exposes a RESTful HTTP API which is used by the web interface, but also is general enough for other applications. We have seen some users interested in using Pi-hole's (PHP) API, which often required them to read the API's code before understanding how to use it. With this new API, we are making that experience better by following RESTful practices and providing a more intuitive interface.
The API (and the new web interface) were started with testing in mind, and both boast a suite of fine-grained tests. As Pi-hole organically grew, we sometimes ran into issues (in production), which could have been easily found by testing. There are some small test suites in some subsystems, but the system which often has the most bugs (the PHP web interface and API) is very hard to test and must rely on manual testing. This does not scale, and so we strove to find a system which would make testing both a breeze and would be less likely to break. This led us to Rust and the architecture we use in the API's codebase.
The main way the API communicates with FTL is via shared memory. This is still a work in progress, but it is proving to be very effective (API branch, FTL branch). The idea is that FTL will store its data in a shared-memory space so that the API can come in and read from that memory to fulfill requests. This makes their communication very fast and easy to code. Also, because they share memory instead of sending messages over sockets, the communication is statically typed. This means that instead of assuming the type of the data coming from FTL, the API knows exactly what type the data is in. The PHP API had issues which arose out of how it serialized the data returned by FTL. It had to read the data as a string and assume how it was structured. This shared memory connection will either eliminate or greatly reduce those issues.
This shared memory system means FTLDNS will be even lighter as it no longer needs to do the work of preparing the data for the API (as it did for PHP). Instead, it will concentrate on generating the statistics, and the API can read FTL’s data directly.
So take a look at the code for our new API and feel free to send us a pull request.
https://github.com/pi-hole/api
Thanks again for all of your continued support.