We have been working on a new Web interface for Pi-hole (referenced internally as Next Gen Admin or NGAdmin). The existing interface built off of AdminLTE has served us well, but we have grown beyond the capabilities of an existing template. We're also looking to implement an HTTP API.
This new interface shares a lot of the visual appearance with the existing one, but it has a lot more going on underneath the hood. One of the biggest benefits is that it removes the need for PHP on the server.
The new interface also:
eliminates the reliance on server-side rendering scripts
eliminates spaghetti code resulted from heavily modifying the base AdminLTE template
reduces attack vectors by using JavaScript's explicit imports instead of importing entire .php files
makes it easier for new developers to figure out the code, which speeds up development
makes the split between client and server code much more explicit
allows us to easily generate fake data for testing
includes all the benefits that come from React (ES6 JavaScript), including automatic DOM manipulations, stateful development, and component-based development
If you are interested in helping develop this new interface, head on over to the GitHub repo. Even if this is your first project, we welcome all contributions.
I definitely agree with the post above me, the pie charts and long graph for clients make it easier to see whats going on rather than the stacked percentage graphs.
The currently shown demo is much in the works and has been started already when the dashboard query types/forwards charts have been still over time instead of pie charts, clients over time didn't even exist and as such does not necessarily present the current state of the work. Also we experience a little bottleneck since a few months as all "core developers" are much more busy than they have been before (changing jobs, moving to new places, etc. etc.).
Currently, the mentioned HTTP API is somewhat higher in priority than the NG web interface as it doesn't make too much sense to code on the web interface without having the API ready to hand out the requested data.
One of the major reasons, if not even No. 1 of them, of making it public was the hope to attract people to look at it and contribute. As such we only put up a working demo that is far from completion to also open up the possibility for the community to shape it as they'd like to. The best way to do this, of course, is by submitting pull requests as it will be much easier for us to "just" review changes instead of having to code it ourselves. We have one core developer who is very experienced with React, but the others are still learning it.
Hi. First comment here. Thanks for this awesome software and all the hard work that went into it. It’s a must to run on any network these days.
The reason I’m commenting is because of how passionate I am about this software. In fact, I was thinking of contributing since I’m a professional developer and open-source enthusiast. One only has 24h a day, so I choose carefully haha.
Anyway, please do not get me wrong, but I would love to join the discussion and ask some questions.
Why is it better without PHP? Of all languages, I believe it has the most programmers world wide. Also, it has great performance and unique benefits. The requirement for PHP is not experienced as a problem for users either, no? The installation handles it?
There is only one (core dev) experienced REACT. Is that smart? Its more niche than PHP. For me personally this might hold me back contributing. I don’t know REACT and as a developer for years, I have seen many niche frameworks come and go. Extra overhead for software. New lingo for devs. Zero benefit for users.
FTL collects and saves data right? PHP reads and interprets? Html/css/js presents, no? How will REACT (js gui framework) replace PHP. Will FTL send it all client-side for processing? Aka, less server power needed but more client-power needed? More data traffic too? I like running pi-hole on a decent powerful server, and with PHP you can cronjob tougher tasks and save results server-side. Personally I prefer that. (running the web gui on apache too).
I think a fresh start is great. Lessons are learned and the software has evolved. Get rid of legacy. But stick with a beautiful simplistic web gui, based on a powerful api so others can fork or make their own. But based on proven and widely used standards. That is my opinion.
Anyway, just me thinking aloud. Please dont see this as an insult or unhappy “customer”. I love pi-hole! Really do.
Our current PHP code base has grown organically and has made adding features more difficult than we'd like. Faced with rewriting it, we decided that strictly splitting the code into an API and client side code would be the most beneficial. Of course all of this is possible while using PHP, but there is more room for growth with this approach. The number of PHP developers is still large, but PHP is falling in popularity and we haven't seen a huge amount of people flocking the contribute because it's in PHP. We've also seen performance issues with PHP, mostly around the Query Log due to its need for high data throughput and general page load speeds.
React is far from niche. It rivals Angular and is rapidly growing in popularity (at a higher rate than Angular and certainly PHP). You need JavaScript on the front end, and using React lets us use many open source components which our previous jQuery interface could not. This will let us develop faster and with more modularity.
We are developing an API which will communicate with FTL over a more efficient layer than what PHP was using, and it will be able to do it quicker than what PHP could achieve. PHP was a slow middleman in the process of connecting the web interface to the client, and the API will bring a more modular and flexible development process to the project. The load will not shift from server side to client side as we are not making the client do much more work than it currently does. The amount of traffic should not be much different. Also, most users run Pi-hole on a Pi, and we want to respect their more limited processing power.
Removing PHP helps remove one of the more troublesome dependencies in terms of support and allows us to create that beautiful and simplistic web interface more easily and quickly. The API will be much nicer and more standard than the current organic one.
I have looked into the workings of pi-hole some more, besides just using it with great appreciation.
If I understand the current working correctly:
pi-hole is a C written application, lets call it the "core". its a service(?)
this core can be interacted with using the command line, singleton, since its a service
this core communicates with DNSMasq, that really does the DNS and optional DHCP stuff
FTL is a second part, complementing the core.
FTL saves statistics for the day in memory and long-term statistics in a DB-file
FTL gets the information by reading logs that the core saves to disk
The core probably gets this information by reading DNSMasq logs(?)
Before FTL, PHP read the core log files directly (and was slow at it, i understand)
Currently PHP uses a socket to connect to either the PHP API or FTL API
The PHP API is the slower one
The FTL API is the faster one
So, to eliminate PHP completely, you need to be able to connect the web interface to FTL without the (current) use of PHP's sockets. Currently both the old PHP way and the new FTL way of getting statistics rely on that PHP socket.
Are you guys planning on using a JS websocket to connect the new web GUI to FTL? This would require FTL to have websocket code. A bit more code. But, you could let go of PHP so you win a lot too of course. Yet, you can't just make a browser connect to anything. You can only upgrade a HTTP-request, so this would mean the webserver needs to be able to handle websockets, plus.. for the webserver to then communicate with FTL. This would require lighttpd to get some custom work done to it (for the lighttpd<>FTL part) leaving nginx and apache to be no longer an option, no?
I'd love to hear if I see the big picture of the workings right, and if my conclusions at the end are right. Most likely I'm still missing something. Hopefully one of you can tell me. I'm especially curious about the (I suppose) JS to FTL connection.
Core Pi-hole is written in Bash: GitHub - pi-hole/pi-hole: A black hole for Internet advertisements
It handles the core routines such as gravity, installing/updating/reconfiguring Pi-hole, managing the lists (white, black, etc), managing access to setupVars for the PHP web interface (this may be moved later though), chronometer, checking the block lists, and some other utilities such as pihole -t.
Core Pi-hole does not have any services of its own (besides dnsmasq and lighttpd).
You got FTL's part right. It provides statistics about the logs generated by Pi-hole Core (dnsmasq). These statistics are used by the web interface and chronometer. Before FTL, PHP would have to parse the dnsmasq log for every query (verrrrrrry expensive and slow).
The PHP web interface has an API file (api.php, which uses api_FTL.php) which connects to FTL over a local socket to get data from it to pass on to whatever was calling the API.
We're developing an HTTP API which will connect to FTL via a Unix socket using the MessagePack protocol to efficiently stream data. Since it runs server-side, it will take the place of the current PHP API, and should be faster too. We considered developing the HTTP library in C inside FTL (and even implemented most of it), but we decided to move that HTTP API functionality to a new program. The reasons for this decision were that we felt that FTL should focus on generating the statistics and implementing a web server in C is not the best idea (especially if we want to easily add advanced features, and other languages are better for that).
You can see the work done in FTL to support this here: https://github.com/pi-hole/FTL/pull/75
We haven't decided when to fully announce the new API, but we hope it will be soon.
Thank you for getting into details. I'm learning and understanding more now.
So, we're looking at..
Pi-Hole core (bash script)
FTL for keeping track of stats
A new component to serve stats from FTL to the webinterface
Webserver of any kind to complete it all
The new component (lets call it Web-API) will use a socket to talk to FTL on the same machine. It uses MessagePack inbetween the two. Cool.
How is the Web-API connected to the Web-GUI though? The GUI gets served by a webserver, and it could request stuff through the webserver obviously. But is it going to talk to the new Web-API around the webserver? through the webserver?
Anyway, agreed in this case that PHP is rather large for the simple job it has in pi-hole. But I see more oppertunities to make it lighter and faster even after PHP is gone.
You said you don't want to create a webserver in C so you can profit from mainstream techniques later on. To me, if the situation above is right, I would be thinking; let's integrate a small simple webserver into the new Web-API. Just serving the few pages the Web-GUI offers, the Web-GUI right now doesn't require any fancy stuff as of now. This way you can get rid of any 3rd party webserver overhead too. Plus, the only thing that should be interesting to have is the powerful and relatively new websocket-tech. Instead of AJAX polling, you can very easily push new stats from the Web-API to the Web-GUI. And the other way around, a few post events from the Web-GUI back to the Web-API. Then the Web-API can fire off core bash commands or talk messagepacked-sockety stuff with FTL and thats it.
What are your thoughts on that?
PS: in the future, SSL would be nice. But since certs in a closed off network are a thorn in developers' eyes for years now (self signing, devices nagging about it), I don't see that getting solved anytime soon.
The web interface will communicate with the API via HTTP requests (the API will be exposed on 0.0.0.0:some_port). We're looking into a few configurations, including serving the web interface through the API like you were thinking (therefore putting it on port 80). That would allow us to not require even Lighttpd, and users can do a reverse proxy to put it in other configurations.
SSL is something that would be helpful for users, and we even have a FAQ now for the current version of Pi-hole if you want to manually add in that functionality:
That might be something that an API could expose for you. We're in the concept stages of looking at ways to get that information to users and it would provide data in a structured format when queried.
If you want to have real-time updates, we should implement that using web sockets. We could use server-side events, but that's not implemented in IE or Edge whereas web sockets are. Ajax would just mean we have to keep a long or short-poll connection to the server, which is not as nice or clean as the first two solutions.
Could you possibly add a few lines to the readme with a developer quickstart guide? This seems like a great project to get familiar with React, but I am a bit out of the loop on the details with the node toolchain. A few quick points on how to build a dev environment would be great.
(therefore putting it on port 80). That would allow us to not require even Lighttpd, and users can do a reverse proxy to put it in other configurations.