So I have one host who has 30k requests and if I click on the IP the page just loads and chrome will eventually ask me if i want to kill the page, and it'll never load.
If I want to load that hosts requests I have to flush the logs.
For the logs to load.
Pihole tries to load the entire log before showing the first page
rs6w79mjl3
Yes, we indeed load all queries for a specific client from FTL into your browser so that you can use local pagination and full search. This may not work as expected for 30k entries - I have to admit that I never tried such a high number (I don't have such a verbose client myself).
I don't think we can offer an immediate solution to this, but we are working on a major re-write which may include pagination being realizes in FTL, i.e. everything should be able to load instantaneously.
I have to admit that I never tried such a high number
That's interesting to hear, and also confuses me.
Are these numbers supposed to reset automatically after a given period of time? Because mine do not, the 30k was over several days.
I'm starting to think they are supposed to reset, because if they aren't how could more people not be having this issue and how could you not have this issue?
The stats are for the last 24 hours.
Does that mean you are getting 30k withing every 24 hours or is it the combined contribution of a much longer period in time?
I'm not entirely sure if the 30k was more than a 24 hour period as I wasn't able to load the logs but it seemed like a rather high number to have when I noticed that this host normally has something like <10k, if even that.
That's why I asked if that specific stat reset at a certain time or if it was cumulative.
This all started cause I wanted to know what the 30k requests were and I couldn't find out.
Edit: I just checked that host now, can't load the specific query log for it, takes a long time and eventually chrome asks to kill page. 15k requests. So I guess it doesnt even take the 30k to make loading it a problem.
Statistics are always reset to represent only the most recent 24 hours. Also the data that is queryable via the Query Log page is limited to the very same 24 hours.
Doesn't the Top Lists on the dashboard help you out with that? If it were 30k requests to the same domain it should clearly stand out in the lists.
Who's to say it was the same domain though?
But no, I don't remember seeing any crazy large number in the top domain when I saw the client requests.
Is there any ETA on that re-write? Or is it still very much a WIP?
ETA is something we cannot provide as all of the developers are working on the project only in their spare times and with extra workload, family, you name it, it is very hard to predict when who will have time for what. Anyhow, @Mcat12 seems to be the core developer of the API rewrite at this point so he might want to comment. However, yes, it is very very much WIP as in the proposed rewrite is not even close to being complete and as such cannot even be used in testing environments right now.
Oh, I was just expecting this, as it is the case in maybe > 99% of the cases as it is usually one software/app/whatever that is going mad and trying to phone home and usually that always gets to the same IP. If it connects to a huge amount of (maybe even random) domains than finding the root of the cause my be a tricky thing to do.
You can try the following on your Pi-hole (directly interact with the Pi-hole FTL daemon):
echo ">getallqueries-client 2.3.4.5" | nc 127.0.0.1 4711 > somefile.log
This will print all queries from client 2.3.4.5 into the file somefile.log in the current directory. Even for 30k entries this should be fairly fast and finish within a few seconds (mainly determined by the write speed of the hard drive/flash disk). Although this output is not meant for visual inspection, it is still fairly easy to parse it also as a human being.
You will get entries like (I edited a header in here for you, it will be missing in the output you'll get)
Unix timestamp | Type | Requested Domain | Requesting client | Status
[...]
1483964299 IPv4 www.google.com 2.3.4.5 2
1483964302 IPv4 www.googleadservices.com 2.3.4.5 1
1483964312 IPv4 www.gstatic.com 2.3.4.5 2
1483964333 IPv4 www.linguee.de 2.3.4.5 2
[...]
where status can be one of the following:
ETA is something we cannot provide as all of the developers are working on the project only in their spare times...
Gotcha, totally understand.
Who’s to say it was the same domain though?
Oh, I was just expecting this, as it is the case in maybe > 99% of the cases as it is usually one 'service going mad'
That does sound like it'd make the most sense, maybe I just didn't look/read the top domain list correctly at the time.
You can try the following on your Pi-hole...
This'll definitely be helpful, at least now I can see the logs if I need to, thank you for the help!
For the web/API rewrite:
It's getting there, and some major features are implemented. I have more time available for the next few weeks than usual, so development should speed up a bit. Also, since we recently released a big update, we have the opportunity to focus more on the rewrite to get it in line with the PHP interface. However, there's still a ways to go, especially as many team members need to get up to speed with the new code.