Pihole should consider adlist content as well to decide if adlist was updated

[yubiuser]

I've noticed that (most of) my adlists get the status Status: Retrieval successful instead of No changes detected when running pihole -g, even when running it twice within a minute.

Looking at the code, I see that the ''Status' is based on HTTP response. The code even states that some servers don't provide the necessary header.

As pihole will save all downloaded lists locally, I request to additionally compare old local lists and (new) downloaded to decide if "Status" should be "changed" or "not changed".

This could be extended to locally saved adlist, which at the moment will never get "Status: no change detected".

This would allow to improve gravity's output by separating it into two lines for each adlist:

Status: Retrieval successful
Status: (No) Changes detected

Furthermore this would reliably allow to determine if the adlist content has been change, even when the server don't provide the necessary HTTP response.

This is also a prerequisite for FR like

Blocklist - Show last modified date

Hi, What I've found over time is that I'm my blocklist (currently 33 individual URLs) is growing to be quite large. Is it possible to include a "Last modified" header field (from the website where the content is being downloaded from) in the list so that I can potentially remove blocklists that are not being maintained by their respective owners? Perhaps add the feature to sort by last modified date so it's easier to quickly identify non-maintained block lists and remove them if applicable. T…

and

'Last modified' for adlists should display modification date of adlist instead of database modification

Since v5.0 pihole displays a mouseover tooltip for each adlist in group management containing information about "Last modified" (field date_modified in gravity.db). From a user's perspective I would expect this date corresponds to the date the content of the adlist has changed. Instead, it shows when the database entry for that adlist changed. This can easily seen by enabling/disabling an adlist and the "Last modified" will change. [Bildschirmfoto zu 2020-07-05 09-21-07] [Bildschirmfoto zu…

[DL6ER]

I was just working on this one when I saw that

@yubiuser

  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful
  [i] Received 57337 domains

looks like this on my Pi-hole:

  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: No changes detected
  [i] Analyzed 58750 domains

Can you confirm the server raw.githubusercontent.com is still not sending the header for you?
I will likely still add this check you requested here, however, I will need some new test lists now

edit @yubiuser can you recommend one for me for testing? I picked a few random ones (not githubusercontent) from https://v.firebog.net/hosts/lists.php?type=tick but all of them seem to be able to deliver the header just fine:

[yubiuser]

Strange that you get a different response..

From today

  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful
  [i] Analyzed 58750 domains

nanopi@nanopi:~$ curl -I https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 1788565
Content-Type: text/plain; charset=utf-8
Cache-Control: max-age=300
Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; sandbox
ETag: "6cfff029d57c53731a9f0b797682f676ed6370af44a99686f04bf242300c052e"
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
Via: 1.1 varnish (Varnish/6.0), 1.1 varnish
X-GitHub-Request-Id: 53EA:DE6F:2FEE7DA:3274584:5FEB14E2
Accept-Ranges: bytes
Date: Tue, 29 Dec 2020 11:47:03 GMT
X-Served-By: cache-fra19131-FRA
X-Cache: HIT, HIT
X-Cache-Hits: 1, 3
X-Timer: S1609242423.314208,VS0,VE0
Vary: Authorization,Accept-Encoding, Accept-Encoding
Access-Control-Allow-Origin: *
X-Fastly-Request-ID: d2099b4af6e09adef7b0c58b92075c4bcc918bc1
Expires: Tue, 29 Dec 2020 11:52:03 GMT
Source-Age: 241

Others do send the header

nanopi@nanopi:~$ curl -I http://dehakkelaar.nl/lists/cryptojacking_campaign.list.txt
HTTP/1.1 200 OK
Date: Tue, 29 Dec 2020 11:52:14 GMT
Server: Apache/2.4.25 (Debian)
Last-Modified: Tue, 29 Dec 2020 04:47:06 GMT
ETag: "38a3-5b79315298b55"
Accept-Ranges: bytes
Content-Length: 14499
Vary: Accept-Encoding
Content-Type: text/plain

---

I enabled all adlists I have available and run gravity twice. I have only one adlist (next to githubusercontent) that does not send the header

https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/hosts

[deHakkelaar]

@yubiuser

nanopi@nanopi:~$ curl -I http://dehakkelaar.nl/lists/cryptojacking_campaign.list.txt

@yubiuser

Last-Modified: Tue, 29 Dec 2020 04:47:06 GMT

Thats oc to remind me I have to create another way to pull the domains from the excel sheet
EDIT: No modified header as well when curl that one.

[DL6ER]

@yubiuser

Strange that you get a different response..

From today
[...]

Don't ask me what happened here, I didn't look at the headers this morning. Just tested this now again and it is back to its usual self:

  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts                           
  [✓] Status: Retrieval successful                                                                                                                                                                                 
  [i] Analyzed 58750 domains 

This feature request is getting implemented by

Store gravity details in gravity.db adlist table

[yubiuser]

This has been implemented.