Gravity Domain Filtering Different

sawsanders · October 19, 2023, 3:42am

There seems to be a difference between the V5 and V6 gravity domain filtering.

Setup: I have one bare metal Pi-hole running V5 and another one running V6. V6 is set as main DNS server for network.

I use this list https://raw.githubusercontent.com/chadmayfield/my-pihole-blocklists/master/lists/pi_blocklist_porn_all.list to filter porn sites for my kids devices.

V5 gravity accepts all entries:

  [i] Target: https://raw.githubusercontent.com/chadmayfield/my-pihole-blocklists/master/lists/pi_blocklist_porn_all.list
  [✓] Status: Retrieval successful
  [✓] Parsed 1906594 exact domains and 0 ABP-style domains (ignored 0 non-domain entries)
  [i] List stayed unchanged

V6 rejects many:

[i] Target: https://raw.githubusercontent.com/chadmayfield/my-pihole-blocklists/master/lists/pi_blocklist_porn_all.list
  [✓] Status: Retrieval successful
  [✓] Parsed 1905909 exact domains and 0 ABP-style domains (blocking, ignored 685 non-domain entries)
      Sample of non-domain entries:
        - "anime5pornlp1663.blogspot.r-porn-dvd-clips-.blogspot.ae"
        - "anime5pornlp1663.blogspot.r-porn-dvd-clips-.blogspot.al"
        - "anime5pornlp1663.blogspot.r-porn-dvd-clips-.blogspot.am"
        - "anime5pornlp1663.blogspot.r-porn-dvd-clips-.blogspot.ba"
        - "anime5pornlp1663.blogspot.r-porn-dvd-clips-.blogspot.be"

My MacBook will resolve those IP's using Pi-hole V6 as resolver:

% dig anime5pornlp1663.blogspot.r-porn-dvd-clips-.blogspot.ae

; <<>> DiG 9.10.6 <<>> anime5pornlp1663.blogspot.r-porn-dvd-clips-.blogspot.ae
;; global options: +cmd

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56485
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232

;; QUESTION SECTION:
;anime5pornlp1663.blogspot.r-porn-dvd-clips-.blogspot.ae. IN A

;; ANSWER SECTION:
anime5pornlp1663.blogspot.r-porn-dvd-clips-.blogspot.ae. 300 IN CNAME blogspot.l.googleusercontent.com.
blogspot.l.googleusercontent.com. 300 IN A 142.251.33.97

;; Query time: 153 msec
;; SERVER: 192.168.10.25#53(192.168.10.25)
;; WHEN: Wed Oct 18 20:34:39 PDT 2023
;; MSG SIZE rcvd: 146

V5 log: https://tricorder.pi-hole.net/PvqnlGax/

V6 log: https://tricorder.pi-hole.net/vp7OaIJo/

DL6ER · October 19, 2023, 5:43pm

This is a regression of

github.com/pi-hole/FTL

Run gravity parseList without regex

pi-hole:development-v6 ← pi-hole:tweak/gravity_booster

opened 12:01PM - 15 Oct 23 UTC

DL6ER

+142 -63

# What does this implement/fix? This PR seeks to accelerate the `gravity pars…eList` process by running it without regular expressions. This relaxes the tests slightly, however, even after adding many additional adlists, I did not see a single difference in reporting of invalid domains. Context: [Discourse thread "Slow gravity update 6.0 beta"](https://discourse.pi-hole.net/t/slow-gravity-update-6-0-beta/65457) As benchmark, I am checking how long `pihole-FTL gravity parseList` needs for ``` [i] Target: https://raw.githubusercontent.com/mkb2091/blockconvert/master/output/domains.txt [✓] Status: Retrieval successful [✓] Parsed 1279214 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries) ``` FTL `development-v6` built on Alpine: ``` real 0m22.213s user 0m10.204s sys 0m11.936s ``` FTL `development-v6` built on Debian: ``` real 0m0.959s user 0m0.826s sys 0m0.088s ``` this branch built on Alpine: ``` real 0m0.876s user 0m0.772s sys 0m0.060s ``` this branch built on Debian: ``` real 0m0.781s user 0m0.662s sys 0m0.072s ``` **Related issue or feature (if applicable):** N/A **Pull request in [docs](https://github.com/pi-hole/docs) with documentation (if applicable):** N/A --- **By submitting this pull request, I confirm the following:** 1. I have read and understood the [contributors guide](https://docs.pi-hole.net/guides/github/contributing/), as well as this entire template. I understand which branch to base my commits and Pull Requests against. 2. I have commented my proposed changes within the code. 3. I am willing to help maintain this change if there are issues with it later. 4. It is compatible with the [EUPL 1.2 license](https://opensource.org/licenses/EUPL-1.1) 5. I have squashed any insignificant commits. ([`git rebase`](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)) ## Checklist: - [x] The code change is tested and works locally. - [x] I based my code and PRs against the repositories `developmental` branch. - [x] I [signed off](https://docs.pi-hole.net/guides/github/how-to-signoff/) all commits. Pi-hole enforces the [DCO](https://docs.pi-hole.net/guides/github/dco/) for all contributions - [x] I [signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) all my commits. Pi-hole requires signatures to verify authorship - [x] I have read the above and my PR is ready for review.

because

is considered an invalid pattern in the following code:

			// Domain must not contain a hyphen immediately before a
			// dot or two consecutive dots
			if(domain[i] == '.' && (domain[i-1] == '-' || domain[i-1] == '.'))
				return false;

This needs to be relaxed, see the following proposed bugfix PR

github.com/pi-hole/FTL

Fix regression of #1667

pi-hole:development-v6 ← pi-hole:fix/gravity_dash_dot

opened 05:42PM - 19 Oct 23 UTC

DL6ER

+13 -21

# What does this implement/fix? Subdomains beginning and ending in dash are a…ctually fine (pattern `[a-z0-9_-]{0,63}\.`), move the corresponding tests to a concluding TLD check region (pattern `[a-z0-9][a-z0-9-]{0,61}[a-z0-9]`) **Related issue or feature (if applicable):** N/A **Pull request in [docs](https://github.com/pi-hole/docs) with documentation (if applicable):** N/A --- **By submitting this pull request, I confirm the following:** 1. I have read and understood the [contributors guide](https://docs.pi-hole.net/guides/github/contributing/), as well as this entire template. I understand which branch to base my commits and Pull Requests against. 2. I have commented my proposed changes within the code. 3. I am willing to help maintain this change if there are issues with it later. 4. It is compatible with the [EUPL 1.2 license](https://opensource.org/licenses/EUPL-1.1) 5. I have squashed any insignificant commits. ([`git rebase`](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)) ## Checklist: - [x] The code change is tested and works locally. - [x] I based my code and PRs against the repositories `developmental` branch. - [x] I [signed off](https://docs.pi-hole.net/guides/github/how-to-signoff/) all commits. Pi-hole enforces the [DCO](https://docs.pi-hole.net/guides/github/dco/) for all contributions - [x] I [signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) all my commits. Pi-hole requires signatures to verify authorship - [x] I have read the above and my PR is ready for review.

sawsanders · October 19, 2023, 7:05pm

Thanks for the quick fix and I appreciate your work on this awesome project.