Apply Pi-Hole blocking to CNAMEs

I'm sorry, I have been deeply buried in code and have not followed each and every message in this discussion 100%. I makes no sense to pretend otherwise, it would just not be honest.

My FTL implementation idea is/was (basic but hopefully clear description):

Assume my DNS data from above:

FTL would check:

  • f7ds.liberation.fr
  • liberation.eulerian.net
  • atc.eulerian.net

If, at least, one of these three is a bad guy, the entire query will be blocked. The check would happen, as usual, against all black- and whitelist entries (both exact and regex) as well as all gravity domains (the formally mentioned performance penalty is mostly gone as of this morning).

So you might want to highlight why you say I'd need to consider this or that special part in FTL. Maybe the discussion already went on and into deeply into something I've missed. It would be great if this could be pointed out then.

seems it's time for a code exchange given that nextdns is, in fact, using pihole. :thinking:

also i've had a lot of stuff break via mobile today...android using nextdns as "private dns" solution. haven't had time to attribute it to (or rule out) their cname resolution upgrade though.

NextDNS is not using Pihole, they just say it's kind of like Pihole in the cloud. They use Unbound and their own software I think.

The example you provided, is actually using the CNAME to disguise the third party domain, so blocking the entire query is the correct decision. Some of the blocklist have already added f7ds.liberation.fr, so this is no longer a valid test case.

pihole -q f7ds.liberation.fr
 Match found in list.5.dbl.oisd.nl.domains:
   f7ds.liberation.fr
 Match found in list.18.pgl.yoyo.org.domains:
   f7ds.liberation.fr
 Match found in list.40.raw.githubusercontent.com.domains:
   f7ds.liberation.fr

I have been running my script (see earlier entry), and it appears that this (cname is in blocklist, domain isn't) is happening a lot more than you would expect. This implies a lot of things that should be blocked, aren't blocked at all, examples:

0.0.0.0 globalredir.akadns.net # CNAME a23.g.akamai.net found in gravity list
0.0.0.0 csw.lenovo.com.edgekey.net.globalredir.akadns.net # CNAME a23.g.akamai.net found in gravity list
0.0.0.0 com-c-3.edgekey.net.globalredir.akadns.net # CNAME a23.g.akamai.net found in gravity list
0.0.0.0 microsoft.com-c-3.edgekey.net.globalredir.akadns.net # CNAME a23.g.akamai.net found in gravity list

Will everything will still work after pihole-FTL is released with CNAME detection, that will be the bigger question. The blocklist(s) do contain the CNAME, but blocking is currently never triggered.

However, I'm also getting some entries where the cname and the domain are in the same dns zone, example:

0.0.0.0 www.nieuwsblad.be # CNAME nieuwsblad.be found in gravity list

When this CNAME is encountered, the site would be blocked, using your logic.
The question is if this isn't a valid use of a CNAME, isn't this what it was initially suppose to do?

After careful consideration, it's probably NOT wise to make an exception in the pihole-FTL code, as it would increase processing time.
Because the CNAME is never detected in the current version of pihole-FTL, but would be in one of the next releases (hopefully), users will suddenly be confronted with false positives, which they will have to whitelist. You should definitely issue a warning, whenever pihole-FTL is going to block cnames.

If one of the domains that occur in the chain of things is in gravity, then I think it is the correct action to block the query. I don't see how a CNAME that contains something like badguy.com can have some valid content in the end (assuming badguy.com is in gravity). I'd say it is rather unlikely that the CNAME goes valid content -> tracking website -> valid content. It is rather likely that the chain ends in the tracker being served.

To your example: If nieuwsblad.be is in gravity, it is rather likely that you also don't want www.nieuwsblad.be, I may not be seeing what you wanted to say here?

1 Like

What does

sqlite3 /etc/pihole/gravity.db "SELECT * FROM vw_gravity WHERE domain = 'liberation.eulerian.net';"

return?

Just wanted to give a big thanks for the team for even looking at this! THANK YOU!

@DL6ER has been banging it out folks.
I know you have been knee deep in it. Thanks again!

Commits on Nov 21, 2019

1. [Add pure attribute to vGet() to signal that this function might be el…](https://github.com/pi-hole/FTL/commit/7fd801dbc250fd706bdf408a624619fee96b75f4)

[ ![@DL6ER|20x20](https://avatars3.githubusercontent.com/u/16748619?s=60&v=4) ](https://github.com/DL6ER)

[DL6ER](https://github.com/pi-hole/FTL/commits?author=DL6ER) committed 2 days ago

[Add a generic dynamic vector implementation. I tested my implementati…](https://github.com/pi-hole/FTL/commit/0baa56c3c3a7486ff48515a050f8df8ec45645db)

Commits on Nov 22, 2019 

[ ![@DL6ER|20x20](https://avatars3.githubusercontent.com/u/16748619?s=60&v=4) ](https://github.com/DL6ER)

[DL6ER](https://github.com/pi-hole/FTL/commits?author=DL6ER) committed 2 days ago

[Fix: Only set domain as not-to-be-blocked when this is true.](https://github.com/pi-hole/FTL/commit/4269897b58bbacca732e6275198a8c0ceecaf803)

[ ![@DL6ER|20x20](https://avatars3.githubusercontent.com/u/16748619?s=60&v=4) ](https://github.com/DL6ER)

[DL6ER](https://github.com/pi-hole/FTL/commits?author=DL6ER) committed yesterday

Signed-off-by: DL6ER <dl6er@dl6er.de>

[Add more debugging output.](https://github.com/pi-hole/FTL/commit/7fc340c16affbc70de72146792f4310bf12bf5e0)

[ ![@DL6ER|20x20](https://avatars3.githubusercontent.com/u/16748619?s=60&v=4) ](https://github.com/DL6ER)

[DL6ER](https://github.com/pi-hole/FTL/commits?author=DL6ER) committed yesterday

[Remove static instruction from query_blocked() as it is called from p…](https://github.com/pi-hole/FTL/commit/25a403a7c4ddb45e70c062795e602a1b8296fa5b)

[ ![@DL6ER|20x20](https://avatars3.githubusercontent.com/u/16748619?s=60&v=4) ](https://github.com/DL6ER)

[DL6ER](https://github.com/pi-hole/FTL/commits?author=DL6ER) committed yesterday

[Improve performance significantly. We store if (and if: why) a certai…](https://github.com/pi-hole/FTL/commit/faac7005d173dd95bf654addfedbcfd39ee0a421)

[ ![@DL6ER|20x20](https://avatars3.githubusercontent.com/u/16748619?s=60&v=4) ](https://github.com/DL6ER)

[DL6ER](https://github.com/pi-hole/FTL/commits?author=DL6ER) committed yesterday

You're assumptions are correct. If it isn't "BAD", it shouldn't be in a blocklist. It's always possible to whitelist it.

Could not agree more.
Also, is that toggle already in uBlock Origin? Can't seem to find it. Thanks

I implemented CNAME during-the-path blocking for branch new/internal-blocking. This is the branch allowing also per-client blocking rules.

There are two points worth mentioning:

  1. Blocked replies are currently replied to with NODATA instead of the configured blocking mode (technical limitation, which does not yet hurt at this point).
  2. More tests are outstanding.

One of the tests I did:

  • Removed f7ds.liberation.fr from gravity
  • Added atc.eulerian.net to my exact blacklist

Result:

$ dig f7ds.liberation.fr

; <<>> DiG 9.10.3-P4-Debian <<>> f7ds.liberation.fr
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42224
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;f7ds.liberation.fr.            IN      A

;; Query time: 19 msec
;; SERVER: ::1#53(::1)
;; WHEN: Mon Nov 25 12:59:54 CET 2019
;; MSG SIZE  rcvd: 47

A post was split to a new topic: Issue with Gravity (development branch)

Thank you guys for working on this topic so hard!

In the meantime, it seems there is another tracking company actively suggesting the CNAME method:

Measure accurately with first-party tracking

Gain at least 20% more data compared to other players in the market. With Ingenious, you bypass ad-blocking technology and collect accurate data for accurate analysis.

According to this Github comment: Address 1st-party tracker blocking · Issue #780 · uBlockOrigin/uBlock-issues · GitHub

Looks like they recommend their customers to use marketing.net.*.

Example:
Non-authoritative answer:
marketing.net.brillen.de anonical name = tr-brillen-de.affex.org.
tr-brillen-de.affex.org canonical name = lb1.affex.org.
Name: lb1.affex.org
Address: 35.187.117.15

Non-authoritative answer:
marketing.net.home24.de canonical name = tr-home24-de.affex.org.
tr-home24-de.affex.org canonical name = lb1.affex.org.
Name: lb1.affex.org
Address: 35.187.117.15

I hope it might help you to get them blocked as well.

If

is on gravity or your personal block list, it should(TM) already work.

If you block all domains containing or starting with "marketing.net" using regex, this fixes this specific problem.

I'm really trying to follow the analogies but I can't relate this to DNS in any way.

1 Like

Friendly reminder to all participants in this thread. Please lets keep the discussion to specifically discussing this feature request in a technical manner. No sales pitches need to be made, we're already working on it.

No need for analogies or hyperbole. I'd really rather not have to lock the continued discussion of this topic down to developers only, but if we get much more noise in the thread then it will be difficult for newer readers to follow along with.

This has been a friendly mod announcement. Lets keep things on topic. Thanks! :heart:

4 Likes

I have now new/internal-blocking active on my backup Pi-hole so I don’t have to revert every time.

Removed f7ds.liberation.fr from gravity:

sqlite3 /etc/pihole/gravity.db "DELETE FROM gravity WHERE domain='f7ds.liberation.fr';"

atc.eulerian.net is added to exact blacklist.

pihole restartdns to be sure and then ran dig.

I have a NOERROR instead of the expected NODATA and the query time is 12ms. For local resolving is less than 2 ms and 12ms is also not a external resolve which is above 40ms.

; <<>> DiG 9.11.5-P4-5.1-Raspbian <<>> f7ds.liberation.fr
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35367
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;f7ds.liberation.fr.            IN      A

;; Query time: 12 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Nov 27 16:47:03 CET 2019
;; MSG SIZE  rcvd: 47

Taking atc.eulerian.net from the exact list then it is not blocked anymore:

; <<>> DiG 9.11.5-P4-5.1-Raspbian <<>> f7ds.liberation.fr
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10563
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;f7ds.liberation.fr.            IN      A

;; ANSWER SECTION:
f7ds.liberation.fr.     1710    IN      CNAME   liberation.eulerian.net.
liberation.eulerian.net. 5310   IN      CNAME   atc.eulerian.net.
atc.eulerian.net.       5310    IN      A       109.232.197.179

;; Query time: 10 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Nov 27 16:50:19 CET 2019
;; MSG SIZE  rcvd: 118

Those 12 ms is not a problem and the CNAME is blocked and so success!

In my earlier version of this posting I thought that new/internal-blocking was merged into FTL development but it seems to be the other way around.

The delay (likely) comes from that we don't inject the CNAME into the DNS cache when the path is blocked half-way through. Hence, the CNAME path needs to be re-preocessed every time. I might be able to come up with an improved implementation but all things DNS cache are rather tricky to get right...

@anon55913113 Can you check in your pihole.log if this query gets forwarded at all or if the 12msec are truely local processing only? If so, how many domains do you have in gravity, and the black- and whitelists?

No, what you see is what is commonly called NODATA. This the response code NOERROR plus an empty response for the requested record (A in your example). The only other negative reply could be NXDOMAIN. This would show up with response code NXDOMAIN and no response for the requested record.
There is no response code NODATA on its own.

A post was split to a new topic: Gravity database issue (branch new/internal-blocking)

The per-client blocking feature includes intermediate CNAME path blocking.
We have been going back and forth on this and came to the conclusion that it is worth delaying the arrival of Pi-hole v5.0 yet more to include both features already here.

Stay tuned!

3 Likes