Apply Pi-Hole blocking to CNAMEs

DL6ER · November 22, 2019, 10:23pm

I'm sorry, I have been deeply buried in code and have not followed each and every message in this discussion 100%. I makes no sense to pretend otherwise, it would just not be honest.

My FTL implementation idea is/was (basic but hopefully clear description):

Assume my DNS data from above:

DL6ER:

;; QUESTION SECTION:
;f7ds.liberation.fr. IN A

;; ANSWER SECTION:
f7ds.liberation.fr. 307 IN CNAME liberation.eulerian.net. 
liberation.eulerian.net. 7199 IN CNAME atc.eulerian.net. 
atc.eulerian.net. 7199 IN A 109.232.197.179

FTL would check:

f7ds.liberation.fr
liberation.eulerian.net
atc.eulerian.net

If, at least, one of these three is a bad guy, the entire query will be blocked. The check would happen, as usual, against all black- and whitelist entries (both exact and regex) as well as all gravity domains (the formally mentioned performance penalty is mostly gone as of this morning).

So you might want to highlight why you say I'd need to consider this or that special part in FTL. Maybe the discussion already went on and into deeply into something I've missed. It would be great if this could be pointed out then.

cheese · November 22, 2019, 10:29pm

seems it's time for a code exchange given that nextdns is, in fact, using pihole.

also i've had a lot of stuff break via mobile today...android using nextdns as "private dns" solution. haven't had time to attribute it to (or rule out) their cname resolution upgrade though.

ajp2k17 · November 23, 2019, 7:58am

NextDNS is not using Pihole, they just say it's kind of like Pihole in the cloud. They use Unbound and their own software I think.

jpgpi250 · November 23, 2019, 10:37am

The example you provided, is actually using the CNAME to disguise the third party domain, so blocking the entire query is the correct decision. Some of the blocklist have already added f7ds.liberation.fr, so this is no longer a valid test case.

pihole -q f7ds.liberation.fr
 Match found in list.5.dbl.oisd.nl.domains:
   f7ds.liberation.fr
 Match found in list.18.pgl.yoyo.org.domains:
   f7ds.liberation.fr
 Match found in list.40.raw.githubusercontent.com.domains:
   f7ds.liberation.fr

I have been running my script (see earlier entry), and it appears that this (cname is in blocklist, domain isn't) is happening a lot more than you would expect. This implies a lot of things that should be blocked, aren't blocked at all, examples:

0.0.0.0 globalredir.akadns.net # CNAME a23.g.akamai.net found in gravity list
0.0.0.0 csw.lenovo.com.edgekey.net.globalredir.akadns.net # CNAME a23.g.akamai.net found in gravity list
0.0.0.0 com-c-3.edgekey.net.globalredir.akadns.net # CNAME a23.g.akamai.net found in gravity list
0.0.0.0 microsoft.com-c-3.edgekey.net.globalredir.akadns.net # CNAME a23.g.akamai.net found in gravity list

Will everything will still work after pihole-FTL is released with CNAME detection, that will be the bigger question. The blocklist(s) do contain the CNAME, but blocking is currently never triggered.

However, I'm also getting some entries where the cname and the domain are in the same dns zone, example:

0.0.0.0 www.nieuwsblad.be # CNAME nieuwsblad.be found in gravity list

When this CNAME is encountered, the site would be blocked, using your logic.
The question is if this isn't a valid use of a CNAME, isn't this what it was initially suppose to do?

After careful consideration, it's probably NOT wise to make an exception in the pihole-FTL code, as it would increase processing time.
Because the CNAME is never detected in the current version of pihole-FTL, but would be in one of the next releases (hopefully), users will suddenly be confronted with false positives, which they will have to whitelist. You should definitely issue a warning, whenever pihole-FTL is going to block cnames.

DL6ER · November 23, 2019, 12:22pm

jpgpi250:

However, I’m also getting some entries where the cname and the domain are in the same dns zone, example:
0.0.0.0 www.nieuwsblad.be # CNAME nieuwsblad.be found in gravity list
When this CNAME is encountered, the site would be blocked, using your logic.
The question is if this isn’t a valid use of a CNAME, isn’t this what it was initially suppose to do?

If one of the domains that occur in the chain of things is in gravity, then I think it is the correct action to block the query. I don't see how a CNAME that contains something like badguy.com can have some valid content in the end (assuming badguy.com is in gravity). I'd say it is rather unlikely that the CNAME goes valid content -> tracking website -> valid content. It is rather likely that the chain ends in the tracker being served.

To your example: If nieuwsblad.be is in gravity, it is rather likely that you also don't want www.nieuwsblad.be, I may not be seeing what you wanted to say here?

DL6ER · November 23, 2019, 12:29pm

What does

sqlite3 /etc/pihole/gravity.db "SELECT * FROM vw_gravity WHERE domain = 'liberation.eulerian.net';"

return?

drewski · November 23, 2019, 2:37pm

Just wanted to give a big thanks for the team for even looking at this! THANK YOU!

@DL6ER has been banging it out folks.
I know you have been knee deep in it. Thanks again!

Commits on Nov 21, 2019

1. [Add pure attribute to vGet() to signal that this function might be el…](https://github.com/pi-hole/FTL/commit/7fd801dbc250fd706bdf408a624619fee96b75f4)

[ ![@DL6ER|20x20](https://avatars3.githubusercontent.com/u/16748619?s=60&v=4) ](https://github.com/DL6ER)

[DL6ER](https://github.com/pi-hole/FTL/commits?author=DL6ER) committed 2 days ago

[Add a generic dynamic vector implementation. I tested my implementati…](https://github.com/pi-hole/FTL/commit/0baa56c3c3a7486ff48515a050f8df8ec45645db)

Commits on Nov 22, 2019 

[ ![@DL6ER|20x20](https://avatars3.githubusercontent.com/u/16748619?s=60&v=4) ](https://github.com/DL6ER)

[DL6ER](https://github.com/pi-hole/FTL/commits?author=DL6ER) committed 2 days ago

[Fix: Only set domain as not-to-be-blocked when this is true.](https://github.com/pi-hole/FTL/commit/4269897b58bbacca732e6275198a8c0ceecaf803)

[ ![@DL6ER|20x20](https://avatars3.githubusercontent.com/u/16748619?s=60&v=4) ](https://github.com/DL6ER)

[DL6ER](https://github.com/pi-hole/FTL/commits?author=DL6ER) committed yesterday

Signed-off-by: DL6ER <dl6er@dl6er.de>

[Add more debugging output.](https://github.com/pi-hole/FTL/commit/7fc340c16affbc70de72146792f4310bf12bf5e0)

[ ![@DL6ER|20x20](https://avatars3.githubusercontent.com/u/16748619?s=60&v=4) ](https://github.com/DL6ER)

[DL6ER](https://github.com/pi-hole/FTL/commits?author=DL6ER) committed yesterday

[Remove static instruction from query_blocked() as it is called from p…](https://github.com/pi-hole/FTL/commit/25a403a7c4ddb45e70c062795e602a1b8296fa5b)

[ ![@DL6ER|20x20](https://avatars3.githubusercontent.com/u/16748619?s=60&v=4) ](https://github.com/DL6ER)

[DL6ER](https://github.com/pi-hole/FTL/commits?author=DL6ER) committed yesterday

[Improve performance significantly. We store if (and if: why) a certai…](https://github.com/pi-hole/FTL/commit/faac7005d173dd95bf654addfedbcfd39ee0a421)

[ ![@DL6ER|20x20](https://avatars3.githubusercontent.com/u/16748619?s=60&v=4) ](https://github.com/DL6ER)

[DL6ER](https://github.com/pi-hole/FTL/commits?author=DL6ER) committed yesterday

jpgpi250 · November 24, 2019, 10:45am

You're assumptions are correct. If it isn't "BAD", it shouldn't be in a blocklist. It's always possible to whitelist it.

sjhgvr · November 25, 2019, 9:07am

Could not agree more.
Also, is that toggle already in uBlock Origin? Can't seem to find it. Thanks

DL6ER · November 25, 2019, 1:10pm

I implemented CNAME during-the-path blocking for branch new/internal-blocking. This is the branch allowing also per-client blocking rules.

github.com/pi-hole/FTL

Per-client blocking rules, intermediate CNAME path blocking and some more

pi-hole:development ← pi-hole:new/internal-blocking

opened 04:02PM - 18 Sep 19 UTC

DL6ER

+1355 -598

**By submitting this pull request, I confirm the following:** - [X] I have re…ad and understood the [contributors guide](https://github.com/pi-hole/pi-hole/blob/master/CONTRIBUTING.md). - [X] I have checked that [another pull request](https://github.com/pi-hole/FTL/pulls) for this purpose does not exist. - [X] I have considered, and confirmed that this submission will be valuable to others. - [X] I accept that this submission may not be used, and the pull request closed at the will of the maintainer. - [X] I give this submission freely, and claim no ownership to its content. **How familiar are you with the codebase?:** ## 10 --- This pull request proposes extensive changes in the way FTL decides how and when to block queries. One result of our work is that we were able to **reduce overall memory requirements of the `pihole-FTL` process significantly** allowing for the size blocking lists solely being limited by the SD card and not anymore by RAM size. Another side effect is that our improved implementation now also allows for **client-dependent blocking** settings which became the tag of this pull request. Per-client blocking will allow you to independently tag - regex blacklist entries - exact blacklist entries - regex whitelist entries - exact whitelist entries, but also - entire adlists to groups and assign groups (also multiple) to individual clients. This allows you to, e.g., set up a group of blacklisted domains you only want to see blocked on Apple devices whereas you want to set up some whitelisted domains you need for properly working Windows devices. The same with adlists: You can specify that some of your adlists (maybe Windows telemetry blocking) to be used by a specific group of your clients in which you put your Windows computers whereas you don't add these adlists for Linux machines. Groups can also be used to enable or disable many domains as once, there is no need to remove entries from lists to disable them. In addition, clients can now be associated with these groups but will, obviously, only use enabled lists. Untagged (= no groups associated to them) clients are the default and use *all* enabled lists. In contrast, groups also provide an elegant way to disable (or reduce) blocking for individual clients. Just assign them to an "empty" group, that is a group not connected to any adlist or blacklist entry. Besides all the great advantages our new implementation brings, we do not want to hide the downsides. It was always of great importance for us to have Pi-hole being able to run on even the lowest-end devices such as the Raspberry Pi version 1. With our new implementation - and all the new logic that enables us to do much more now - performance restrictions seem to suggest that we slowly start to deviate from the lowest-end devices. Pi-hole will still run on such devices, however, many clients with complex per-client configurations or huge blocking lists or many regex filters (or any combination of these) will cause Pi-hole to maybe require a more beefy device to still meet your latency expectations. Having said all this, note that if you run Pi-hole in a normal household network (< 10 clients), use standard lists and no or only a few regex filters, there should be nothing to worry about for you. **Update 1** We were able to resolve the majority of the feared performance penalty with a clever vector implementation memorizing whether a specific client has queried a specific domain already before and what the result was. **Update 2** The traditional way to reload the information after changing the lists was to send a reload command (`SIGHUP`) to FTL. This had the (necessary) side-effect of clearing the entire DNS cache. ~~With this new version of FTL, the user can send `SIGUSR2` to only clear the domain-blocking cache.~~ The DNS cache itself stays intact and is **not flushed**. **Update 3** The signal for flushing only the domain-blocking cache (and reload the regex lists) has been changes to the unused `SIGRT0`. Use, e.g., `sudo kill -SIGRTMIN $(pidof pihole-FTL)`.

There are two points worth mentioning:

Blocked replies are currently replied to with NODATA instead of the configured blocking mode (technical limitation, which does not yet hurt at this point).
More tests are outstanding.

One of the tests I did:

Removed f7ds.liberation.fr from gravity
Added atc.eulerian.net to my exact blacklist

Result:

$ dig f7ds.liberation.fr

; <<>> DiG 9.10.3-P4-Debian <<>> f7ds.liberation.fr
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42224
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;f7ds.liberation.fr.            IN      A

;; Query time: 19 msec
;; SERVER: ::1#53(::1)
;; WHEN: Mon Nov 25 12:59:54 CET 2019
;; MSG SIZE  rcvd: 47

DL6ER · November 26, 2019, 9:59am

A post was split to a new topic: Issue with Gravity (development branch)

9mXg37 · November 26, 2019, 7:07pm

Thank you guys for working on this topic so hard!

In the meantime, it seems there is another tracking company actively suggesting the CNAME method:

Measure accurately with first-party tracking

Gain at least 20% more data compared to other players in the market. With Ingenious, you bypass ad-blocking technology and collect accurate data for accurate analysis.

According to this Github comment: Address 1st-party tracker blocking · Issue #780 · uBlockOrigin/uBlock-issues · GitHub

Looks like they recommend their customers to use marketing.net.*.

Example:
Non-authoritative answer:
marketing.net.brillen.de anonical name = tr-brillen-de.affex.org.
tr-brillen-de.affex.org canonical name = lb1.affex.org.
Name: lb1.affex.org
Address: 35.187.117.15

Non-authoritative answer:
marketing.net.home24.de canonical name = tr-home24-de.affex.org.
tr-home24-de.affex.org canonical name = lb1.affex.org.
Name: lb1.affex.org
Address: 35.187.117.15

I hope it might help you to get them blocked as well.

DL6ER · November 26, 2019, 9:02pm

If

is on gravity or your personal block list, it should(TM) already work.

jfb · November 26, 2019, 9:39pm

If you block all domains containing or starting with "marketing.net" using regex, this fixes this specific problem.

DanSchaper · November 26, 2019, 10:04pm

I'm really trying to follow the analogies but I can't relate this to DNS in any way.

PromoFaux · November 26, 2019, 10:27pm

Friendly reminder to all participants in this thread. Please lets keep the discussion to specifically discussing this feature request in a technical manner. No sales pitches need to be made, we're already working on it.

No need for analogies or hyperbole. I'd really rather not have to lock the continued discussion of this topic down to developers only, but if we get much more noise in the thread then it will be difficult for newer readers to follow along with.

This has been a friendly mod announcement. Lets keep things on topic. Thanks!

anon55913113 · November 27, 2019, 3:40pm

I have now new/internal-blocking active on my backup Pi-hole so I don’t have to revert every time.

Removed f7ds.liberation.fr from gravity:

sqlite3 /etc/pihole/gravity.db "DELETE FROM gravity WHERE domain='f7ds.liberation.fr';"

atc.eulerian.net is added to exact blacklist.

pihole restartdns to be sure and then ran dig.

I have a NOERROR instead of the expected NODATA and the query time is 12ms. For local resolving is less than 2 ms and 12ms is also not a external resolve which is above 40ms.

; <<>> DiG 9.11.5-P4-5.1-Raspbian <<>> f7ds.liberation.fr
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35367
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;f7ds.liberation.fr.            IN      A

;; Query time: 12 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Nov 27 16:47:03 CET 2019
;; MSG SIZE  rcvd: 47

Taking atc.eulerian.net from the exact list then it is not blocked anymore:

; <<>> DiG 9.11.5-P4-5.1-Raspbian <<>> f7ds.liberation.fr
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10563
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;f7ds.liberation.fr.            IN      A

;; ANSWER SECTION:
f7ds.liberation.fr.     1710    IN      CNAME   liberation.eulerian.net.
liberation.eulerian.net. 5310   IN      CNAME   atc.eulerian.net.
atc.eulerian.net.       5310    IN      A       109.232.197.179

;; Query time: 10 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Nov 27 16:50:19 CET 2019
;; MSG SIZE  rcvd: 118

Those 12 ms is not a problem and the CNAME is blocked and so success!

In my earlier version of this posting I thought that new/internal-blocking was merged into FTL development but it seems to be the other way around.

DL6ER · November 27, 2019, 4:05pm

The delay (likely) comes from that we don't inject the CNAME into the DNS cache when the path is blocked half-way through. Hence, the CNAME path needs to be re-preocessed every time. I might be able to come up with an improved implementation but all things DNS cache are rather tricky to get right...

@anon55913113 Can you check in your pihole.log if this query gets forwarded at all or if the 12msec are truely local processing only? If so, how many domains do you have in gravity, and the black- and whitelists?

No, what you see is what is commonly called NODATA. This the response code NOERROR plus an empty response for the requested record (A in your example). The only other negative reply could be NXDOMAIN. This would show up with response code NXDOMAIN and no response for the requested record.
There is no response code NODATA on its own.

DL6ER · November 27, 2019, 10:37pm

A post was split to a new topic: Gravity database issue (branch new/internal-blocking)

DL6ER · December 11, 2019, 9:29pm

The per-client blocking feature includes intermediate CNAME path blocking.
We have been going back and forth on this and came to the conclusion that it is worth delaying the arrival of Pi-hole v5.0 yet more to include both features already here.

Stay tuned!