Allow to prioritize RegEx over domainlist

[Anova3]

Agreed on all accounts. I can't replace my devices/apps at the moment, and the vendors will not respond to any complaints. Being a technophobe isn't going to save us here.

Back on topic: this feature request would also provide a more flexible and graphical method of implementing the blocks for iCloud's Public Relay and Mozilla Canary.
Normally you have to open the FTLDNS config to insert those blocks. This could do the same thing, but within the web UI, and it would cover a handful of other similar/future use cases like this without having to implement more one-off FTLDNS config settings.

[jfb]

@Anova3

this feature request would also provide a more flexible and graphical method of implementing the blocks for iCloud's Public Relay and Mozilla Canary.

How so? The specific response provided by Pi-hole to those domains is NXDOMAIN, not the default NULL for other blocked domains.

[Anova3]

If we can specify a ;reply=nxdomain for those domains in such a way that it overrides adlists, then isn't that the same thing as what those settings are doing?

Example blacklist entry for mozilla canary:
use-application-dns\.net;reply=nxdomain

[jpgpi250]

@Anova3

If we can specify a ;reply=nxdomain for those domains in such a way that it overrides adlists, then isn't that the same thing as what those settings are doing?

@jfb

The iCloud Private relay blocking is done at the FTL level.

Never tested this, but does pihole-FTL block anything while pihole is disabled (menu or desktop shortcut)?

User example: a page doesn't load, the user hits the desktop shortcut (or any other method) to disable pihole temporarily. The page can now be refreshed and loads as expected. Unfortunately, all other devices on the network (Network-wide Ad Blocking) are also no longer blocked (temporarily) and can possibly pick up the IP addres(ses) for the otherwise blocked domain(s).

To overcome (prevent) this (when using unbound), you can add a configuration file to unbound to provide the desired reply, even if pihole is temporarily disabled. example ( /etc/unbound/unbound.conf.d/blocked.conf)

# https://nlnetlabs.nl/documentation/unbound/unbound.conf/
server:
        local-zone: "use-application-dns.net." always_nxdomain
        local-zone: "mask.icloud.com." always_nxdomain
        local-zone: "mask-h2.icloud.com." always_nxdomain

If you are speciffically targetting (o)DoH, read this unbound manual, explaining how response policy zones work. The (o)DoH rpz file contains the known domains from various sources.

[jfb]

@jpgpi250

Never tested this, but does pihole-FTL block anything while pihole is disabled

No.

[pi@Pi-3B-DEV:~ $ pihole disable
  [i] Disabling blocking
  [✓] Reloading DNS lists
  [✓] Pi-hole Disabled
pi@Pi-3B-DEV:~ $ dig mask.icloud.com

; <<>> DiG 9.16.27-Raspbian <<>> mask.icloud.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13021
;; flags: qr rd ra; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;mask.icloud.com.		IN	A

;; ANSWER SECTION:
mask.icloud.com.	43200	IN	CNAME	mask.apple-dns.net.
mask.apple-dns.net.	3600	IN	A	17.248.139.229
mask.apple-dns.net.	3600	IN	A	17.248.139.231
mask.apple-dns.net.	3600	IN	A	17.248.139.230
mask.apple-dns.net.	3600	IN	A	17.248.139.232
mask.apple-dns.net.	3600	IN	A	17.248.139.228
mask.apple-dns.net.	3600	IN	A	17.248.139.233

;; Query time: 109 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Jun 28 17:47:48 CDT 2022
;; MSG SIZE  rcvd: 172

pi@Pi-3B-DEV:~ $ pihole enable
  [i] Enabling blocking
  [✓] Reloading DNS lists
  [✓] Pi-hole Enabled
pi@Pi-3B-DEV:~ $ dig mask.icloud.com

; <<>> DiG 9.16.27-Raspbian <<>> mask.icloud.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 40088
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;mask.icloud.com.		IN	A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Jun 28 17:48:07 CDT 2022
;; MSG SIZE  rcvd: 44

[DanSchaper]

@jpgpi250

The page can now be refreshed and loads as expected. Unfortunately, all other devices on the network (Network-wide Ad Blocking) are also no longer blocked (temporarily) and can possibly pick up the IP addres(ses) for the otherwise blocked domain(s).

This has been a long time nit for me. We've modified the development code to change the default condition when FTL is offline in an error condition. I remember having a discussion about being able to disable per group instead of for the system as a whole, I don't remember how that chat ended.

@jpgpi250

The (o)DoH rpz file contains the known domains from various sources.

RPZ's are kind of interesting to me. I remember Paul Vixie talking about them a few years ago in relation to Pi-hole...

[jpgpi250]

@DanSchaper

RPZ's are kind of interesting to me.

I've been using them for a while now, there are some interesting sources (other than mine - (o)DoH only), example: urlhaus, updated every 5 minutes.

After I've made a request, the unbound team made a change to their code to allow pihole-FTL to (sort of) indicate a domain was blocked due to an RPZ entry.

A long time ago, DL6ER added code to indicate a domain was blocked by external (status 8 - By upstream server (NXDOMAIN with RA bit unset)). This was originally intended to give a different status for domains, filtered by quad9 (if I remember correctly).
This status value can now be used, if configured in the unbound rpz config, to indicate a domain was blocked by an RPZ entry (unbound, but also possible with bind, knot-resolver, ...), details in my RPZ document.

edit

@jfb

No.

thanks for the test (blocking when pihole is disabled).

A user can add additional entries to the above blocked.conf example to ensure blocking is applied for domains that require blocking, regardles of pihole-FTL disabled/enabled status. Example:

local-zone: "userlocation.googleapis.com." always_null

notice the canary domains where blocked with always_nxdomain, to trigger the desired response (example: firefox browser), but it is possible to use always_null, this to deliver the same response as pihole-FTL (default blocking mode) would provide.

If you're going to use a large number of entries, response policy zones are a better option.
/edit

[DanSchaper]

@jpgpi250

This was originally intended to give a different status for domains, filtered by quad9 (if I remember correctly).
This status value can now be used, if configured in the unbound rpz config, to indicate a domain was blocked by an RPZ entry

Yes, you remember correctly. And nice to know that it's useful in other places as well.

[DL6ER]

Sorry, I have been swamped with other (real life) projects and didn't check back here for quite some time.

Firstly, yes, this looks about right (I changed your itemization into an enumeration):

@jpgpi250

1. Exact Whitelist
2. Regex Whitelist
3. Exact Blacklist
4. Blocklist domains (AKA gravity)
5. Regex Blacklist
6. dnsmasq configuration files (/etc/dnsmasq.d/)

with come comments:

1. There is a "zeroth" step you have to prepend which are the "special domains" (use-application-dns.net and mask.icloud.com + mask-h2.icloud.com). But their behavior can be changes easily via config options.
2. Executing regular expressions is a lot more computationally expensive than doing lookups in our exact domain. While the exact domains are organized in a B-tree (allowing logarithmic lookup times), regular expressions have to be iterated one after another for every domain. Running a single regular expression against a domain can easily be more work than comparing the domain against several millions of exact domains (not because regular expressions are so slow but because looking up in the tree is so blazingly fast).
   Furthermore, FTL tries to stay out of your everyday browsing as much as possible. Hence, we look at whitelisted domains first (this is typically stuff you want to visit) while you usually don't care too much if it takes some extra milliseconds until ads are blocked.
3. You have to make an important distinction in the code-flow between 5. and 6. because this is where we leave FTL code and enter dnsmasq code. When nothing matches in steps 0-5, FTL hands the query to the embedded dnsmasq core as if nothing would have happened. Hence, we have no influence on prioritizing and dnsmasq config files.

---

The second being

@jpgpi250

reply types for whitelist regex

which is something worth considering. However, it also needs some (maybe non-trivial) code changes as whitelist is currently simply pushing though queries as fast as possible to the embedded dnsmasq for further processing. Equipping whitelist with the reply option means we need to add another code path that basically handles these queries as if they were blacklisted (while they are not) so FTL steps in and crafts the DNS response with the requested reply type.

I'll think about how to achieve this, however, it will either be hacky or require some good amount of generalization in the code (basically widely remove the separate code paths for black and white domains). As it will be the latter, it is some work.
Feel free to ping me when you get the impression that I forgot about this.

[DL6ER]

Adding to what I wrote (I just re-read some posts here):

1. Arbitrarily changing priority would be complicated to implement. Allowing to specify that black regex should be lookup up before the adlists which be fairly simple, however, I'm not sure how easy it would be to explain what this config option is doing. When a feature isn't easily explainable, it seems wrong to have it (as there should be a better alternative).
2. We cannot add extensions like ;reply=... to the exact lists as this would completely destroy the extremely favorable fast lookup times.
3. If you whitelist, you can do everything what you want with this domain in a dnsmasq config file right now, so this comment

   @jpgpi250

   You would than be able to specify the desired response in a dnsmasq conf file.

   is directly applicable to whitelisted domains even without any code changes and with all versions of FTL.

[jpgpi250]

@DL6ER

If you whitelist, you can do everything what you want with this domain in a dnsmasq config file right now

@yubiuser: you really need to provide the example for which you are trying to change the response.

If I understand DL6ER correctly, this would mean the reply a normal dig for an exact whitelisted domain can be changed with a dnsmasq.conf entry.

Just tested this:

• whitelist entry for assets.adobedtm.com
• dig assets.adobedtm.com -> 2.21.17.6
• added dnsmasq conf file:

address=/assets.adobedtm.com/192.168.2.74

• dig assets.adobedtm.com -> 192.168.2.74

Thus, confirmed, this clearly eliminates (partially??) the need for whitelist regex entries with the ;reply= option

Thank you DL6ER, for your valuable insight (as always)...

[jfb]

@jpgpi250

@yubiuser: you really need to provide the example for which you are trying to change the response.

Yubi is not the originator of the thread. He made the feature request in response to a separate post as linked at the top of this feature request. The author of the original post is the person who can provide the matching examples.

[Anova3]

I'm the originator of the help ticket.

Some examples of things I'm using, since you asked for them:
^scribe\.logs\.roku\.com$;reply=refused
^api2\.branch\.io$;reply=192.168.192.168
^cws\.conviva\.com$;reply=nxdomain

It's all well and good that folks want to be thorough with all of this, but I feel like we've over-complicated the problem a bit.

I have specific domains (not some kind of wildcard range of domains) that I want to give specific reply types to. The devices calling them get spammy when supplied with the default nodata reply type.

The catch is, these domains are already in the adlists I use (and not just one or two) and they deserve to stay there for folks who don't know enough yet to pursue alternative blocking modes like I'm doing now to optimize my network.

Regex blacklists have this feature, but they're overridden if the domain in question is in an adlist. That's the only reason I brought them into this.

I don't need full regex blocking. I just need the ability to specify a reply type for specific domains, and I need that to override what's in the adlists.
We've suggested a lot of options for implementing this feature, and regex lists have been a recurring thing mentioned, but regex itself is not needed.

So, if you tell me there's a way to accomplish this by putting some lines in the dnsmasq conf files, and whitelisting the domains in FTLDNS, then that's great! I don't know how to do that yet, but I can figure it out, I'm sure.
One question about this method: do domains whitelisted by FTLDNS and blocked/redirected by dnsmasq still show up in the logs as blocked? or are they shown as allowed?

Alternatively, is there a chance we could have a way to write custom "zeroth step" rules? In other words, is there a way we can specify our own "special domains"? This sounds close to what I'm trying to do. I'm guessing there's some reason we can't do this, but I figured it didn't hurt to ask.

[jpgpi250]

@Anova3

if you tell me there's a way to accomplish this by putting some lines in the dnsmasq conf files, and whitelisting the domains in FTLDNS

@Anova3

^scribe\.logs\.roku\.com$;reply=refused
^api2\.branch\.io$;reply=192.168.192.168
^cws\.conviva\.com$;reply=nxdomain

exact whitelist:

scribe.logs.roku.com
api2.branch.io
cws.conviva.com

in /etc/dnnsmasq.d/change_reply.conf

address=/api2.branch.io/192.168.192.168
server=/cws.conviva.com/

unfortunately, there doesn't appear to be a dnsmasq option to reply refused, however, if you also use unbound, they have a solution:
in /etc/unbound/unbound.conf.d/change_reply.conf

local-zone: "scribe.logs.roku.com." always_refuse

tested with latest pihole-FTL (v5.15) and latest unbound (v1.16.1), replies as indicated in your post.

edit

@Anova3

One question about this method: do domains whitelisted by FTLDNS and blocked/redirected by dnsmasq still show up in the logs as blocked? or are they shown as allowed?

/edit

[DL6ER]

I checked the code and it is as I feared: Even when it looks simple from the outside, it is actually very complicated to extend the reply extension to white regex. This would require a massive rewrite of a lot of code. The reason being that FTL is "just" sitting on-top of dnsmasq as I said above. DNS queries for whitelisted domains are simply returned immediately to dnsmasq for further processing - we cannot enforce any reply type here. On the other hand, blacklisted queries are handled by FTL itself, we never let dnsmasq reply to these itself. Hence, we can craft any reply the user wants.

If we were to allow modifying the type of reply for whitelisted domains, we'd have to treat them as blocked instead which would defeat many of the current optimizations trying to reply to whitelisted domains as fast as possible (by returning the query to dnsmasq as early as possible). We'd basically create another type like a black-whitelist (maybe that's a "graylist").

Going back and forth about this for two days now, I basically see two possibilities:

1. Use the solution suggested by @jpgpi250 which works already with all recent versions of Pi-hole without any need for further modifications.
2. We add an option to move checking the black regex before the adlists. This will have a clear performance impact (more notable the more and more complex regex you have) so we will have to stress this.

I still don't really like the idea of adding the new option but maybe I can be convinced if this is really wanted.

[jpgpi250]

@DL6ER

Going back and forth about this for two days now, I basically see two possibilities:

1. Use the solution suggested by @jpgpi250 which works already with all recent versions of Pi-hole without any need for further modifications.
2. We add an option to move checking the black regex before the adlists. This will have a clear performance impact (more notable the more and more complex regex you have) so we will have to stress this.

I still don't really like the idea of adding the new option but maybe I can be convinced if this is really wanted.

just my 2 cents...
since there is a solution to change the reply type, even when a gravity entry exist, i see no need for massive changes in pihole-FTL. The only thing I'm missing, hence the mixed solution (dnsmasq and / or unbound entries), is the ability of dnsmasq to reply refused.

I'm going to (as soon as I have the time) investigate if it is possible to move all "change_reply" configuration entries into unbound, e.g. no dnsmasq conf file, a single (point of administration) unbound configuration file. This also has the benifit of always getting the desired reply, even when pihole-FTL is disabled, see my comment here. To be continued...

edit
an unbound only configuration file ( /etc/unbound/unbound.conf.d/change_reply.conf) to achieve this (single point of administartion), eliminating the need for a dnsmasq configuration file.

local-zone: "scribe.logs.roku.com." always_refuse
local-zone: "cws.conviva.com." always_nxdomain
local-zone: "api2.branch.io." redirect
local-data: "api2.branch.io. A 192.168.192.168"
local-data: "api2.branch.io. AAAA 2a01::93a1"

There are lots of reply options, when using unbound, see here...

/edit

[Anova3]

@DL6ER

We'd basically create another type like a black-whitelist (maybe that's a "graylist").

What if, instead, we just added a reply type option to the normal blacklist? Is that not an option? Is there some reason the reply type change has to piggyback the regex lists? I understand it would mean adding an extra UI element to the list, but hopefully that's not too hard to do.

I'm assuming, of course, that the normal blacklist still happens before the adlists come into the picture.

[jpgpi250]

@DL6ER

Firstly, yes, this looks about right (I changed your itemization into an enumeration):

@jpgpi250

1. Exact Whitelist
2. Regex Whitelist
3. Exact Blacklist
4. Blocklist domains (AKA gravity)
5. Regex Blacklist
6. dnsmasq configuration files (/etc/dnsmasq.d/)

with come comments:

1. There is a "zeroth" step you have to prepend which are the "special domains" (use-application-dns.net and mask.icloud.com + mask-h2.icloud.com). But their behavior can be changes easily via config options.

The mind is a strange thing, this keeps haunting me. The order 0..6 has been confirmed by you. I wonder where the "deep CNAME inspection" fits in this picture?

[DL6ER]

@jpgpi250

I wonder where the "deep CNAME inspection" fits in this picture?

It doesn't. For every domain, the steps 0...6 are done as needed. The same procedure is done for every domain in a CNAME chain, again, with the performance tweak of allowing a shortcut when a domain is, e.g., whitelisted early in the chain.