Allow to prioritize RegEx over domainlist

[jpgpi250]

@Anova3

if you tell me there's a way to accomplish this by putting some lines in the dnsmasq conf files, and whitelisting the domains in FTLDNS

@Anova3

^scribe\.logs\.roku\.com$;reply=refused
^api2\.branch\.io$;reply=192.168.192.168
^cws\.conviva\.com$;reply=nxdomain

exact whitelist:

scribe.logs.roku.com
api2.branch.io
cws.conviva.com

in /etc/dnnsmasq.d/change_reply.conf

address=/api2.branch.io/192.168.192.168
server=/cws.conviva.com/

unfortunately, there doesn't appear to be a dnsmasq option to reply refused, however, if you also use unbound, they have a solution:
in /etc/unbound/unbound.conf.d/change_reply.conf

local-zone: "scribe.logs.roku.com." always_refuse

tested with latest pihole-FTL (v5.15) and latest unbound (v1.16.1), replies as indicated in your post.

edit

@Anova3

One question about this method: do domains whitelisted by FTLDNS and blocked/redirected by dnsmasq still show up in the logs as blocked? or are they shown as allowed?

/edit

[DL6ER]

I checked the code and it is as I feared: Even when it looks simple from the outside, it is actually very complicated to extend the reply extension to white regex. This would require a massive rewrite of a lot of code. The reason being that FTL is "just" sitting on-top of dnsmasq as I said above. DNS queries for whitelisted domains are simply returned immediately to dnsmasq for further processing - we cannot enforce any reply type here. On the other hand, blacklisted queries are handled by FTL itself, we never let dnsmasq reply to these itself. Hence, we can craft any reply the user wants.

If we were to allow modifying the type of reply for whitelisted domains, we'd have to treat them as blocked instead which would defeat many of the current optimizations trying to reply to whitelisted domains as fast as possible (by returning the query to dnsmasq as early as possible). We'd basically create another type like a black-whitelist (maybe that's a "graylist").

Going back and forth about this for two days now, I basically see two possibilities:

1. Use the solution suggested by @jpgpi250 which works already with all recent versions of Pi-hole without any need for further modifications.
2. We add an option to move checking the black regex before the adlists. This will have a clear performance impact (more notable the more and more complex regex you have) so we will have to stress this.

I still don't really like the idea of adding the new option but maybe I can be convinced if this is really wanted.

[jpgpi250]

@DL6ER

Going back and forth about this for two days now, I basically see two possibilities:

1. Use the solution suggested by @jpgpi250 which works already with all recent versions of Pi-hole without any need for further modifications.
2. We add an option to move checking the black regex before the adlists. This will have a clear performance impact (more notable the more and more complex regex you have) so we will have to stress this.

I still don't really like the idea of adding the new option but maybe I can be convinced if this is really wanted.

just my 2 cents...
since there is a solution to change the reply type, even when a gravity entry exist, i see no need for massive changes in pihole-FTL. The only thing I'm missing, hence the mixed solution (dnsmasq and / or unbound entries), is the ability of dnsmasq to reply refused.

I'm going to (as soon as I have the time) investigate if it is possible to move all "change_reply" configuration entries into unbound, e.g. no dnsmasq conf file, a single (point of administration) unbound configuration file. This also has the benifit of always getting the desired reply, even when pihole-FTL is disabled, see my comment here. To be continued...

edit
an unbound only configuration file ( /etc/unbound/unbound.conf.d/change_reply.conf) to achieve this (single point of administartion), eliminating the need for a dnsmasq configuration file.

local-zone: "scribe.logs.roku.com." always_refuse
local-zone: "cws.conviva.com." always_nxdomain
local-zone: "api2.branch.io." redirect
local-data: "api2.branch.io. A 192.168.192.168"
local-data: "api2.branch.io. AAAA 2a01::93a1"

There are lots of reply options, when using unbound, see here...

/edit

[Anova3]

@DL6ER

We'd basically create another type like a black-whitelist (maybe that's a "graylist").

What if, instead, we just added a reply type option to the normal blacklist? Is that not an option? Is there some reason the reply type change has to piggyback the regex lists? I understand it would mean adding an extra UI element to the list, but hopefully that's not too hard to do.

I'm assuming, of course, that the normal blacklist still happens before the adlists come into the picture.

[jpgpi250]

@DL6ER

Firstly, yes, this looks about right (I changed your itemization into an enumeration):

@jpgpi250

1. Exact Whitelist
2. Regex Whitelist
3. Exact Blacklist
4. Blocklist domains (AKA gravity)
5. Regex Blacklist
6. dnsmasq configuration files (/etc/dnsmasq.d/)

with come comments:

1. There is a "zeroth" step you have to prepend which are the "special domains" (use-application-dns.net and mask.icloud.com + mask-h2.icloud.com). But their behavior can be changes easily via config options.

The mind is a strange thing, this keeps haunting me. The order 0..6 has been confirmed by you. I wonder where the "deep CNAME inspection" fits in this picture?

[DL6ER]

@jpgpi250

I wonder where the "deep CNAME inspection" fits in this picture?

It doesn't. For every domain, the steps 0...6 are done as needed. The same procedure is done for every domain in a CNAME chain, again, with the performance tweak of allowing a shortcut when a domain is, e.g., whitelisted early in the chain.