# RegEx engine improvements
**URL:** https://discourse.pi-hole.net/t/regex-engine-improvements/34751
**Category:** General
**Tags:** regex, v5-2
**Created:** [June 22, 2020, 7:44pm UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751 "2020-06-22T19:44:35Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@Coro](https://discourse.pi-hole.net/u/Coro)
#### Post date: [June 30, 2020, 8:48am UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/30 "2020-06-30T08:48:42Z")
So when can you release this? 🙂
(Yes, I'm just kidding)
---
### Author:  [@DL6ER](https://discourse.pi-hole.net/u/DL6ER)
#### Post date: [July 6, 2020, 8:23pm UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/31 "2020-07-06T20:23:17Z")
New feature: Apply regex only to a specific query type.
Example:
```auto
abc;querytype=AAAA
```
will block
```auto
dig AAAA abc
```
but not
```auto
dig A abc
```
This is still experimental. Tests would be appreciated.
---
### Author:  [@jpgpi250](https://discourse.pi-hole.net/u/jpgpi250)
#### Post date: [July 6, 2020, 9:10pm UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/32 "2020-07-06T21:10:04Z")
> [@DL6ER](#):
>
> This is still experimental. Tests would be appreciated.
installed FTL (pihole checkout ftl new/tre-regex).
works apparently A and AAAA...
- **could you give an example of why/when you want to do this (would be useful)?**
- doesn't appear to work for other querytypes (NS)?
- web interface turns (example) (.|^)google.com$ **;querytype=A** into (.|^)google.com$ **;querytype=a** (querytype is always saved in lowercase)
---
### Author:  [@DL6ER](https://discourse.pi-hole.net/u/DL6ER)
#### Post date: [July 6, 2020, 9:24pm UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/33 "2020-07-06T21:24:00Z")
> [@jpgpi250](#):
>
> could you give an example of why/when you want to do this (would be useful)?
We have seen requests for blocking specific query types. This seems to be the simplest realization. I do not expect extensive usage. You typically want one for all.
> [@jpgpi250](#):
>
> doesn't appear to work for other querytypes (NS)?
True, support for further query types has just been added in
[https://github.com/pi-hole/FTL/pull/819](https://github.com/pi-hole/FTL/pull/819)
This code has not yet reached the regex branch.
> [@jpgpi250](#):
>
> web interface turns (example) (.|^)google.com$ **;querytype=A** into (.|^)google.com$ **;querytype=a** (querytype is always saved in lowercase)
Yes. The query types are intentionally recognized case-insensitive to compensate for this.
---
### Author:  [@jpgpi250](https://discourse.pi-hole.net/u/jpgpi250)
#### Post date: [July 7, 2020, 7:16am UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/34 "2020-07-07T07:16:23Z")
> [@DL6ER](#):
>
> We have seen requests for blocking specific query types. This seems to be the simplest realization. I do not expect extensive usage.
I assume you refer to [this](https://discourse.pi-hole.net/t/option-to-block-not-forward-all-aaaa-queries/34837) topic (blocking specific query types).
I have been thinking about this, and found a feature is missing in pihole-FTL, to turn this into a valid business case, I'll try to explain.
I'm already using the database schema, that allows duplicate entries in the domainlist table, so entering an identical whitelist/blacklist is possible. This works great if you want to block something for all clients (default group), but allow access for some clients (example used in earlier conversations: allowfacebook group)
This method **cannot be used** if a regex like **.\*; querytype=AAAA** is used, because it would result in allowing **all** AAAA queries for certain clients, when using it as a whitelist regex (whitelist always wins). In order to use the above regex, it needs to be used as a blacklist regex entry, targeting specific clients.
Now comes the dilemma. If I want to apply this regex to all but some clients (use it as a blacklist regex), I need to create a group with all the clients, except the ones I want to allow making AAAA queries. This list (can be) very large, and probably will not be effective (new clients aren't member of this group)
Most firewalls have a solution for this dilemma, simply specify the clients (IPs) you want to be unaffected by the rule, and **invert the selection**. The result is all clients (IPs) except the ones listed. It looks like this:
and the result is this:
for pihole, this would mean you assign a limited number of clients (IP's) to a group, **invert the selection** , thus effectively targeting all clients, except the ones listed.
The above regex example would than target the **! AllowAAAAqueries** group, making it a lot more effective in an environment where new clients come and go on a regular bases.
**Something to consider** , while your making all these great changes to pihole-FTL?
---
### Author:  [@yubiuser](https://discourse.pi-hole.net/u/yubiuser)
#### Post date: [July 7, 2020, 8:24am UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/36 "2020-07-07T08:24:21Z")
Immediately started working, will report anomalies
---
### Author:  [@DL6ER](https://discourse.pi-hole.net/u/DL6ER)
#### Post date: [July 7, 2020, 5:18pm UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/37 "2020-07-07T17:18:25Z")
> [@jpgpi250](#):
>
> This method **cannot be used** if a regex like **.\*; querytype=AAAA** is used, because it would result in allowing **all** AAAA queries for certain clients, when using it as a whitelist regex (whitelist always wins). In order to use the above regex, it needs to be used as a blacklist regex entry, targeting specific clients.
Blocking `AAAA` for all clients and allowing it only for some seems a legit use case (at least theoretically).
> [@jpgpi250](#):
>
> If I want to apply this regex to all but some clients (use it as a blacklist regex), I need to create a group with all the clients, except the ones I want to allow making AAAA queries. This list (can be) very large, and probably will not be effective (new clients aren't member of this group)
You should do the inverse: Create a group with the clients that should not match and add them in there. Then add the new regex only to `Default` which covers also the new clients.
> [@jpgpi250](#):
>
> The above regex example would than target the **! AllowAAAAqueries** group, making it a lot more effective in an environment where new clients come and go on a regular bases.
Even when I tend to disagree on your conclusion above, I do agree that is useful to be able to invert a regular expression _altogether_. Hence, I added the new keyword `;invert`
For instance,
```auto
^abc$;querytype=AAAA;invert
```
will **not** block `abc` with type `AAAA` (but everything else) for the clients attached to it.
This has the same effect as inverting the client selection, however, it is even somewhat more powerful.
---
### Author:  [@jpgpi250](https://discourse.pi-hole.net/u/jpgpi250)
#### Post date: [July 7, 2020, 9:27pm UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/38 "2020-07-07T21:27:02Z")
something wrong???
I installed the latest pihole-FTL version (pihole checkout ftl new/tre-regex), the client name doesn't appear to be resolved anymore, only the IP is shown...
---
### Author:  [@DL6ER](https://discourse.pi-hole.net/u/DL6ER)
#### Post date: [July 7, 2020, 9:31pm UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/39 "2020-07-07T21:31:12Z")
I haven't touched any code there, maybe you just have to wait a little longer. If they do not appear, check your `/var/log/pihole.log` if you see according PTR requests with answers.
---
### Author:  [@jpgpi250](https://discourse.pi-hole.net/u/jpgpi250)
#### Post date: [July 7, 2020, 9:36pm UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/40 "2020-07-07T21:36:30Z")
> [@DL6ER](#):
>
> maybe you just have to wait a little longer
correct, problem auto corrected
sorry...
---
### Author:  [@jpgpi250](https://discourse.pi-hole.net/u/jpgpi250)
#### Post date: [July 7, 2020, 9:47pm UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/41 "2020-07-07T21:47:16Z")
> [@DL6ER](#):
>
> ```auto
> ^abc$;querytype=AAAA;invert
>
> ```
>
> will **not** block `abc` with type `AAAA` (but everything else) for the clients attached to it.
I entered the blacklist regex
```auto
^abc$;querytype=a;invert
```
I expected a correct answer for the A record, no answer for AAAA (or anthing else, however, dig results ( **what am I missing?** ):
```auto
pi@raspberrypi:~ $ dig A abc.com
; <<>> DiG 9.11.5-P4-5.1+deb10u1-Raspbian <<>> A abc.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57238
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;abc.com. IN A
;; ANSWER SECTION:
abc.com. 2 IN A 0.0.0.0
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Jul 07 23:40:09 CEST 2020
;; MSG SIZE rcvd: 41
pi@raspberrypi:~ $ dig AAAA abc.com
; <<>> DiG 9.11.5-P4-5.1+deb10u1-Raspbian <<>> AAAA abc.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12030
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;abc.com. IN AAAA
;; AUTHORITY SECTION:
abc.com. 873 IN SOA ns-318.awsdns-39.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Jul 07 23:40:13 CEST 2020
;; MSG SIZE rcvd: 114
```
If I disable the regex entry, I get all correct answers.
Sorry for waisting your time (you wanted feedback)...
---
### Author:  [@Coro](https://discourse.pi-hole.net/u/Coro)
#### Post date: [July 8, 2020, 5:40am UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/42 "2020-07-08T05:40:00Z")
> [@DL6ER](#):
>
> I do agree that is useful to be able to invert a regular expression _altogether_ . Hence, I added the new keyword `;invert`
This is awesome, can we have a subcategory for Feature Requests only about additional Regex features like this one? 🙂
> [@jpgpi250](#):
>
> I entered the blacklist regex
>
> ```auto
> ^abc$;querytype=a;invert
>
> ```
>
> I expected a correct answer for the A record, no answer for AAAA (or anthing else, however, dig results ( **what am I missing?** ):
Your regex says
> [@jpgpi250](#):
>
> `^abc$`
however your test is
> [@jpgpi250](#):
>
> `dig A abc.com`
`^abc$` does not match `abc.com`.
The invert makes it match, however.
So the result you see is expected:
- Block all `A` queries which match `^abc$` + invert = don't block `A` queries which match `^abc$` (but everything else because this is not-matching without `invert`!)
- Don't do anything for any other type (you requested the regex to be valid only for `A` queries)
> [@jpgpi250](#):
>
> Sorry for waisting your time (you wanted feedback)...
What the?...Sure he wants feedback. Even misunderstandings are important IMO as seeing users having issues understanding things may help writing the documentation in the end.
---
### Author:  [@jpgpi250](https://discourse.pi-hole.net/u/jpgpi250)
#### Post date: [July 8, 2020, 6:56am UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/43 "2020-07-08T06:56:01Z")
I assumed the `invert` option only affected the `querytype`, but after doing some more tests, I must conclude it works as follows (correct me if I'm wrong).
regex from [this topic](https://discourse.pi-hole.net/t/option-to-block-not-forward-all-aaaa-queries/34837/36)
```auto
".*;querytype=A;invert
```
- pihole-FTL looks at the result of `pihole-FTL regex-test "google.be" ".*;querytype=A"`, **result** : `.*;querytype=A matches`
- pihole-FTL than **inverts** the result.
There appears to be no way to achieve what I wanted (hoped for), apply all rules (gravity, regex, ...) but only allow A queries for a specific device (with a single regex).
I totally misinterpreted the `invert` function, my mistake...
---
### Author:  [@yubiuser](https://discourse.pi-hole.net/u/yubiuser)
#### Post date: [July 8, 2020, 7:13am UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/44 "2020-07-08T07:13:06Z")
> [@jpgpi250](#):
>
> There appears to be no way to achieve what I wanted (hoped for), apply all rules (gravity, regex, ...) but only allow A queries for a specific device (with a single regex).
I might misunderstood what you want, but blocking all A queries for all except one device should be possible by:
1. add `.*;querytype=A` to group default as blacklist regex
2. add `.*;querytype=A` to group "Allow A" as whitelist regex
3. assign specific client to group "Allow A"
---
### Author:  [@jpgpi250](https://discourse.pi-hole.net/u/jpgpi250)
#### Post date: [July 8, 2020, 8:13am UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/45 "2020-07-08T08:13:47Z")
No, that's NOT what I want.
for some devices, I only want to allow A queries, block all others query types (not sure if the devices will still work correctly, need to be able to setup the rules, in order to test)
- device is in the default group
- block, if gravity entry or regex match for default group
- block if query != A (device also assigned to allowOnlyAqueries group)
It really is a low priority (test). If it isn't possible, so be it...
---
### Author:  [@Coro](https://discourse.pi-hole.net/u/Coro)
#### Post date: [July 8, 2020, 8:16am UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/46 "2020-07-08T08:16:53Z")
> [@jpgpi250](#):
>
> I assumed the `invert` option only affected the `querytype` , but after doing some more tests, I must conclude it works as follows (correct me if I'm wrong).
This is what he said:
> [@DL6ER](#):
>
> invert a regular expression _altogether_
but I can see how this can be confusing a bit.
> [@jpgpi250](#):
>
> I only want to allow A queries, block all others query types
This does not seem possible with **a single regex**. Is there really a "business case" for this? If so, I they can surely add this as well, otherwise the advise of @yubiuser seems to indeed do what you are looking for.
---
### Author:  [@yubiuser](https://discourse.pi-hole.net/u/yubiuser)
#### Post date: [July 8, 2020, 8:20am UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/47 "2020-07-08T08:20:31Z")
> [@jpgpi250](#):
>
> for some devices, I only want to allow A queries, block all others query types
Blacklist `.*`, and whitelist `.*;querytype=A` ?
---
### Author:  [@jpgpi250](https://discourse.pi-hole.net/u/jpgpi250)
#### Post date: [July 8, 2020, 8:29am UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/48 "2020-07-08T08:29:25Z")
whitelist always wins, as far as I know, whitelist entries are processed first by pihole-FTL.
`.*;querytype=A` would almost always result in successful resolution of the DNS query (All A queries are successful).
That is my assumption (test case), I think some devices can work with only results for type A queries, but I still want them to follow the general (default group) rules, e.g. for example `ssl.google-analytics.com` should remain inaccessible for the device.
---
### Author:  [@yubiuser](https://discourse.pi-hole.net/u/yubiuser)
#### Post date: [July 8, 2020, 8:34am UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/49 "2020-07-08T08:34:34Z")
OK now I got it.... tricky case
---
### Author:  [@DL6ER](https://discourse.pi-hole.net/u/DL6ER)
#### Post date: [July 8, 2020, 7:16pm UTC](https://discourse.pi-hole.net/t/regex-engine-improvements/34751/50 "2020-07-08T19:16:32Z")
> [@jpgpi250](#):
>
> block if query != A (device also assigned to allowOnlyAqueries group)
Use a negation for the query type, like:
```auto
.*;querytype=!A
```
I just added this.
[Previous page](https://discourse.pi-hole.net/t/regex-engine-improvements/34751.md?page=1)
[Next page](https://discourse.pi-hole.net/t/regex-engine-improvements/34751.md?page=3)