RegEx engine improvements

something wrong???

I installed the latest pihole-FTL version (pihole checkout ftl new/tre-regex), the client name doesn't appear to be resolved anymore, only the IP is shown...

I haven't touched any code there, maybe you just have to wait a little longer. If they do not appear, check your /var/log/pihole.log if you see according PTR requests with answers.

correct, problem auto corrected

sorry...

I entered the blacklist regex

^abc$;querytype=a;invert

I expected a correct answer for the A record, no answer for AAAA (or anthing else, however, dig results (what am I missing?):

pi@raspberrypi:~ $ dig A abc.com

; <<>> DiG 9.11.5-P4-5.1+deb10u1-Raspbian <<>> A abc.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57238
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;abc.com.                       IN      A

;; ANSWER SECTION:
abc.com.                2       IN      A       0.0.0.0

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Jul 07 23:40:09 CEST 2020
;; MSG SIZE  rcvd: 41

pi@raspberrypi:~ $ dig AAAA abc.com

; <<>> DiG 9.11.5-P4-5.1+deb10u1-Raspbian <<>> AAAA abc.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12030
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;abc.com.                       IN      AAAA

;; AUTHORITY SECTION:
abc.com.                873     IN      SOA     ns-318.awsdns-39.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Jul 07 23:40:13 CEST 2020
;; MSG SIZE  rcvd: 114

If I disable the regex entry, I get all correct answers.

Sorry for waisting your time (you wanted feedback)...

This is awesome, can we have a subcategory for Feature Requests only about additional Regex features like this one? :slight_smile:

Your regex says

however your test is

^abc$ does not match abc.com.

The invert makes it match, however.
So the result you see is expected:

  • Block all A queries which match ^abc$ + invert = don't block A queries which match ^abc$ (but everything else because this is not-matching without invert!)
  • Don't do anything for any other type (you requested the regex to be valid only for A queries)

What the?...Sure he wants feedback. Even misunderstandings are important IMO as seeing users having issues understanding things may help writing the documentation in the end.

I assumed the invert option only affected the querytype, but after doing some more tests, I must conclude it works as follows (correct me if I'm wrong).

regex from this topic

".*;querytype=A;invert
  • pihole-FTL looks at the result of pihole-FTL regex-test "google.be" ".*;querytype=A", result: .*;querytype=A matches
  • pihole-FTL than inverts the result.

There appears to be no way to achieve what I wanted (hoped for), apply all rules (gravity, regex, ...) but only allow A queries for a specific device (with a single regex).

I totally misinterpreted the invert function, my mistake...

I might misunderstood what you want, but blocking all A queries for all except one device should be possible by:

  1. add .*;querytype=A to group default as blacklist regex
  2. add .*;querytype=A to group "Allow A" as whitelist regex
  3. assign specific client to group "Allow A"

No, that's NOT what I want.

for some devices, I only want to allow A queries, block all others query types (not sure if the devices will still work correctly, need to be able to setup the rules, in order to test)

  • device is in the default group
  • block, if gravity entry or regex match for default group
  • block if query != A (device also assigned to allowOnlyAqueries group)

It really is a low priority (test). If it isn't possible, so be it...

This is what he said:

but I can see how this can be confusing a bit.

This does not seem possible with a single regex. Is there really a "business case" for this? If so, I they can surely add this as well, otherwise the advise of @yubiuser seems to indeed do what you are looking for.

Blacklist .*, and whitelist .*;querytype=A ?

whitelist always wins, as far as I know, whitelist entries are processed first by pihole-FTL.
.*;querytype=A would almost always result in successful resolution of the DNS query (All A queries are successful).

That is my assumption (test case), I think some devices can work with only results for type A queries, but I still want them to follow the general (default group) rules, e.g. for example ssl.google-analytics.com should remain inaccessible for the device.

OK now I got it.... tricky case

Use a negation for the query type, like:

.*;querytype=!A

I just added this.

appears to be working, test (time) will tell if the device(s) continue to work without problems...

Thanks for testing. Just keep in mind that this is about the new regex feature not about experience exchange about how devices behave under this or that DNS protocol surgery. :wink:

quick question

Do the branches new/tre-regex and new/cname_inspection_logging contain all the functionality that pihole v5.1.1 offers?

I installed pihole v5.1.1, but now want to continue testing .*;querytype=!A
So far, it has been tested successfully on chromecast, ps4 and a windows 7 computer that can only access other local devices (no internet connections allowed, blocked on firewall). I'm expecting to have to whitelist PTR queries, some connections on the win 7 to other local devices fail at the first try.

Short answer: "no"

Longer answer:

new/tre-regex compared with release/v5.1

new/cname_inspection_logging compared with release/v5.1

FTL 5.1.1 has not been released (yet), only the core is at 5.1.1

This summarizes everything. FTL v5.1.1 will come sooner than later (maybe even today, maybe Monday, who knows) and once it is out, I will merge everything into the various feature branches floating around. And, finally, they will be marked as ready for merging into development a few days later. Even if they contain some subtle bugs we haven't seen so far, it is more easy to have everything in one place and fix it there. We need to test their (possible) interaction anyway at some point.

Thanks, I'll resume testing, once the new dev branch is up to date, which will allow me to test both new/tre-regex and new/cname_inspection_logging

give us (all testers) a signal when this is done please.

Update:

  • new/tre-regex has been updated to the latest version of the code (release/v5.2)
  • new/cname_inspection_logging is already included in here through the merged PR