Regex not working in latest FTLDNS beta

[sn2411]

This is the result from

dig example.com

; <<>> DiG 9.10.3-P4-Debian <<>> example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6743
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;example.com.                   IN      A

;; ANSWER SECTION:
example.com.            60605   IN      A       93.184.216.34

;; AUTHORITY SECTION:
example.com.            60604   IN      NS      a.iana-servers.net.
example.com.            60604   IN      NS      b.iana-servers.net.

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Jul 30 16:26:39 +08 2018
;; MSG SIZE  rcvd: 104

Here are the results from

tail /var/log/pihole-FTL.log

[2018-07-30 16:26:33.334] Listening on Unix socket
[2018-07-30 16:26:33.335] Compiled 3 Regex filters and 144 whitelisted domains in 0.2 msec (0 errors)
[2018-07-30 16:26:33.335] /etc/pihole/black.list: parsed 1 domains (took 0.0 ms)
[2018-07-30 16:26:34.878] /etc/pihole/gravity.list: parsed 855697 domains (took 1542.0 ms)
[2018-07-30 16:26:35.039] Notice: Increasing queries struct size from 0 to 10000
[2018-07-30 16:26:35.039] Notice: Increasing overTime struct size from 0 to 100
[2018-07-30 16:26:35.039] Notice: Increasing domains struct size from 0 to 1000
[2018-07-30 16:26:35.039] Notice: Increasing clients struct size from 0 to 10
[2018-07-30 16:26:39.276] New forward server: 127.0.0.1 (0/0)
[2018-07-30 16:26:39.276] Notice: Increasing forwarded struct size from 0 to 4

And from

tail /var/log/pihole.log

Jul 30 16:26:33 dnsmasq[7207]: DNS service limited to local subnets
Jul 30 16:26:33 dnsmasq[7207]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify
Jul 30 16:26:33 dnsmasq[7207]: DNSSEC validation enabled
Jul 30 16:26:33 dnsmasq[7207]: warning: failed to change owner of /var/log/pihole.log: Operation not permitted
Jul 30 16:26:33 dnsmasq-dhcp[7207]: DHCP, IP range 192.168.1.1 -- 192.168.1.240, lease time 1d
Jul 30 16:26:33 dnsmasq[7207]: using nameserver 127.0.0.1#5353
Jul 30 16:26:33 dnsmasq[7207]: read /etc/hosts - 5 addresses
Jul 30 16:26:33 dnsmasq[7207]: read /etc/pihole/local.list - 11 addresses
Jul 30 16:26:33 dnsmasq[7207]: read /etc/pihole/black.list - 2 addresses
Jul 30 16:26:34 dnsmasq[7207]: read /etc/pihole/gravity.list - 1711394 addresses

[DL6ER]

Do you have query logging enabled? If not, please do this.

Also, what is the output of

pihole-FTL -v

?

[sn2411]

Strange... the wildcard regex works as expected when privacy level is set to 0, i.e. "Show everything and record everything". Setting privacy level to 1, 2 or 3 breaks wildcard regex.

Query logging does not affect the functionality of wildcard regex.

With PRIVACYLEVEL=0 in /etc/pihole/pihole-FTL.conf, and query logging enabled, here are the results for:

 dig example.com

; <<>> DiG 9.10.3-P4-Debian <<>> example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26846
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.com.                   IN      A

;; ANSWER SECTION:
example.com.            2       IN      A       0.0.0.0

;; Query time: 13 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Jul 30 16:47:48 +08 2018
;; MSG SIZE  rcvd: 56

From

tail /var/log/pihole-FTL.log

[2018-07-30 16:43:14.734]  -> Known forward destinations: 1
[2018-07-30 16:43:14.734] Successfully accessed setupVars.conf
[2018-07-30 16:43:14.737] PID of FTL process: 9046
[2018-07-30 16:43:14.737] Listening on port 4711 for incoming IPv4 telnet connections
[2018-07-30 16:43:14.737] Listening on port 4711 for incoming IPv6 telnet connections
[2018-07-30 16:43:14.737] Listening on Unix socket
[2018-07-30 16:43:14.739] Compiled 3 Regex filters and 144 whitelisted domains in 1.0 msec (0 errors)
[2018-07-30 16:43:14.739] /etc/pihole/black.list: parsed 1 domains (took 0.0 ms)
[2018-07-30 16:43:16.295] /etc/pihole/gravity.list: parsed 855697 domains (took 1556.7 ms)
[2018-07-30 16:47:48.946] DEBUG: Regex in line 2 "((^)|(\.))example\.com$" matches "example.com"

And from

tail /var/log/pihole.log

Jul 30 16:50:01 dnsmasq[9046]: 58 127.0.0.1/34516 cached checkip.dyndns.com is NODATA-IPv6
Jul 30 16:50:02 dnsmasq[9046]: 59 127.0.0.1/55606 query[A] www.duckdns.org from 127.0.0.1
Jul 30 16:50:02 dnsmasq[9046]: 59 127.0.0.1/55606 cached www.duckdns.org is <CNAME>
Jul 30 16:50:02 dnsmasq[9046]: 59 127.0.0.1/55606 cached DuckDNSAppELB-570522007.us-west-2.elb.amazonaws.com is 54.213.176.83
Jul 30 16:50:02 dnsmasq[9046]: 59 127.0.0.1/55606 cached DuckDNSAppELB-570522007.us-west-2.elb.amazonaws.com is 52.89.140.116
Jul 30 16:50:02 dnsmasq[9046]: 60 127.0.0.1/55606 query[AAAA] www.duckdns.org from 127.0.0.1
Jul 30 16:50:02 dnsmasq[9046]: 60 127.0.0.1/55606 cached www.duckdns.org is <CNAME>
Jul 30 16:50:02 dnsmasq[9046]: 60 127.0.0.1/55606 cached DuckDNSAppELB-570522007.us-west-2.elb.amazonaws.com is NODATA-IPv6
Jul 30 16:50:14 dnsmasq[9046]: 61 127.0.0.1/35884 query[A] example.com from 127.0.0.1
Jul 30 16:50:14 dnsmasq[9046]: 61 127.0.0.1/35884 <unknown> example.com is 0.0.0.0

Edit: output from pihole-FTL -v is vDev-8e56b61

[DL6ER]

@sn2411

Setting privacy level to 1, 2 or 3 breaks wildcard regex.

Oh, yes, I remember having seen this before but then forgot about it before I fixed this. For "perfect privacy", we obfuscate the domain early on in FTL's internal processing. This has the drawback that it is already hidden once we arrive at the regex validation. I will make FTL use a domain buffer for regex validation also with privacy level != 0.

Thanks for reporting this!

[DL6ER]

Fix incoming and scheduled to go in before we release Pi-hole v4.0

Fix for regex filters + privacy level > 0

[sn2411]

Good to know! Thanks so much for all the hard work!