Faulty regex consumes all memory and causes issues with pihole -q

Okay, try going through the adlists then.

This is the line for the PHP Fatal error:

Tried with the 6 default adlists: failed

nanopi@nanopi:~$ pihole -g
[sudo] password for nanopi: 
  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful
  [i] Received 51220 domains

  [i] Target: https://mirror1.malwaredomains.com/files/justdomains
  [✓] Status: No changes detected
  [i] Received 26860 domains

  [i] Target: http://sysctl.org/cameleon/hosts
  [✓] Status: No changes detected
  [i] Received 20567 domains

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
  [✓] Status: No changes detected
  [i] Received 34 domains

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
  [✓] Status: No changes detected
  [i] Received 2701 domains

  [i] Target: https://hosts-file.net/ad_servers.txt
  [✓] Status: No changes detected
  [i] Received 45736 domains

  [✓] Storing downloaded domains in new gravity database
  [✓] Building tree
  [✓] Swapping databases
  [i] Number of gravity domains: 147118 (125108 unique domains)
  [i] Number of exact blacklisted domains: 6
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 3
  [i] Number of regex whitelist filters: 0
  [✓] Flushing DNS cache
  [✓] Cleaning up stray matter

  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled
nanopi@nanopi:~$ pihole -q gstatic.com
/opt/pihole/query.sh: line 37: 14510 Killed                  awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' <(echo "${lists}") <(echo "${domain}") 2> /dev/null
 Match found in https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts:
   csi.gstatic.com 
 Match found in https://hosts-file.net/ad_servers.txt:
   csi.gstatic.com 
   p2-aahhyknavsj2m-wtnlrzkba6lht33q-if-v6exp3-v4.metric.gstatic.com 
   p2-f6rp6piuxns4u-uzq4vp76bu3w2tso-if-v6exp3-v4.metric.gstatic.com 
   p2-n3zurhre4jjvk-can5rb2f2a4urcxh-if-v6exp3-v4.metric.gstatic.com 
   p4-ajvwyt5lpjazy-us7r2dzqcjsqh7pt-164149-i1-v6exp3-v4.metric.gstatic.com 
   p4-ajvwyt5lpjazy-us7r2dzqcjsqh7pt-164149-i2-v6exp3-ds.metric.gstatic.com 
   p4-ajvwyt5lpjazy-us7r2dzqcjsqh7pt-164149-s1-v6exp3-v4.metric.gstatic.com 
   p4-ajvwyt5lpjazy-us7r2dzqcjsqh7pt-if-v6exp3-v4.metric.gstatic.com 
   p5-lj5aujgj7jl7w-r2pmxqvndsgx2im2-931517-i1-v6exp3-v4.metric.gstatic.com 
   p5-lj5aujgj7jl7w-r2pmxqvndsgx2im2-931517-i2-v6exp3-ds.metric.gstatic.com 
   s6.netlogstatic.com 
   v6exp3-ds.metric.gstatic.com 
   v6exp3-v4.metric.gstatic.com 

Tried with only one default adlist: failed

nanopi@nanopi:~$ pihole -g
  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful
  [i] Received 51220 domains

  [✓] Storing downloaded domains in new gravity database
  [✓] Building tree
  [✓] Swapping databases
  [i] Number of gravity domains: 51220 (51220 unique domains)
  [i] Number of exact blacklisted domains: 6
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 3
  [i] Number of regex whitelist filters: 0
  [✓] Flushing DNS cache
  [✓] Cleaning up stray matter

  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled
nanopi@nanopi:~$ pihole -q gstatic.com
/opt/pihole/query.sh: line 37: 14732 Killed                  awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' <(echo "${lists}") <(echo "${domain}") 2> /dev/null
 Match found in https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts:
   csi.gstatic.com 

Tried with another default adlist: failed

nanopi@nanopi:~$ pihole -g
  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [i] Target: https://mirror1.malwaredomains.com/files/justdomains
  [✓] Status: Retrieval successful
  [i] Received 26860 domains

  [✓] Storing downloaded domains in new gravity database
  [✓] Building tree
  [✓] Swapping databases
  [i] Number of gravity domains: 26860 (26860 unique domains)
  [i] Number of exact blacklisted domains: 6
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 3
  [i] Number of regex whitelist filters: 0
  [✓] Flushing DNS cache
  [✓] Cleaning up stray matter

  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled
nanopi@nanopi:~$ pihole -q gstatic.com
/opt/pihole/query.sh: line 37: 14867 Killed                  awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' <(echo "${lists}") <(echo "${domain}") 2> /dev/null
  [i] No results found for gstatic.com within the block lists

Disabled all exact white- and blacklist and only one default adlist: failed

nanopi@nanopi:~$ pihole -g
  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [i] Target: https://mirror1.malwaredomains.com/files/justdomains
  [✓] Status: No changes detected
  [i] Received 26860 domains

  [✓] Storing downloaded domains in new gravity database
  [✓] Building tree
  [✓] Swapping databases
  [i] Number of gravity domains: 26860 (26860 unique domains)
  [i] Number of exact blacklisted domains: 0
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 0
  [i] Number of regex whitelist filters: 0
  [✓] Flushing DNS cache
  [✓] Cleaning up stray matter

  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled
nanopi@nanopi:~$ pihole -q gstatic.com
/opt/pihole/query.sh: line 37: 15062 Killed                  awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' <(echo "${lists}") <(echo "${domain}") 2> /dev/null
  [i] No results found for gstatic.com within the block lists

Thank you for the detailed information. I'm not sure what is causing the issue, I'll try to get you a test command that runs query.sh with bash -x, I need to see what command line positional variables to use.

sudo bash -x /opt/pihole/query.sh <XXX> <YYY>

I'm not quite sure what you meant with <XXX> <YYY>

nanopi@nanopi:~$ sudo bash -x /opt/pihole/query.sh gstatic.com
[sudo] password for nanopi: 
+ piholeDir=/etc/pihole
+ gravityDBfile=/etc/pihole/gravity.db
+ options=gstatic.com
+ all=
+ exact=
+ blockpage=
+ matchType=match
+ colfile=/opt/pihole/COL_TABLE
+ source /opt/pihole/COL_TABLE
++ [[ -t 1 ]]
+++ tput colors
++ [[ 256 -ge 8 ]]
++ COL_BOLD=''
++ COL_ULINE=''
++ COL_NC=''
++ COL_GRAY=''
++ COL_RED=''
++ COL_GREEN=''
++ COL_YELLOW=''
++ COL_BLUE=''
++ COL_PURPLE=''
++ COL_CYAN=''
++ COL_WHITE=''
++ COL_BLACK=''
++ COL_LIGHT_BLUE=''
++ COL_LIGHT_GREEN=''
++ COL_LIGHT_CYAN=''
++ COL_LIGHT_RED=''
++ COL_URG_RED=''
++ COL_LIGHT_PURPLE=''
++ COL_BROWN=''
++ COL_LIGHT_GRAY=''
++ COL_DARK_GRAY=''
++ TICK='[✓]'
++ CROSS='[✗]'
++ INFO='[i]'
++ QST='[?]'
++ DONE=' done!'
++ OVER='\r'
+ [[ gstatic.com == \-\h ]]
+ [[ gstatic.com == \-\-\h\e\l\p ]]
+ [[ gstatic.com == *\-\b\p* ]]
+ [[ gstatic.com == *\-\a\l\l* ]]
+ [[ gstatic.com == *\-\e\x\a\c\t* ]]
++ sed -E 's/ ?-(bp|adlists?|all|exact) ?//g'
+ options=gstatic.com
+ case "${options}" in
+ domainQuery=gstatic.com
+ [[ -n '' ]]
+ scanDatabaseTable gstatic.com whitelist 0
+ local domain table type querystr result extra
++ printf %q gstatic.com
+ domain=gstatic.com
+ table=whitelist
+ type=0
+ [[ whitelist == \g\r\a\v\i\t\y ]]
+ case "${exact}" in
+ querystr='SELECT domain,enabled FROM domainlist WHERE type = '\''0'\'' AND domain LIKE '\''%gstatic.com%'\'' ESCAPE '\''\'\'''
++ sqlite3 /etc/pihole/gravity.db 'SELECT domain,enabled FROM domainlist WHERE type = '\''0'\'' AND domain LIKE '\''%gstatic.com%'\'' ESCAPE '\''\'\'''
+ result=
+ [[ -z '' ]]
+ return
+ scanDatabaseTable gstatic.com blacklist 1
+ local domain table type querystr result extra
++ printf %q gstatic.com
+ domain=gstatic.com
+ table=blacklist
+ type=1
+ [[ blacklist == \g\r\a\v\i\t\y ]]
+ case "${exact}" in
+ querystr='SELECT domain,enabled FROM domainlist WHERE type = '\''1'\'' AND domain LIKE '\''%gstatic.com%'\'' ESCAPE '\''\'\'''
++ sqlite3 /etc/pihole/gravity.db 'SELECT domain,enabled FROM domainlist WHERE type = '\''1'\'' AND domain LIKE '\''%gstatic.com%'\'' ESCAPE '\''\'\'''
+ result=
+ [[ -z '' ]]
+ return
+ scanRegexDatabaseTable gstatic.com whitelist 2
+ local domain list
+ domain=gstatic.com
+ list=whitelist
+ type=2
+ mapfile -t regexList
++ sqlite3 /etc/pihole/gravity.db 'SELECT domain FROM domainlist WHERE type = 2'
+ [[ 1 -ne 0 ]]
++ printf '%s\n' '(\.|^)*\.services\.generalmagic\.com$'
+ str_regexList='(\.|^)*\.services\.generalmagic\.com$'
+ mapfile -t regexMatches
++ scanList gstatic.com '(\.|^)*\.services\.generalmagic\.com$' regex
++ local domain=gstatic.com 'esc_domain=gstatic\.com' 'lists=(\.|^)*\.services\.generalmagic\.com$' type=regex
++ cd /etc/pihole
++ export LC_CTYPE=C
++ LC_CTYPE=C
++ case "${type}" in
+++ echo '(\.|^)*\.services\.generalmagic\.com$'
++ awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' /dev/fd/63 /dev/fd/62
+++ echo gstatic.com
/opt/pihole/query.sh: line 37:  4134 Killed                  awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' <(echo "${lists}") <(echo "${domain}") 2> /dev/null
+ [[ 0 -ne 0 ]]
+ scanRegexDatabaseTable gstatic.com blacklist 3
+ local domain list
+ domain=gstatic.com
+ list=blacklist
+ type=3
+ mapfile -t regexList
++ sqlite3 /etc/pihole/gravity.db 'SELECT domain FROM domainlist WHERE type = 3'
+ [[ 23 -ne 0 ]]
++ printf '%s\n' '^(.+[.-])?ad[sxv]?[0-9]*[.-]' '^(.+[.-])?adse?rv(er?|ice)?s?[0-9]*[.-]' '^(.+[.-])?telemetry[.-]' '^(www[0-9].)?xn–' '^adim(age|g)s?[0-9][.-]' '^adtrack(er|ing)?[0-9]*[.-]' '^advert(s|is(ing|ements?))?[0-9][.-]' '^aff(iliat(es?|ion))?[.-]' '^analytics?[.-]' '^banners?[.-]' '^beacons?[0-9][.-]' '^count(ers?)?[0-9]*[.-]' '^mads.' '^pixels?[-.]' '^stat(s|istics)?[0-9][_.-]' '^track(ers?|ing)?[0-9][_.-]' '^traff(ic)?[.-]' '^.*metric.*\..*\..*$' '^logs?\..*\..*$' '(^|[-_.]+)(m?a(d((vert(s|is(ing|e?ments?))?|im(age|g)s?)|([vx]|s(e?rv(er?|ice)?s?)?)|track(ers?|ing))?|ff(iliat(es?|ion))?|nalytics?)|b((anner|eacon)s?)|count(ers?)?|pixels?|stat(s|istics?)?|t(elemetry|ra(ffic|ck(ers?|ing))))[-_]*[0-9]*[-_.]' '^(.+[_.-])?adse?rv(er?|ice)?s?[0-9]*[_.-]' '^(.+[_.-])?ad[sxv]?[0-9]*[_.-]' '(^|\.)asn\.advolution\.de$'
+ str_regexList='^(.+[.-])?ad[sxv]?[0-9]*[.-]
^(.+[.-])?adse?rv(er?|ice)?s?[0-9]*[.-]
^(.+[.-])?telemetry[.-]
^(www[0-9].)?xn–
^adim(age|g)s?[0-9][.-]
^adtrack(er|ing)?[0-9]*[.-]
^advert(s|is(ing|ements?))?[0-9][.-]
^aff(iliat(es?|ion))?[.-]
^analytics?[.-]
^banners?[.-]
^beacons?[0-9][.-]
^count(ers?)?[0-9]*[.-]
^mads.
^pixels?[-.]
^stat(s|istics)?[0-9][_.-]
^track(ers?|ing)?[0-9][_.-]
^traff(ic)?[.-]
^.*metric.*\..*\..*$
^logs?\..*\..*$
(^|[-_.]+)(m?a(d((vert(s|is(ing|e?ments?))?|im(age|g)s?)|([vx]|s(e?rv(er?|ice)?s?)?)|track(ers?|ing))?|ff(iliat(es?|ion))?|nalytics?)|b((anner|eacon)s?)|count(ers?)?|pixels?|stat(s|istics?)?|t(elemetry|ra(ffic|ck(ers?|ing))))[-_]*[0-9]*[-_.]
^(.+[_.-])?adse?rv(er?|ice)?s?[0-9]*[_.-]
^(.+[_.-])?ad[sxv]?[0-9]*[_.-]
(^|\.)asn\.advolution\.de$'
+ mapfile -t regexMatches
++ scanList gstatic.com '^(.+[.-])?ad[sxv]?[0-9]*[.-]
^(.+[.-])?adse?rv(er?|ice)?s?[0-9]*[.-]
^(.+[.-])?telemetry[.-]
^(www[0-9].)?xn–
^adim(age|g)s?[0-9][.-]
^adtrack(er|ing)?[0-9]*[.-]
^advert(s|is(ing|ements?))?[0-9][.-]
^aff(iliat(es?|ion))?[.-]
^analytics?[.-]
^banners?[.-]
^beacons?[0-9][.-]
^count(ers?)?[0-9]*[.-]
^mads.
^pixels?[-.]
^stat(s|istics)?[0-9][_.-]
^track(ers?|ing)?[0-9][_.-]
^traff(ic)?[.-]
^.*metric.*\..*\..*$
^logs?\..*\..*$
(^|[-_.]+)(m?a(d((vert(s|is(ing|e?ments?))?|im(age|g)s?)|([vx]|s(e?rv(er?|ice)?s?)?)|track(ers?|ing))?|ff(iliat(es?|ion))?|nalytics?)|b((anner|eacon)s?)|count(ers?)?|pixels?|stat(s|istics?)?|t(elemetry|ra(ffic|ck(ers?|ing))))[-_]*[0-9]*[-_.]
^(.+[_.-])?adse?rv(er?|ice)?s?[0-9]*[_.-]
^(.+[_.-])?ad[sxv]?[0-9]*[_.-]
(^|\.)asn\.advolution\.de$' regex
++ local domain=gstatic.com 'esc_domain=gstatic\.com' 'lists=^(.+[.-])?ad[sxv]?[0-9]*[.-]
^(.+[.-])?adse?rv(er?|ice)?s?[0-9]*[.-]
^(.+[.-])?telemetry[.-]
^(www[0-9].)?xn–
^adim(age|g)s?[0-9][.-]
^adtrack(er|ing)?[0-9]*[.-]
^advert(s|is(ing|ements?))?[0-9][.-]
^aff(iliat(es?|ion))?[.-]
^analytics?[.-]
^banners?[.-]
^beacons?[0-9][.-]
^count(ers?)?[0-9]*[.-]
^mads.
^pixels?[-.]
^stat(s|istics)?[0-9][_.-]
^track(ers?|ing)?[0-9][_.-]
^traff(ic)?[.-]
^.*metric.*\..*\..*$
^logs?\..*\..*$
(^|[-_.]+)(m?a(d((vert(s|is(ing|e?ments?))?|im(age|g)s?)|([vx]|s(e?rv(er?|ice)?s?)?)|track(ers?|ing))?|ff(iliat(es?|ion))?|nalytics?)|b((anner|eacon)s?)|count(ers?)?|pixels?|stat(s|istics?)?|t(elemetry|ra(ffic|ck(ers?|ing))))[-_]*[0-9]*[-_.]
^(.+[_.-])?adse?rv(er?|ice)?s?[0-9]*[_.-]
^(.+[_.-])?ad[sxv]?[0-9]*[_.-]
(^|\.)asn\.advolution\.de$' type=regex
++ cd /etc/pihole
++ export LC_CTYPE=C
++ LC_CTYPE=C
++ case "${type}" in
+++ echo '^(.+[.-])?ad[sxv]?[0-9]*[.-]
^(.+[.-])?adse?rv(er?|ice)?s?[0-9]*[.-]
^(.+[.-])?telemetry[.-]
^(www[0-9].)?xn–
^adim(age|g)s?[0-9][.-]
^adtrack(er|ing)?[0-9]*[.-]
^advert(s|is(ing|ements?))?[0-9][.-]
^aff(iliat(es?|ion))?[.-]
^analytics?[.-]
^banners?[.-]
^beacons?[0-9][.-]
^count(ers?)?[0-9]*[.-]
^mads.
^pixels?[-.]
^stat(s|istics)?[0-9][_.-]
^track(ers?|ing)?[0-9][_.-]
^traff(ic)?[.-]
^.*metric.*\..*\..*$
^logs?\..*\..*$
(^|[-_.]+)(m?a(d((vert(s|is(ing|e?ments?))?|im(age|g)s?)|([vx]|s(e?rv(er?|ice)?s?)?)|track(ers?|ing))?|ff(iliat(es?|ion))?|nalytics?)|b((anner|eacon)s?)|count(ers?)?|pixels?|stat(s|istics?)?|t(elemetry|ra(ffic|ck(ers?|ing))))[-_]*[0-9]*[-_.]
^(.+[_.-])?adse?rv(er?|ice)?s?[0-9]*[_.-]
^(.+[_.-])?ad[sxv]?[0-9]*[_.-]
(^|\.)asn\.advolution\.de$'
++ awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' /dev/fd/63 /dev/fd/62
+++ echo gstatic.com
+ [[ 0 -ne 0 ]]
+ mapfile -t results
++ scanDatabaseTable gstatic.com gravity
++ local domain table type querystr result extra
+++ printf %q gstatic.com
++ domain=gstatic.com
++ table=gravity
++ type=
++ [[ gravity == \g\r\a\v\i\t\y ]]
++ case "${exact}" in
++ querystr='SELECT gravity.domain,adlist.address,adlist.enabled FROM gravity LEFT JOIN adlist ON adlist.id = gravity.adlist_id WHERE domain LIKE '\''%gstatic.com%'\'' ESCAPE '\''\'\'''
+++ sqlite3 /etc/pihole/gravity.db 'SELECT gravity.domain,adlist.address,adlist.enabled FROM gravity LEFT JOIN adlist ON adlist.id = gravity.adlist_id WHERE domain LIKE '\''%gstatic.com%'\'' ESCAPE '\''\'\'''
++ result=
++ [[ -z '' ]]
++ return
+ [[ -z '' ]]
+ [[ -z '' ]]
+ [[ -z '' ]]
+ echo -e '  [i] No results found for gstatic.com within the block lists'
  [i] No results found for gstatic.com within the block lists
+ exit 0

It seems to be killed while scanning the whitelist regex. Not sure why it is still scanning this list, because I disabled all regex for this test.

EDIT

Yes, it was the whitelist regex

(\.|^)*\.services\.generalmagic\.com$

Despite being already disabled I removed it via web UI and pihole -q runs very fast without error.

nanopi@nanopi:~$ pihole -q gstatic.com
  [i] No results found for gstatic.com within the block lists
nanopi@nanopi:~$ sudo bash -x /opt/pihole/query.sh gstatic.com
+ piholeDir=/etc/pihole
+ gravityDBfile=/etc/pihole/gravity.db
+ options=gstatic.com
+ all=
+ exact=
+ blockpage=
+ matchType=match
+ colfile=/opt/pihole/COL_TABLE
+ source /opt/pihole/COL_TABLE
++ [[ -t 1 ]]
+++ tput colors
++ [[ 256 -ge 8 ]]
++ COL_BOLD=''
++ COL_ULINE=''
++ COL_NC=''
++ COL_GRAY=''
++ COL_RED=''
++ COL_GREEN=''
++ COL_YELLOW=''
++ COL_BLUE=''
++ COL_PURPLE=''
++ COL_CYAN=''
++ COL_WHITE=''
++ COL_BLACK=''
++ COL_LIGHT_BLUE=''
++ COL_LIGHT_GREEN=''
++ COL_LIGHT_CYAN=''
++ COL_LIGHT_RED=''
++ COL_URG_RED=''
++ COL_LIGHT_PURPLE=''
++ COL_BROWN=''
++ COL_LIGHT_GRAY=''
++ COL_DARK_GRAY=''
++ TICK='[✓]'
++ CROSS='[✗]'
++ INFO='[i]'
++ QST='[?]'
++ DONE=' done!'
++ OVER='\r'
+ [[ gstatic.com == \-\h ]]
+ [[ gstatic.com == \-\-\h\e\l\p ]]
+ [[ gstatic.com == *\-\b\p* ]]
+ [[ gstatic.com == *\-\a\l\l* ]]
+ [[ gstatic.com == *\-\e\x\a\c\t* ]]
++ sed -E 's/ ?-(bp|adlists?|all|exact) ?//g'
+ options=gstatic.com
+ case "${options}" in
+ domainQuery=gstatic.com
+ [[ -n '' ]]
+ scanDatabaseTable gstatic.com whitelist 0
+ local domain table type querystr result extra
++ printf %q gstatic.com
+ domain=gstatic.com
+ table=whitelist
+ type=0
+ [[ whitelist == \g\r\a\v\i\t\y ]]
+ case "${exact}" in
+ querystr='SELECT domain,enabled FROM domainlist WHERE type = '\''0'\'' AND domain LIKE '\''%gstatic.com%'\'' ESCAPE '\''\'\'''
++ sqlite3 /etc/pihole/gravity.db 'SELECT domain,enabled FROM domainlist WHERE type = '\''0'\'' AND domain LIKE '\''%gstatic.com%'\'' ESCAPE '\''\'\'''
+ result=
+ [[ -z '' ]]
+ return
+ scanDatabaseTable gstatic.com blacklist 1
+ local domain table type querystr result extra
++ printf %q gstatic.com
+ domain=gstatic.com
+ table=blacklist
+ type=1
+ [[ blacklist == \g\r\a\v\i\t\y ]]
+ case "${exact}" in
+ querystr='SELECT domain,enabled FROM domainlist WHERE type = '\''1'\'' AND domain LIKE '\''%gstatic.com%'\'' ESCAPE '\''\'\'''
++ sqlite3 /etc/pihole/gravity.db 'SELECT domain,enabled FROM domainlist WHERE type = '\''1'\'' AND domain LIKE '\''%gstatic.com%'\'' ESCAPE '\''\'\'''
+ result=
+ [[ -z '' ]]
+ return
+ scanRegexDatabaseTable gstatic.com whitelist 2
+ local domain list
+ domain=gstatic.com
+ list=whitelist
+ type=2
+ mapfile -t regexList
++ sqlite3 /etc/pihole/gravity.db 'SELECT domain FROM domainlist WHERE type = 2'
+ [[ 0 -ne 0 ]]
+ scanRegexDatabaseTable gstatic.com blacklist 3
+ local domain list
+ domain=gstatic.com
+ list=blacklist
+ type=3
+ mapfile -t regexList
++ sqlite3 /etc/pihole/gravity.db 'SELECT domain FROM domainlist WHERE type = 3'
+ [[ 23 -ne 0 ]]
++ printf '%s\n' '^(.+[.-])?ad[sxv]?[0-9]*[.-]' '^(.+[.-])?adse?rv(er?|ice)?s?[0-9]*[.-]' '^(.+[.-])?telemetry[.-]' '^(www[0-9].)?xn–' '^adim(age|g)s?[0-9][.-]' '^adtrack(er|ing)?[0-9]*[.-]' '^advert(s|is(ing|ements?))?[0-9][.-]' '^aff(iliat(es?|ion))?[.-]' '^analytics?[.-]' '^banners?[.-]' '^beacons?[0-9][.-]' '^count(ers?)?[0-9]*[.-]' '^mads.' '^pixels?[-.]' '^stat(s|istics)?[0-9][_.-]' '^track(ers?|ing)?[0-9][_.-]' '^traff(ic)?[.-]' '^.*metric.*\..*\..*$' '^logs?\..*\..*$' '(^|[-_.]+)(m?a(d((vert(s|is(ing|e?ments?))?|im(age|g)s?)|([vx]|s(e?rv(er?|ice)?s?)?)|track(ers?|ing))?|ff(iliat(es?|ion))?|nalytics?)|b((anner|eacon)s?)|count(ers?)?|pixels?|stat(s|istics?)?|t(elemetry|ra(ffic|ck(ers?|ing))))[-_]*[0-9]*[-_.]' '^(.+[_.-])?adse?rv(er?|ice)?s?[0-9]*[_.-]' '^(.+[_.-])?ad[sxv]?[0-9]*[_.-]' '(^|\.)asn\.advolution\.de$'
+ str_regexList='^(.+[.-])?ad[sxv]?[0-9]*[.-]
^(.+[.-])?adse?rv(er?|ice)?s?[0-9]*[.-]
^(.+[.-])?telemetry[.-]
^(www[0-9].)?xn–
^adim(age|g)s?[0-9][.-]
^adtrack(er|ing)?[0-9]*[.-]
^advert(s|is(ing|ements?))?[0-9][.-]
^aff(iliat(es?|ion))?[.-]
^analytics?[.-]
^banners?[.-]
^beacons?[0-9][.-]
^count(ers?)?[0-9]*[.-]
^mads.
^pixels?[-.]
^stat(s|istics)?[0-9][_.-]
^track(ers?|ing)?[0-9][_.-]
^traff(ic)?[.-]
^.*metric.*\..*\..*$
^logs?\..*\..*$
(^|[-_.]+)(m?a(d((vert(s|is(ing|e?ments?))?|im(age|g)s?)|([vx]|s(e?rv(er?|ice)?s?)?)|track(ers?|ing))?|ff(iliat(es?|ion))?|nalytics?)|b((anner|eacon)s?)|count(ers?)?|pixels?|stat(s|istics?)?|t(elemetry|ra(ffic|ck(ers?|ing))))[-_]*[0-9]*[-_.]
^(.+[_.-])?adse?rv(er?|ice)?s?[0-9]*[_.-]
^(.+[_.-])?ad[sxv]?[0-9]*[_.-]
(^|\.)asn\.advolution\.de$'
+ mapfile -t regexMatches
++ scanList gstatic.com '^(.+[.-])?ad[sxv]?[0-9]*[.-]
^(.+[.-])?adse?rv(er?|ice)?s?[0-9]*[.-]
^(.+[.-])?telemetry[.-]
^(www[0-9].)?xn–
^adim(age|g)s?[0-9][.-]
^adtrack(er|ing)?[0-9]*[.-]
^advert(s|is(ing|ements?))?[0-9][.-]
^aff(iliat(es?|ion))?[.-]
^analytics?[.-]
^banners?[.-]
^beacons?[0-9][.-]
^count(ers?)?[0-9]*[.-]
^mads.
^pixels?[-.]
^stat(s|istics)?[0-9][_.-]
^track(ers?|ing)?[0-9][_.-]
^traff(ic)?[.-]
^.*metric.*\..*\..*$
^logs?\..*\..*$
(^|[-_.]+)(m?a(d((vert(s|is(ing|e?ments?))?|im(age|g)s?)|([vx]|s(e?rv(er?|ice)?s?)?)|track(ers?|ing))?|ff(iliat(es?|ion))?|nalytics?)|b((anner|eacon)s?)|count(ers?)?|pixels?|stat(s|istics?)?|t(elemetry|ra(ffic|ck(ers?|ing))))[-_]*[0-9]*[-_.]
^(.+[_.-])?adse?rv(er?|ice)?s?[0-9]*[_.-]
^(.+[_.-])?ad[sxv]?[0-9]*[_.-]
(^|\.)asn\.advolution\.de$' regex
++ local domain=gstatic.com 'esc_domain=gstatic\.com' 'lists=^(.+[.-])?ad[sxv]?[0-9]*[.-]
^(.+[.-])?adse?rv(er?|ice)?s?[0-9]*[.-]
^(.+[.-])?telemetry[.-]
^(www[0-9].)?xn–
^adim(age|g)s?[0-9][.-]
^adtrack(er|ing)?[0-9]*[.-]
^advert(s|is(ing|ements?))?[0-9][.-]
^aff(iliat(es?|ion))?[.-]
^analytics?[.-]
^banners?[.-]
^beacons?[0-9][.-]
^count(ers?)?[0-9]*[.-]
^mads.
^pixels?[-.]
^stat(s|istics)?[0-9][_.-]
^track(ers?|ing)?[0-9][_.-]
^traff(ic)?[.-]
^.*metric.*\..*\..*$
^logs?\..*\..*$
(^|[-_.]+)(m?a(d((vert(s|is(ing|e?ments?))?|im(age|g)s?)|([vx]|s(e?rv(er?|ice)?s?)?)|track(ers?|ing))?|ff(iliat(es?|ion))?|nalytics?)|b((anner|eacon)s?)|count(ers?)?|pixels?|stat(s|istics?)?|t(elemetry|ra(ffic|ck(ers?|ing))))[-_]*[0-9]*[-_.]
^(.+[_.-])?adse?rv(er?|ice)?s?[0-9]*[_.-]
^(.+[_.-])?ad[sxv]?[0-9]*[_.-]
(^|\.)asn\.advolution\.de$' type=regex
++ cd /etc/pihole
++ export LC_CTYPE=C
++ LC_CTYPE=C
++ case "${type}" in
+++ echo '^(.+[.-])?ad[sxv]?[0-9]*[.-]
^(.+[.-])?adse?rv(er?|ice)?s?[0-9]*[.-]
^(.+[.-])?telemetry[.-]
^(www[0-9].)?xn–
^adim(age|g)s?[0-9][.-]
^adtrack(er|ing)?[0-9]*[.-]
^advert(s|is(ing|ements?))?[0-9][.-]
^aff(iliat(es?|ion))?[.-]
^analytics?[.-]
^banners?[.-]
^beacons?[0-9][.-]
^count(ers?)?[0-9]*[.-]
^mads.
^pixels?[-.]
^stat(s|istics)?[0-9][_.-]
^track(ers?|ing)?[0-9][_.-]
^traff(ic)?[.-]
^.*metric.*\..*\..*$
^logs?\..*\..*$
(^|[-_.]+)(m?a(d((vert(s|is(ing|e?ments?))?|im(age|g)s?)|([vx]|s(e?rv(er?|ice)?s?)?)|track(ers?|ing))?|ff(iliat(es?|ion))?|nalytics?)|b((anner|eacon)s?)|count(ers?)?|pixels?|stat(s|istics?)?|t(elemetry|ra(ffic|ck(ers?|ing))))[-_]*[0-9]*[-_.]
^(.+[_.-])?adse?rv(er?|ice)?s?[0-9]*[_.-]
^(.+[_.-])?ad[sxv]?[0-9]*[_.-]
(^|\.)asn\.advolution\.de$'
++ awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' /dev/fd/63 /dev/fd/62
+++ echo gstatic.com
+ [[ 0 -ne 0 ]]
+ mapfile -t results
++ scanDatabaseTable gstatic.com gravity
++ local domain table type querystr result extra
+++ printf %q gstatic.com
++ domain=gstatic.com
++ table=gravity
++ type=
++ [[ gravity == \g\r\a\v\i\t\y ]]
++ case "${exact}" in
++ querystr='SELECT gravity.domain,adlist.address,adlist.enabled FROM gravity LEFT JOIN adlist ON adlist.id = gravity.adlist_id WHERE domain LIKE '\''%gstatic.com%'\'' ESCAPE '\''\'\'''
+++ sqlite3 /etc/pihole/gravity.db 'SELECT gravity.domain,adlist.address,adlist.enabled FROM gravity LEFT JOIN adlist ON adlist.id = gravity.adlist_id WHERE domain LIKE '\''%gstatic.com%'\'' ESCAPE '\''\'\'''
++ result=
++ [[ -z '' ]]
++ return
+ [[ -z '' ]]
+ [[ -z '' ]]
+ [[ -z '' ]]
+ echo -e '  [i] No results found for gstatic.com within the block lists'
  [i] No results found for gstatic.com within the block lists
+ exit 0

I re-enabled all adlists and black/whitelist (without the specified whitelist regex) and it worked without issues.
I'm not very good at regex, is this an faulty one?

(\.|^)*\.services\.generalmagic\.com

If I remember correctly I created it with pihole webUI by entering *.service*.generalmagic*.com and setting type to 'whitecard whitelist'

Judging from the results, yes.

https://docs.pi-hole.net/ftldns/regex/tutorial/

Wildcard is for exact domains, the wildcard blocks subdomains of the top domain. Entering regex chars like the * and choosing wildcard is trying to wildcard that value.

That says literally “Zero or more (.|^) chars” which is not right.

regexr.com/4v4vl

Edit: copy paste breaks the formatting but the regexr link shows what I mean.

1 Like

Thanks for the answer. I think I misinterpreted wildcard - I'm used to understand 'I can use wildcard characters'.
I tried to match domains like

content1.ro.m71os.services.generalmagic.com
shop1.m7.services.generalmagic.com
pubsub1.ro.m71os.services.generalmagic.com
overlays1.ro.m71os.services.generalmagic.com

I think using services.generalmagic.com as wildcard whitelist will be sufficient.

I don't think pihole should have a full-grown regex validator (In terms of 'This is what the regex means') but maybe a simple syntax checker ('This is not a valid regex und will never match any domain.) - especially as a non-valid regex can break pihole's scripts. Or pihole should be more robust against faulty regex.

edit
Changes the topic's title to what I think reflects the issue better.

Thank you, that will help with users searching the forums.

Regex is a very powerful tool and can be used to customize the user experience. It's not easy to validate regex but I agree that having some kind of check would be helpful. In this case it would not have applied since the input was for wildcard values and should not have contained any regex values at all. This particular issue could be helped with checking for input that should only contain chars found in domain names. Opening a bug report on our core or adminlte repositories would be helpful in tracking that specific case.

I selected "Type: Wildcard Whitelist" but it appeared as "Regex Whitelist". Not sure if it should be like this - it is automatically converted to a regex style. See picture for test.domain - it was added as wildcard whitelist but appears as Regex Whitelist.

Will do that.

Yes, that is correct. Wildcards are converted to regex value when committed.

Issue opend:

1 Like

This is functionally the same issue as: Pi-hole becomes unresponsive during pihole -d and -q

Faulty regex causes -q (and, by extension -d) to stall out

(You're not alone!)

Thanks for the hint :slight_smile:

As the root cause was the same with us I opened a new topic on that issue:

4 posts were merged into an existing topic: Rename "wildcard" blacklist/whitelist because it is misleading and lead to regex errors

As mentioned in the other thread, this is already in the works for quite some time but will likely not hit v5.0 not not push back the release unnecessarily long.

Thanks for this, however, we will simply reuse the inbuilt PHP domain validator we already used before. It also checks for the maximum length of subdomains, etc.

3 Likes

Can we call this the solution to the topic then?