I've been using pihole -q and got some strange output
nanopi@nanopi:~$ pihole -q prophet.heise.de
/opt/pihole/query.sh: line 37: 21386 Killed awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' <(echo "${lists}") <(echo "${domain}") 2> /dev/null
Match found in https://hosts-file.net/ad_servers.txt:
prophet.heise.de
I'm unsure what the first line means - error at line 37?
/opt/pihole/query.sh: line 37: 21500 Killed awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' <(echo "${lists}") <(echo "${domain}") 2> /dev/null
The relevant part of query.sh is
# /dev/null forces filename to be printed when only one list has been generated
# shellcheck disable=SC2086
case "${type}" in
"exact" ) grep -i -E -l "(^|(?<!#)\\s)${esc_domain}($|\\s|#)" ${lists} /dev/null 2>/dev/null;;
# Create array of regexps
# Iterate through each regexp and check whether it matches the domainQuery
# If it does, print the matching regexp and continue looping
# Input 1 - regexps | Input 2 - domainQuery
"regex" ) awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' \
<(echo "${lists}") <(echo "${domain}") 2>/dev/null;;
* ) grep -i "${esc_domain}" ${lists} /dev/null 2>/dev/null;;
esac
Pi-hole version is v4.3.2-433-g8ecaaba (Latest: v4.3.5)
AdminLTE version is v4.3.2-433-g0b4a5de5 (Latest: v4.3.3)
FTL version is vDev-71e8498 (Latest: v4.3.1)
Pi-hole version is v4.3.2-433-g8ecaaba (Latest: v4.3.5)
AdminLTE version is v4.3.2-433-g0b4a5de (Latest: v4.3.3)
FTL version is vDev-71e8498 (Latest: v4.3.1
Your init system has killed the process for some reason. Could be the process took too long, used too many resources or the system needed to purge running tasks for its own needs.
Edit: May be a bad regex is looping as well. The debug log will help a lot.
What is the board that this is running on? How much RAM and free disk space?
You're running out of resources and the init system is trying to kill what it can to keep from going down:
2020-02-25 11:06:07: (server.c.1464) server started (lighttpd/1.4.53)
2020-02-25 12:17:17: (mod_fastcgi.c.421) FastCGI-stderr: PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/admin/api_db.php on line 112
2020-02-25 19:34:45: (mod_fastcgi.c.421) FastCGI-stderr: PHP Warning: file(http://127.0.0.1/admin/scripts/pi-hole/php/queryads.php?domain=localhost&bp): failed to open stream: HTTP request failed! in /var/www/html/pihole/index.php on line 134
2020-02-25 19:34:45: (mod_fastcgi.c.421) FastCGI-stderr: PHP Warning: array_filter() expects parameter 1 to be array, string given in /var/www/html/pihole/index.php on line 135
2020-02-25 19:34:45: (mod_fastcgi.c.421) FastCGI-stderr: PHP Warning: array_values() expects parameter 1 to be array, null given in /var/www/html/pihole/index.php on line 135
2020-02-25 19:34:48: (mod_fastcgi.c.421) FastCGI-stderr: PHP Warning: file(http://127.0.0.1/admin/scripts/pi-hole/php/queryads.php?domain=localhost&bp): failed to open stream: HTTP request failed! in /var/www/html/pihole/index.php on line 134
2020-02-25 19:34:48: (mod_fastcgi.c.421) FastCGI-stderr: PHP Warning: array_filter() expects parameter 1 to be array, string given in /var/www/html/pihole/index.php on line 135
2020-02-25 19:34:48: (mod_fastcgi.c.421) FastCGI-stderr: PHP Warning: array_values() expects parameter 1 to be array, null given in /var/www/html/pihole/index.php on line 135
[2020-02-25 12:24:01.862 19005] Received: Real-time signal 0 (34 -> 0)
[2020-02-25 12:24:01.912 19005] Compiled 1 whitelist and 23 blacklist regex filters in 15.1 msec
[2020-02-25 12:24:14.388 19005] Received: Real-time signal 0 (34 -> 0)
[2020-02-25 12:24:14.439 19005] Compiled 1 whitelist and 23 blacklist regex filters in 15.1 msec
[2020-02-25 12:24:17.269 19005] Received: Real-time signal 0 (34 -> 0)
[2020-02-25 12:24:17.320 19005] Compiled 1 whitelist and 23 blacklist regex filters in 15.0 msec
[2020-02-25 12:24:17.930 19005] Received: Real-time signal 0 (34 -> 0)
[2020-02-25 12:24:17.980 19005] Compiled 1 whitelist and 23 blacklist regex filters in 15.0 msec
[2020-02-25 12:24:18.938 19005] Received: Real-time signal 0 (34 -> 0)
[2020-02-25 12:24:18.988 19005] Compiled 1 whitelist and 23 blacklist regex filters in 15.1 msec
[2020-02-25 12:24:20.229 19005] Received: Real-time signal 0 (34 -> 0)
[2020-02-25 12:24:20.278 19005] Compiled 1 whitelist and 23 blacklist regex filters in 15.1 msec
[2020-02-25 12:24:20.911 19005] Received: Real-time signal 0 (34 -> 0)
[2020-02-25 12:24:20.976 19005] Compiled 1 whitelist and 23 blacklist regex filters in 15.1 msec
[2020-02-25 12:24:24.105 19005] Received: Real-time signal 0 (34 -> 0)
[2020-02-25 12:24:24.155 19005] Compiled 1 whitelist and 23 blacklist regex filters in 15.2 msec
[2020-02-25 12:24:28.357 19005] Received: Real-time signal 0 (34 -> 0)
[2020-02-25 12:24:28.407 19005] Compiled 1 whitelist and 23 blacklist regex filters in 15.2 msec
[2020-02-25 12:24:30.845 19005] Received: Real-time signal 0 (34 -> 0)
[2020-02-25 12:24:30.892 19005] Compiled 1 whitelist and 23 blacklist regex filters in 15.2 msec
[2020-02-25 12:24:32.088 19005] Received: Real-time signal 0 (34 -> 0)
[2020-02-25 12:24:32.140 19005] Compiled 1 whitelist and 23 blacklist regex filters in 15.2 msec
[2020-02-25 12:24:32.638 19005] Received: Real-time signal 0 (34 -> 0)
[2020-02-25 12:24:32.687 19005] Compiled 1 whitelist and 23 blacklist regex filters in 15.3 msec
[2020-02-25 12:24:34.373 19005] Received: Real-time signal 0 (34 -> 0)
[2020-02-25 12:24:34.425 19005] Compiled 1 whitelist and 23 blacklist regex filters in 15.3 msec
[2020-02-25 12:24:35.045 19005] Received: Real-time signal 0 (34 -> 0)
[2020-02-25 12:24:35.093 19005] Compiled 1 whitelist and 23 blacklist regex filters in 15.3 msec
Disabled all regex filters via web GUI and the file:/// adlist. Performed pihole -g, but error still occurs
[✓] Storing downloaded domains in new gravity database
[✓] Building tree
[✓] Swapping databases
[i] Number of gravity domains: 252334 (204806 unique domains)
[i] Number of exact blacklisted domains: 6
[i] Number of regex blacklist filters: 0
[i] Number of exact whitelisted domains: 3
[i] Number of regex whitelist filters: 0
[✓] Flushing DNS cache
[✓] Cleaning up stray matter
[✓] DNS service is running
[✓] Pi-hole blocking is Enabled
nanopi@nanopi:~$ pihole -q gstatic.com
/opt/pihole/query.sh: line 37: 11252 Killed awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' <(echo "${lists}") <(echo "${domain}") 2> /dev/null
Match found in https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts:
csi.gstatic.com
nanopi@nanopi:~$ pihole -g
[sudo] password for nanopi:
[i] Neutrino emissions detected...
[✓] Pulling blocklist source list into range
[✓] Preparing new gravity database
[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
[✓] Status: Retrieval successful
[i] Received 51220 domains
[i] Target: https://mirror1.malwaredomains.com/files/justdomains
[✓] Status: No changes detected
[i] Received 26860 domains
[i] Target: http://sysctl.org/cameleon/hosts
[✓] Status: No changes detected
[i] Received 20567 domains
[i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
[✓] Status: No changes detected
[i] Received 34 domains
[i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
[✓] Status: No changes detected
[i] Received 2701 domains
[i] Target: https://hosts-file.net/ad_servers.txt
[✓] Status: No changes detected
[i] Received 45736 domains
[✓] Storing downloaded domains in new gravity database
[✓] Building tree
[✓] Swapping databases
[i] Number of gravity domains: 147118 (125108 unique domains)
[i] Number of exact blacklisted domains: 6
[i] Number of regex blacklist filters: 0
[i] Number of exact whitelisted domains: 3
[i] Number of regex whitelist filters: 0
[✓] Flushing DNS cache
[✓] Cleaning up stray matter
[✓] DNS service is running
[✓] Pi-hole blocking is Enabled
nanopi@nanopi:~$ pihole -q gstatic.com
/opt/pihole/query.sh: line 37: 14510 Killed awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' <(echo "${lists}") <(echo "${domain}") 2> /dev/null
Match found in https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts:
csi.gstatic.com
Match found in https://hosts-file.net/ad_servers.txt:
csi.gstatic.com
p2-aahhyknavsj2m-wtnlrzkba6lht33q-if-v6exp3-v4.metric.gstatic.com
p2-f6rp6piuxns4u-uzq4vp76bu3w2tso-if-v6exp3-v4.metric.gstatic.com
p2-n3zurhre4jjvk-can5rb2f2a4urcxh-if-v6exp3-v4.metric.gstatic.com
p4-ajvwyt5lpjazy-us7r2dzqcjsqh7pt-164149-i1-v6exp3-v4.metric.gstatic.com
p4-ajvwyt5lpjazy-us7r2dzqcjsqh7pt-164149-i2-v6exp3-ds.metric.gstatic.com
p4-ajvwyt5lpjazy-us7r2dzqcjsqh7pt-164149-s1-v6exp3-v4.metric.gstatic.com
p4-ajvwyt5lpjazy-us7r2dzqcjsqh7pt-if-v6exp3-v4.metric.gstatic.com
p5-lj5aujgj7jl7w-r2pmxqvndsgx2im2-931517-i1-v6exp3-v4.metric.gstatic.com
p5-lj5aujgj7jl7w-r2pmxqvndsgx2im2-931517-i2-v6exp3-ds.metric.gstatic.com
s6.netlogstatic.com
v6exp3-ds.metric.gstatic.com
v6exp3-v4.metric.gstatic.com
Tried with only one default adlist: failed
nanopi@nanopi:~$ pihole -g
[i] Neutrino emissions detected...
[✓] Pulling blocklist source list into range
[✓] Preparing new gravity database
[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
[✓] Status: Retrieval successful
[i] Received 51220 domains
[✓] Storing downloaded domains in new gravity database
[✓] Building tree
[✓] Swapping databases
[i] Number of gravity domains: 51220 (51220 unique domains)
[i] Number of exact blacklisted domains: 6
[i] Number of regex blacklist filters: 0
[i] Number of exact whitelisted domains: 3
[i] Number of regex whitelist filters: 0
[✓] Flushing DNS cache
[✓] Cleaning up stray matter
[✓] DNS service is running
[✓] Pi-hole blocking is Enabled
nanopi@nanopi:~$ pihole -q gstatic.com
/opt/pihole/query.sh: line 37: 14732 Killed awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' <(echo "${lists}") <(echo "${domain}") 2> /dev/null
Match found in https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts:
csi.gstatic.com
Tried with another default adlist: failed
nanopi@nanopi:~$ pihole -g
[i] Neutrino emissions detected...
[✓] Pulling blocklist source list into range
[✓] Preparing new gravity database
[i] Target: https://mirror1.malwaredomains.com/files/justdomains
[✓] Status: Retrieval successful
[i] Received 26860 domains
[✓] Storing downloaded domains in new gravity database
[✓] Building tree
[✓] Swapping databases
[i] Number of gravity domains: 26860 (26860 unique domains)
[i] Number of exact blacklisted domains: 6
[i] Number of regex blacklist filters: 0
[i] Number of exact whitelisted domains: 3
[i] Number of regex whitelist filters: 0
[✓] Flushing DNS cache
[✓] Cleaning up stray matter
[✓] DNS service is running
[✓] Pi-hole blocking is Enabled
nanopi@nanopi:~$ pihole -q gstatic.com
/opt/pihole/query.sh: line 37: 14867 Killed awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' <(echo "${lists}") <(echo "${domain}") 2> /dev/null
[i] No results found for gstatic.com within the block lists
Disabled all exact white- and blacklist and only one default adlist: failed
nanopi@nanopi:~$ pihole -g
[i] Neutrino emissions detected...
[✓] Pulling blocklist source list into range
[✓] Preparing new gravity database
[i] Target: https://mirror1.malwaredomains.com/files/justdomains
[✓] Status: No changes detected
[i] Received 26860 domains
[✓] Storing downloaded domains in new gravity database
[✓] Building tree
[✓] Swapping databases
[i] Number of gravity domains: 26860 (26860 unique domains)
[i] Number of exact blacklisted domains: 0
[i] Number of regex blacklist filters: 0
[i] Number of exact whitelisted domains: 0
[i] Number of regex whitelist filters: 0
[✓] Flushing DNS cache
[✓] Cleaning up stray matter
[✓] DNS service is running
[✓] Pi-hole blocking is Enabled
nanopi@nanopi:~$ pihole -q gstatic.com
/opt/pihole/query.sh: line 37: 15062 Killed awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' <(echo "${lists}") <(echo "${domain}") 2> /dev/null
[i] No results found for gstatic.com within the block lists
Thank you for the detailed information. I'm not sure what is causing the issue, I'll try to get you a test command that runs query.sh with bash -x, I need to see what command line positional variables to use.
I re-enabled all adlists and black/whitelist (without the specified whitelist regex) and it worked without issues.
I'm not very good at regex, is this an faulty one?
(\.|^)*\.services\.generalmagic\.com
If I remember correctly I created it with pihole webUI by entering *.service*.generalmagic*.com and setting type to 'whitecard whitelist'
Wildcard is for exact domains, the wildcard blocks subdomains of the top domain. Entering regex chars like the * and choosing wildcard is trying to wildcard that value.
That says literally “Zero or more (.|^) chars” which is not right.
regexr.com/4v4vl
Edit: copy paste breaks the formatting but the regexr link shows what I mean.
I think using services.generalmagic.com as wildcard whitelist will be sufficient.
I don't think pihole should have a full-grown regex validator (In terms of 'This is what the regex means') but maybe a simple syntax checker ('This is not a valid regex und will never match any domain.) - especially as a non-valid regex can break pihole's scripts. Or pihole should be more robust against faulty regex.
edit
Changes the topic's title to what I think reflects the issue better.
