Yes, in between FTL is re-reading the lists. I have never seen this taking more than (at most!) very few seconds. I wonder if this has to do with your 30 clients your tiny devices handles. Could you put
DEBUG_DATABASE=true
into your /etc/pihole/pihole-FTL.conf, run pihole restartdns and repeat you test?
I wonder what the log may reveal to us.
PM'd the results, as they were too long to share here.
On a side-note, given that I'm the only one who seems to have experienced these issues (I keep hoping for my sanity that someone else chimes in with a "me too!") I'm starting to wonder if it's specific to my setup (RPi-0, running DietPi).
I may go with a clean install of Raspbian Lite to see if the issue persists, as I'd hate to take up too much developer time if I'm all alone!
It is worth mentioning though that everything was working flawlessly in v4, this issue is in some way related to the beta as far as I can tell.
Due to the per-client options, a lot of the internal machinery got more complex. The issue is that we have to reread this all when changing something somewhere, I just never saw any notable delay in any of my tests.
Okay, so let's look at your log excerpt. The disable command initiates this instantaneously:
[2020-01-27 16:26:43.842 638] Reloading DNS cache
Querying the configuration for all the clients is finished here:
[2020-01-27 16:26:44.172 638] Querying gravity database table vw_gravity
which is finished only two minutes later:
[2020-01-27 16:28:13.311 638] Querying gravity database table vw_regex_blacklist
[2020-01-27 16:28:13.316 638] Querying gravity database table vw_regex_whitelist
This is very strange as in here we only COUNT the number of distinct domains. This almost looks like you are missing the index on the gravity table?...
Could you please run
time sqlite3 /etc/pihole/gravity.db "SELECT count(DISTINCT domain) FROM gravity;"
time sqlite3 /etc/pihole/gravity.db "SELECT count(DISTINCT domain) FROM vw_gravity;"
and report the output?
Once done, does a run of pihole -g help resolving this issue (test again with above two lines)?
I have the same problem. When enabling or disabling blocking, there is an approx. 1 minute delay before DNS resolution works.
Might be on the right track, as this is the result of the above 2 time commands:
time sqlite3 /etc/pihole/gravity.db "SELECT count(DISTINCT domain) FROM gravity;"
2378482
real 1m1.658s
user 0m32.470s
sys 0m3.541s
time sqlite3 /etc/pihole/gravity.db "SELECT count(DISTINCT domain) FROM vw_gravity;"
2378482
real 0m51.983s
user 0m43.606s
sys 0m3.199s
I'm guesing the Pi (I'm running a Pi3b) just can't easily handle parsing out the distinct records, when the blocklist is so large and with so many duplicates:
[i] Number of gravity domains: 4645932 (2378482 unique domains)
I use a Raspberry Pi 2B and I have the same problem.
I am using Raspbian Lite.
The problem continues after run pihole -g.
time sqlite3 /etc/pihole/gravity.db "SELECT count(DISTINCT domain) FROM gravity;"
1085359real 0m31.359s
user 0m26.293s
sys 0m4.994stime sqlite3 /etc/pihole/gravity.db "SELECT count(DISTINCT domain) FROM vw_gravity;"
1085357real 0m43.238s
user 0m38.431s
sys 0m4.676sNumber of gravity domains: 2272668 (1085357 unique domains)
The problem, as I see it, isn't limited to disabling pihole. I noticed the delay (DNS resolution unavailable) when trying to get acquainted with group management. In the group management documentation, it says you need to enter the command pihole restartdns reload-lists, whenever changing things in group management. I never assumed this could affect other pihole features, such as disabling pihole, so I requested to add code, already available in the sources, to display a warning (count down), whenever this delay occurred.
You can read my request here, unfortunately, nobody confirmed the delay. You still might want to add a comment in the topic, this to ensure that you get at least a warning (on the console, if possible in the web interface), in case the problem cannot be resolved entirely (I'm hoping for a performance improvement, but don't know if this is possible).
Off topic:
I haven't been using pihole disable for a long time, since the feature (disabling / enabling pihole) is applied to pihole and thus to ALL clients.
In pihole v.4.3.2, disabling pihole resulted in simply commenting out the gravity list in /etc/dnsmasq.d/01-pihole.conf and inform pihole-FTL.
In pihole beta5, it looks like this is achieved (NOT TESTED) by changing the variable BLOCKING_ENABLED=true and inform pihole-FTL.
Both solutions result in a delay, practically unnoticeable in v4.3.2, aparentlly very much noticeable in beta5.
To overcome this problem (impact on ALL clients), I searched and found a solution for Windows devices.
My solution is explained here, it assumes using unbound, thus avoiding having to go outside the LAN for DNS resolutions.
Note that the solution (the windows command script) can be used without unbound, by simply using an other resolver (208.67.222.222 - OpenDNS). I even implemented (NOT explained in the topic, explained here) running the script, without having the UAC prompt every time.
The down side of this solution: you have to install the script + desktop shortcut on each windows device you want to give the option.
My result for pi zero. My reloading takes minutes. Zero being single core aint helping.
time sqlite3 /etc/pihole/gravity.db "SELECT count(DISTINCT domain) FROM gravity;"
1672511
real 1m32.540s
user 0m54.671s
sys 0m11.784s
pi@RaspPi:~$ time sqlite3 /etc/pihole/gravity.db "SELECT count(DISTINCT domain) FROM vw_gravity;"
1672507
real 1m45.270s
user 1m15.627s
sys 0m13.049s
Thanks for confirming the delay. I will have pihole -g compute the number once and simply store them in the database as numeric value. Your devices will be able to read this number instantly.
WIP
Done. Please try
pihole checkout core tweak/gravity_count
pihole checkout ftl tweak/gravity_count
and check if this reduces the delay you've been observing here. If it does not work initially (strange number shown on the dashboard), make sure you run pihole -g.
Requested branch "tweak/gravity_count" is not available for the ftl checkout
only having core checked out does not solve the problem.
I see the same.
Sorry, it seems my git push command got interrupted without me noticing. FTL should now be available.
After following your steps it works fine.
pihole disable
[2020-02-03 10:04:27.036 28138] Reloading DNS cache
[2020-02-03 10:04:27.036 28138] Blocking status is disabled
[2020-02-03 10:04:27.098 28138] INFO: No regex blacklist entries found
[2020-02-03 10:04:27.099 28138] INFO: No regex whitelist entries found
[2020-02-03 10:04:27.140 28138] Compiled 0 whitelist and 0 blacklist regex filters in 43.4 msec
pihole enable
[2020-02-03 10:06:02.006 28138] Reloading DNS cache
[2020-02-03 10:06:02.006 28138] Blocking status is enabled
[2020-02-03 10:06:02.069 28138] INFO: No regex blacklist entries found
[2020-02-03 10:06:02.071 28138] INFO: No regex whitelist entries found
[2020-02-03 10:06:02.100 28138] Compiled 0 whitelist and 0 blacklist regex filters in 32.4 msec
FTL crashing on the test branch. It will run fine until it gets queried then it crashes.
[2020-02-04 22:00:33.674 26682] Reloading DNS cache
[2020-02-04 22:00:33.684 26682] Blocking status is enabled
[2020-02-04 22:00:33.985 26682] INFO: No regex whitelist entries found
[2020-02-04 22:00:34.119 26682] Compiled 0 whitelist and 7 blacklist regex filters in 162.0 msec
[2020-02-04 22:00:40.931 26682] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-02-04 22:00:40.944 26682] ----------------------------> FTL crashed! <----------------------------
[2020-02-04 22:00:40.946 26682] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-02-04 22:00:40.947 26682] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-02-04 22:00:40.949 26682] and include in your report already the following details:
[2020-02-04 22:00:40.950 26682] FTL has been running for 9 seconds
[2020-02-04 22:00:40.953 26682] FTL branch: tweak/gravity_count
[2020-02-04 22:00:40.959 26682] FTL version: vDev-1af4748
[2020-02-04 22:00:40.961 26682] FTL commit: 1af4748
[2020-02-04 22:00:40.962 26682] FTL date: 2020-02-03 08:43:06 +0100
[2020-02-04 22:00:40.964 26682] FTL user: started as pihole, ended as pihole
[2020-02-04 22:00:40.968 26682] Compiled for arm (compiled on CI) using arm-linux-gnueabihf-gcc (crosstool-NG crosstool-ng-1.22.0-88-g8460611) 4.9.3
[2020-02-04 22:00:40.971 26682] Received signal: Segmentation fault
[2020-02-04 22:00:40.974 26682] at address: 0x226
[2020-02-04 22:00:40.976 26682] with code: SEGV_MAPERR (Address not mapped to object)
[2020-02-04 22:00:40.981 26682] Backtrace:
[2020-02-04 22:00:40.983 26682] B[0000]: 0x483a34, /usr/bin/pihole-FTL(+0x2ba34) [0x483a34]
[2020-02-04 22:00:40.993 26682] B[0001]: 0xb6dbe130, /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0xb6dbe130]
[2020-02-04 22:00:40.995 26682] ------ Listing content of directory /dev/shm ------
[2020-02-04 22:00:40.996 26682] File Mode User:Group Filesize Filename
[2020-02-04 22:00:40.998 26682] rwxrwxrwx root:root 260 .
[2020-02-04 22:00:41.000 26682] rwxr-xr-x root:root 3K ..
[2020-02-04 22:00:41.003 26682] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-02-04 22:00:41.006 26682] rw------- pihole:pihole 12K FTL-dns-cache
[2020-02-04 22:00:41.010 26682] rw------- pihole:pihole 53K FTL-overTime
[2020-02-04 22:00:41.014 26682] rw------- pihole:pihole 1M FTL-queries
[2020-02-04 22:00:41.018 26682] rw------- pihole:pihole 20K FTL-forwarded
[2020-02-04 22:00:41.022 26682] rw------- pihole:pihole 20K FTL-clients
[2020-02-04 22:00:41.027 26682] rw------- pihole:pihole 66K FTL-domains
[2020-02-04 22:00:41.035 26682] rw------- pihole:pihole 45K FTL-strings
[2020-02-04 22:00:41.037 26682] rw------- pihole:pihole 12 FTL-settings
[2020-02-04 22:00:41.041 26682] rw------- pihole:pihole 120 FTL-counters
[2020-02-04 22:00:41.047 26682] rw------- pihole:pihole 28 FTL-lock
[2020-02-04 22:00:41.052 26682] ---------------------------------------------------
[2020-02-04 22:00:41.053 26682] Thank you for helping us to improve our FTL engine!
[2020-02-04 22:00:41.057 26682] FTL terminated!
Second try (okay... maybe 3rd/4th)
pihole.log
Feb 4 22:09:51 dnsmasq[28510]: started, version pi-hole-2.80 cachesize 10000
Feb 4 22:09:51 dnsmasq[28510]: DNS service limited to local subnets
Feb 4 22:09:51 dnsmasq[28510]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify dumpfile
Feb 4 22:09:51 dnsmasq[28510]: using local addresses only for domain use-application-dns.net
Feb 4 22:09:51 dnsmasq[28510]: using nameserver 192.168.1.1#53 for domain 1.168.192.in-addr.arpa
Feb 4 22:09:51 dnsmasq[28510]: using nameserver 192.168.1.1#53 for domain lan
Feb 4 22:09:51 dnsmasq[28510]: using nameserver 127.0.0.1#5353
Feb 4 22:09:52 dnsmasq[28510]: read /etc/hosts - 11 addresses
Feb 4 22:09:52 dnsmasq[28510]: read /etc/pihole/custom.list - 0 addresses
Feb 4 22:09:52 dnsmasq[28510]: read /etc/pihole/local.list - 2 addresses
Feb 4 22:10:16 dnsmasq[28510]: query[A] app-measurement.com from 192.168.1.x
Feb 4 22:10:16 dnsmasq[28510]: gravity blocked app-measurement.com is 0.0.0.0
Feb 4 22:10:53 dnsmasq[28510]: query[A] pi.hole from 192.168.1.x
Feb 4 22:10:53 dnsmasq[28510]: /etc/hosts pi.hole is 192.168.1.x
Feb 4 22:11:20 dnsmasq[28510]: query[A] www.youtube.com from 192.168.1.x
pihole-FTL.log
[2020-02-04 22:09:51.785 28510] Reloading DNS cache
[2020-02-04 22:09:51.792 28510] Blocking status is enabled
[2020-02-04 22:09:52.070 28510] INFO: No regex whitelist entries found
[2020-02-04 22:09:52.231 28510] Compiled 0 whitelist and 7 blacklist regex filters in 191.2 msec
[2020-02-04 22:11:20.795 28510] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-02-04 22:11:20.800 28510] ----------------------------> FTL crashed! <----------------------------
[2020-02-04 22:11:20.803 28510] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-02-04 22:11:20.806 28510] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-02-04 22:11:20.807 28510] and include in your report already the following details:
[2020-02-04 22:11:20.809 28510] FTL has been running for 92 seconds
[2020-02-04 22:11:20.811 28510] FTL branch: tweak/gravity_count
[2020-02-04 22:11:20.813 28510] FTL version: vDev-1af4748
[2020-02-04 22:11:20.815 28510] FTL commit: 1af4748
[2020-02-04 22:11:20.817 28510] FTL date: 2020-02-03 08:43:06 +0100
[2020-02-04 22:11:20.820 28510] FTL user: started as pihole, ended as pihole
[2020-02-04 22:11:20.821 28510] Compiled for arm (compiled on CI) using arm-linux-gnueabihf-gcc (crosstool-NG crosstool-ng-1.22.0-88-g8460611) 4.9.3
[2020-02-04 22:11:20.823 28510] Received signal: Segmentation fault
[2020-02-04 22:11:20.825 28510] at address: 0x226
[2020-02-04 22:11:20.827 28510] with code: SEGV_MAPERR (Address not mapped to object)
[2020-02-04 22:11:20.829 28510] Backtrace:
[2020-02-04 22:11:20.832 28510] B[0000]: 0x45ca34, /usr/bin/pihole-FTL(+0x2ba34) [0x45ca34]
[2020-02-04 22:11:20.834 28510] B[0001]: 0xb6d59130, /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0xb6d59130]
[2020-02-04 22:11:20.836 28510] ------ Listing content of directory /dev/shm ------
[2020-02-04 22:11:20.838 28510] File Mode User:Group Filesize Filename
[2020-02-04 22:11:20.841 28510] rwxrwxrwx root:root 260 .
[2020-02-04 22:11:20.843 28510] rwxr-xr-x root:root 3K ..
[2020-02-04 22:11:20.846 28510] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-02-04 22:11:20.847 28510] rw------- pihole:pihole 12K FTL-dns-cache
[2020-02-04 22:11:20.850 28510] rw------- pihole:pihole 53K FTL-overTime
[2020-02-04 22:11:20.852 28510] rw------- pihole:pihole 1M FTL-queries
[2020-02-04 22:11:20.855 28510] rw------- pihole:pihole 20K FTL-forwarded
[2020-02-04 22:11:20.858 28510] rw------- pihole:pihole 20K FTL-clients
[2020-02-04 22:11:20.860 28510] rw------- pihole:pihole 66K FTL-domains
[2020-02-04 22:11:20.863 28510] rw------- pihole:pihole 45K FTL-strings
[2020-02-04 22:11:20.866 28510] rw------- pihole:pihole 12 FTL-settings
[2020-02-04 22:11:20.869 28510] rw------- pihole:pihole 120 FTL-counters
[2020-02-04 22:11:20.871 28510] rw------- pihole:pihole 28 FTL-lock
[2020-02-04 22:11:20.873 28510] ---------------------------------------------------
[2020-02-04 22:11:20.874 28510] Thank you for helping us to improve our FTL engine!
[2020-02-04 22:11:20.876 28510] FTL terminated!
Meant to chime in on this yesterday morning but I could not get FTL to run after this patch. pihole -r did not fix anything
FTL crashes immediately with this branch on a pi 0.
[2020-02-04 06:18:34.788 15857] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-02-04 06:18:34.788 15857] ----------------------------> FTL crashed! <----------------------------
[2020-02-04 06:18:34.789 15857] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-02-04 06:18:34.789 15857] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-02-04 06:18:34.789 15857] and include in your report already the following details:
[2020-02-04 06:18:34.789 15857] FTL has been running for 8 seconds
[2020-02-04 06:18:34.789 15857] FTL branch: tweak/gravity_count
[2020-02-04 06:18:34.789 15857] FTL version: vDev-1af4748
[2020-02-04 06:18:34.790 15857] FTL commit: 1af4748
[2020-02-04 06:18:34.790 15857] FTL date: 2020-02-03 08:43:06 +0100
[2020-02-04 06:18:34.790 15857] FTL user: started as pihole, ended as pihole
[2020-02-04 06:18:34.790 15857] Compiled for arm (compiled on CI) using arm-linux-gnueabihf-gcc (crosstool-NG crosstool-ng-1.22.0-88-g8460611) 4.9.3
[2020-02-04 06:18:34.791 15857] Received signal: Segmentation fault
[2020-02-04 06:18:34.791 15857] at address: 0x226
[2020-02-04 06:18:34.791 15857] with code: SEGV_MAPERR (Address not mapped to object)
[2020-02-04 06:18:34.791 15857] Backtrace:
[2020-02-04 06:18:34.792 15857] B[0000]: 0x4bda34, /usr/bin/pihole-FTL(+0x2ba34) [0x4bda34]
[2020-02-04 06:18:34.803 15857] B[0001]: 0xb6d39130, /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0xb6d39130]
[2020-02-04 06:18:34.803 15857] ------ Listing content of directory /dev/shm ------
[2020-02-04 06:18:34.803 15857] File Mode User:Group Filesize Filename
[2020-02-04 06:18:34.804 15857] rwxrwxrwx root:root 260 .
[2020-02-04 06:18:34.804 15857] rwxr-xr-x root:root 4K ..
[2020-02-04 06:18:34.805 15857] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-02-04 06:18:34.805 15857] rw------- pihole:pihole 12K FTL-dns-cache
[2020-02-04 06:18:34.806 15857] rw------- pihole:pihole 53K FTL-overTime
[2020-02-04 06:18:34.806 15857] rw------- pihole:pihole 2M FTL-queries
[2020-02-04 06:18:34.807 15857] rw------- pihole:pihole 20K FTL-forwarded
[2020-02-04 06:18:34.808 15857] rw------- pihole:pihole 20K FTL-clients
[2020-02-04 06:18:34.808 15857] rw------- pihole:pihole 66K FTL-domains
[2020-02-04 06:18:34.809 15857] rw------- pihole:pihole 12K FTL-strings
[2020-02-04 06:18:34.809 15857] rw------- pihole:pihole 12 FTL-settings
[2020-02-04 06:18:34.810 15857] rw------- pihole:pihole 120 FTL-counters
[2020-02-04 06:18:34.811 15857] rw------- pihole:pihole 28 FTL-lock
[2020-02-04 06:18:34.811 15857] ---------------------------------------------------
[2020-02-04 06:18:34.811 15857] Thank you for helping us to improve our FTL engine!
[2020-02-04 06:18:34.811 15857] FTL terminated!
pi@pi0:~ $ pihole -v
Pi-hole version is v4.3.2-392-g2a5cf22 (Latest: v4.3.2)
AdminLTE version is v4.3.2-378-g4958ed2 (Latest: v4.3.2)
FTL version is vDev-1af4748 (Latest: v4.3.1)
After little digging, seems like this test branch doesn't have the previous fix for pi zero... oh well... back to release for now. Will try again once it's updated.
You're right. I'm very sorry for this, it seems my local release/v5.0 branch was lacking behind. I guess this is what you get when you're commute between two flats with independent Pi-holes... Should be fixes by now!
Connected pull requests: