High CPU, High RAM, not working, dnsmasq issues? - Solved

Nighty · August 21, 2018, 8:55am

Please follow the below template, it will help us to help you!

Expected Behaviour:

Expected behaviour? That it works.
No issues pre-4.0, possibly wanting to downgrade.

Actual Behaviour:

Raspberry Pi Zero W.

Fresh Raspbian install, after it failed to update FTL from 3 to 4 on previous install.
Same result after each reinstall, I've noticed in the debug that it mentions:
[✗] dnsmasq daemon is inactive

I've tried manually installing it, no luck (after 3rd time reinstalling everything).

Results of each reinstall, and pre-reinstall is the same:
High CPU Usage. Load never really falls below 1.5 - often closer to 4.0. RAM usage never below 88+%.
Blocked domains randomly jumps between either 4 (just 4) or ~2.8 mill. Blocked percentage is off, queries not realistic. Query types lists a whole long list of things (unsure if just a new thing, ANY, SRV, SOA, PTR, TXT), and the graphics on 'Queries answered by' is off - such as writing one.one.one.one, instead of 1.1.1.1. Top Clients doesn't load.

Can hardly log in through PuTTY, as the CPU load is as high as it is.

Edit: Another reinstall, CPU usage better, RAM still at 89.4%.
Queries answered by still off.. Basically all the exact same issues as above.

Debug Token:

I didn't actually upload it to your server on the first go, and trying to run it again proves impossible, as CPU Load is now at 3+, and PuTTY no longer really does a whole lot. So here's the entire thing, not really much to hide in it anyhow:

*** [ INITIALIZING ]
[i] 2018-08-21:10:39:42 debug log has been initialized.

*** [ INITIALIZING ] Sourcing setup variables
[i] Sourcing /etc/pihole/setupVars.conf...

*** [ DIAGNOSING ]: Core version
[i] Core: v4.0 (How do I update Pi-hole?)
[i] Branch: master
[i] Commit: v4.0-0-gddbdb51

*** [ DIAGNOSING ]: Web version
[i] Web: v4.0 (How do I update Pi-hole?)
[i] Branch: master
[i] Commit: v4.0-0-gaf8c926

*** [ DIAGNOSING ]: FTL version
[✓] FTL: v4.0

*** [ DIAGNOSING ]: dnsmasq version
[i] 2.76

*** [ DIAGNOSING ]: lighttpd version
[i] 1.4.45

*** [ DIAGNOSING ]: php version
[i] 7.0.30

*** [ DIAGNOSING ]: Operating system
[✓] Raspbian GNU/Linux 9 (stretch)

*** [ DIAGNOSING ]: SELinux
[i] SELinux not detected

*** [ DIAGNOSING ]: Processor
[✓] armv6l

*** [ DIAGNOSING ]: Networking
[✓] IPv4 address(es) bound to the wlan0 interface:
192.168.xx.xx/24 matches the IP found in /etc/pihole/setupVars.conf

[✓] IPv6 address(es) bound to the wlan0 interface:
fe80::aeb2:3392:6391:b480 does not match the IP found in /etc/pihole/setupVars.conf (Use IPv6 ULA addresses for Pi-hole)

^ Please note that you may have more than one IP address listed.
As long as one of them is green, and it matches what is in /etc/pihole/setupVars.conf, there is no need for concern.

The link to the FAQ is for an issue that sometimes occurs when the IPv6 address changes, which is why we check for it.

[i] Default IPv4 gateway: 192.168.xx.xx

Pinging 192.168.xx.xx...
[✓] Gateway responded.

*** [ DIAGNOSING ]: Ports in use
*:22 sshd (IPv4)
*:22 sshd (IPv6)
*:80 lighttpd (IPv4)
*:80 lighttpd (IPv6)
*:53 pihole-FTL (IPv4)
*:53 pihole-FTL (IPv6)
127.0.0.1:4711 pihole-FTL (IPv4)
[::1]:4711 pihole-FTL (IPv6)

*** [ DIAGNOSING ]: Name resolution (IPv4) using a random blocked domain and a known ad-serving domain
[✓] adserver.aol.advertising.com is 0.0.0.0 via localhost (127.0.0.1)
[✓] adserver.aol.advertising.com is 0.0.0.0 via Pi-hole (192.168.xx.xx)
[✓] doubleclick.com is 216.58.208.46 via a remote, public DNS server (8.8.8.8)

*** [ DIAGNOSING ]: Pi-hole processes
[✗] dnsmasq daemon is inactive
[✓] lighttpd daemon is active
[✓] pihole-FTL daemon is active

*** [ DIAGNOSING ]: Setup variables
PIHOLE_INTERFACE=wlan0
IPV4_ADDRESS=192.168.xx.xx/24
IPV6_ADDRESS=
PIHOLE_DNS_1=1.1.1.1
PIHOLE_DNS_2=1.0.0.1
QUERY_LOGGING=true
INSTALL_WEB_SERVER=true
INSTALL_WEB_INTERFACE=true
LIGHTTPD_ENABLED=true

*** [ DIAGNOSING ]: Dashboard and block page
[✗] Block page X-Header: X-Header does not match or could not be retrieved.
HTTP/1.1 200 OK
Content-type: text/html; charset=UTF-8
Date: Tue, 21 Aug 2018 08:40:27 GMT
Server: lighttpd/1.4.45

[✓] Web interface X-Header: X-Pi-hole: The Pi-hole Web interface is working!

*** [ DIAGNOSING ]: Gravity list
-rw-r--r-- 1 root root 66489913 Aug 21 10:34 /etc/pihole/gravity.list
-----head of gravity.list------
--little--princess--.tumblr.com
-allporn-.tumblr.com
-becca-anal-.tumblr.com
-celestial-beings-.tumblr.com

-----tail of gravity.list------
zzzz2gbht6.info
zzzzz4.52896368.com
zzzzzz.com
ɢoogle.com

*** [ DIAGNOSING ]: contents of /etc/pihole

-rw-r--r-- 1 root root 43 Aug 21 10:34 /etc/pihole/local.list
192.168.xx.xx pihole
192.168.xx.xx pi.hole

-rw-r--r-- 1 root root 234 Aug 21 10:28 /etc/pihole/logrotate
/var/log/pihole.log {
su root root
daily
copytruncate
rotate 5
compress
delaycompress
notifempty
nomail
}
/var/log/pihole-FTL.log {
su root root
weekly
copytruncate
rotate 3
compress
delaycompress
notifempty
nomail
}

*** [ DIAGNOSING ]: contents of /etc/dnsmasq.d

-rw-r--r-- 1 root root 1509 Aug 21 10:29 /etc/dnsmasq.d/01-pihole.conf
addn-hosts=/etc/pihole/gravity.list
addn-hosts=/etc/pihole/black.list
addn-hosts=/etc/pihole/local.list
localise-queries
no-resolv
cache-size=10000
log-queries=extra
log-facility=/var/log/pihole.log
local-ttl=2
log-async
server=1.1.1.1
server=1.0.0.1
interface=wlan0

*** [ DIAGNOSING ]: contents of /etc/lighttpd

-rw-r--r-- 1 root root 3027 Aug 21 10:28 /etc/lighttpd/lighttpd.conf
server.modules = (
"mod_access",
"mod_accesslog",
"mod_auth",
"mod_expire",
"mod_compress",
"mod_redirect",
"mod_setenv",
"mod_rewrite"
)
server.document-root = "/var/www/html"
server.error-handler-404 = "pihole/index.php"
server.upload-dirs = ( "/var/cache/lighttpd/uploads" )
server.errorlog = "/var/log/lighttpd/error.log"
server.pid-file = "/var/run/lighttpd.pid"
server.username = "www-data"
server.groupname = "www-data"
server.port = 80
accesslog.filename = "/var/log/lighttpd/access.log"
accesslog.format = "%{%s}t|%V|%r|%s|%b"
index-file.names = ( "index.php", "index.html", "index.lighttpd.html" )
url.access-deny = ( "~", ".inc", ".md", ".yml", ".ini" )
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
compress.cache-dir = "/var/cache/lighttpd/compress/"
compress.filetype = ( "application/javascript", "text/css", "text/html", "text/plain" )
include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port
include_shell "/usr/share/lighttpd/create-mime.assign.pl"
include_shell "find /etc/lighttpd/conf-enabled -name '*.conf' -a ! -name 'letsencrypt.conf' -printf 'include "%p"
' 2>/dev/null"
$HTTP["url"] =~ "^/admin/" {

   setenv.add-response-header = (
       "X-Pi-hole" => "The Pi-hole Web interface is working!",
       "X-Frame-Options" => "DENY"
   )
   $HTTP["url"] =~ ".ttf$" {

       setenv.add-response-header = ( "Access-Control-Allow-Origin" => "*" )
   }

}
$HTTP["url"] =~ "^/admin/.(.*)" {
url.access-deny = ("")
}
include_shell "cat external.conf 2>/dev/null"

*** [ DIAGNOSING ]: contents of /etc/cron.d

-rw-r--r-- 1 root root 1495 Aug 21 10:28 /etc/cron.d/pihole
37 3 * * 7 root PATH="$PATH:/usr/local/bin/" pihole updateGravity
00 00 * * * root PATH="$PATH:/usr/local/bin/" pihole flush once quiet
@reboot root /usr/sbin/logrotate /etc/pihole/logrotate
*/10 * * * * root PATH="$PATH:/usr/local/bin/" pihole updatechecker local
3 19 * * * root PATH="$PATH:/usr/local/bin/" pihole updatechecker remote
@reboot root PATH="$PATH:/usr/local/bin/" pihole updatechecker remote reboot

*** [ DIAGNOSING ]: contents of /var/log/lighttpd

-rw-r--r-- 1 www-data www-data 3220 Aug 21 10:40 /var/log/lighttpd/error.log
2018-08-21 10:13:02: (log.c.217) server started
2018-08-21 10:16:34: (server.c.1828) server stopped by UID = 0 PID = 1
2018-08-21 10:16:35: (log.c.217) server started
2018-08-21 10:18:37: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: file_get_contents(/var/run/pihole-FTL.port): failed to open stream: No such file or directory in /var/www/html/admin/scripts/pi-hole/php/FTL.php on line 14
2018-08-21 10:18:48: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: file_get_contents(/var/run/pihole-FTL.port): failed to open stream: No such file or directory in /var/www/html/admin/scripts/pi-hole/php/FTL.php on line 14
2018-08-21 10:18:59: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: file_get_contents(/var/run/pihole-FTL.port): failed to open stream: No such file or directory in /var/www/html/admin/scripts/pi-hole/php/FTL.php on line 14
2018-08-21 10:19:10: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: file_get_contents(/var/run/pihole-FTL.port): failed to open stream: No such file or directory in /var/www/html/admin/scripts/pi-hole/php/FTL.php on line 14
2018-08-21 10:19:21: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: file_get_contents(/var/run/pihole-FTL.port): failed to open stream: No such file or directory in /var/www/html/admin/scripts/pi-hole/php/FTL.php on line 14
2018-08-21 10:19:32: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: file_get_contents(/var/run/pihole-FTL.port): failed to open stream: No such file or directory in /var/www/html/admin/scripts/pi-hole/php/FTL.php on line 14
2018-08-21 10:19:43: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: file_get_contents(/var/run/pihole-FTL.port): failed to open stream: No such file or directory in /var/www/html/admin/scripts/pi-hole/php/FTL.php on line 14
2018-08-21 10:23:53: (server.c.1828) server stopped by UID = 0 PID = 1
2018-08-21 10:24:30: (log.c.217) server started
2018-08-21 10:29:03: (server.c.1828) server stopped by UID = 0 PID = 1
2018-08-21 10:29:05: (log.c.217) server started
2018-08-21 10:40:20: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: file(http://127.0.0.1/admin/scripts/pi-hole/php/queryads.php?domain=localhost&bp): failed to open stream: HTTP request failed! in /var/www/html/pihole/index.php on line 135
2018-08-21 10:40:20: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: array_filter() expects parameter 1 to be array, string given in /var/www/html/pihole/index.php on line 136
2018-08-21 10:40:20: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: array_values() expects parameter 1 to be array, null given in /var/www/html/pihole/index.php on line 136
2018-08-21 10:40:27: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: file(http://127.0.0.1/admin/scripts/pi-hole/php/queryads.php?domain=localhost&bp): failed to open stream: HTTP request failed! in /var/www/html/pihole/index.php on line 135
2018-08-21 10:40:27: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: array_filter() expects parameter 1 to be array, string given in /var/www/html/pihole/index.php on line 136
2018-08-21 10:40:27: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Warning: array_values() expects parameter 1 to be array, null given in /var/www/html/pihole/index.php on line 136

*** [ DIAGNOSING ]: contents of /var/log

-rw-r--r-- 1 pihole pihole 3922 Aug 21 10:35 /var/log/pihole-FTL.log
-----head of pihole-FTL.log------
[2018-08-21 10:29:11.211] ########## FTL started! ##########
[2018-08-21 10:29:11.212] FTL branch:
[2018-08-21 10:29:11.212] FTL version: v4.0
[2018-08-21 10:29:11.212] FTL commit: 8493df4
[2018-08-21 10:29:11.212] FTL date: 2018-08-05 13:40:30 -0700
[2018-08-21 10:29:11.212] FTL user: pihole
[2018-08-21 10:29:11.213] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
[2018-08-21 10:29:11.213] SOCKET_LISTENING: only local
[2018-08-21 10:29:11.213] AAAA_QUERY_ANALYSIS: Show AAAA queries
[2018-08-21 10:29:11.213] MAXDBDAYS: max age for stored queries is 365 days
[2018-08-21 10:29:11.213] RESOLVE_IPV6: Resolve IPv6 addresses
[2018-08-21 10:29:11.214] RESOLVE_IPV4: Resolve IPv4 addresses
[2018-08-21 10:29:11.214] DBINTERVAL: saving to DB file every minute
[2018-08-21 10:29:11.214] DBFILE: Using /etc/pihole/pihole-FTL.db
[2018-08-21 10:29:11.214] MAXLOGAGE: Importing up to 24.0 hours of log data
[2018-08-21 10:29:11.215] PRIVACYLEVEL: Set to 0
[2018-08-21 10:29:11.215] IGNORE_LOCALHOST: Show queries from localhost
[2018-08-21 10:29:11.215] BLOCKINGMODE: Null IPs for blocked domains
[2018-08-21 10:29:11.215] REGEX_DEBUGMODE: Inactive
[2018-08-21 10:29:11.216] Finished config file parsing
[2018-08-21 10:29:11.216] INFO: No whitelist file found
[2018-08-21 10:29:11.216] Compiled 0 Regex filters and -1 whitelisted domains in 0.4 msec (0 errors)
[2018-08-21 10:29:11.217] db_init() - Cannot open database (14): unable to open database file
[2018-08-21 10:29:11.218] Creating new (empty) database
[2018-08-21 10:29:11.603] Database successfully initialized
[2018-08-21 10:29:11.605] Imported 0 queries from the long-term database
[2018-08-21 10:29:11.606] -> Total DNS queries: 0
[2018-08-21 10:29:11.606] -> Cached DNS queries: 0
[2018-08-21 10:29:11.606] -> Forwarded DNS queries: 0
[2018-08-21 10:29:11.607] -> Exactly blocked DNS queries: 0
[2018-08-21 10:29:11.607] -> Unknown DNS queries: 0
[2018-08-21 10:29:11.607] -> Unique domains: 0
[2018-08-21 10:29:11.607] -> Unique clients: 0
[2018-08-21 10:29:11.607] -> Known forward destinations: 0
[2018-08-21 10:29:11.607] Successfully accessed setupVars.conf

-----tail of pihole-FTL.log------
[2018-08-21 10:29:11.217] db_init() - Cannot open database (14): unable to open database file
[2018-08-21 10:29:11.218] Creating new (empty) database
[2018-08-21 10:29:11.603] Database successfully initialized
[2018-08-21 10:29:11.605] Imported 0 queries from the long-term database
[2018-08-21 10:29:11.606] -> Total DNS queries: 0
[2018-08-21 10:29:11.606] -> Cached DNS queries: 0
[2018-08-21 10:29:11.606] -> Forwarded DNS queries: 0
[2018-08-21 10:29:11.607] -> Exactly blocked DNS queries: 0
[2018-08-21 10:29:11.607] -> Unknown DNS queries: 0
[2018-08-21 10:29:11.607] -> Unique domains: 0
[2018-08-21 10:29:11.607] -> Unique clients: 0
[2018-08-21 10:29:11.607] -> Known forward destinations: 0
[2018-08-21 10:29:11.607] Successfully accessed setupVars.conf
[2018-08-21 10:29:11.635] PID of FTL process: 1353
[2018-08-21 10:29:11.636] Listening on port 4711 for incoming IPv4 telnet connections
[2018-08-21 10:29:11.637] Listening on port 4711 for incoming IPv6 telnet connections
[2018-08-21 10:29:11.638] Listening on Unix socket
[2018-08-21 10:29:11.640] FATAL: Trying to free NULL pointer in free_whitelist_domains() (regex.c:72)
[2018-08-21 10:29:11.641] INFO: No whitelist file found
[2018-08-21 10:29:11.641] Compiled 0 Regex filters and -1 whitelisted domains in 0.3 msec (0 errors)
[2018-08-21 10:29:16.121] Notice: Increasing queries struct size from 0 to 10000
[2018-08-21 10:29:16.122] Notice: Increasing overTime struct size from 0 to 100
[2018-08-21 10:29:16.122] Notice: Increasing domains struct size from 0 to 1000
[2018-08-21 10:29:16.122] Notice: Increasing clients struct size from 0 to 10
[2018-08-21 10:29:16.124] New forward server: 1.0.0.1 (0/0)
[2018-08-21 10:29:16.124] Notice: Increasing forwarded struct size from 0 to 4
[2018-08-21 10:29:16.153] New forward server: 1.1.1.1 (1/4)
[2018-08-21 10:29:16.533] FATAL: Trying to free NULL pointer in free_whitelist_domains() (regex.c:72)
[2018-08-21 10:29:16.534] INFO: No whitelist file found
[2018-08-21 10:29:16.534] Compiled 0 Regex filters and -1 whitelisted domains in 0.5 msec (0 errors)
[2018-08-21 10:29:16.536] /etc/pihole/gravity.list: parsed 0 domains (took 0.0 ms)
[2018-08-21 10:34:10.879] FATAL: Trying to free NULL pointer in free_whitelist_domains() (regex.c:72)
[2018-08-21 10:34:10.894] INFO: No whitelist file found
[2018-08-21 10:34:10.894] Compiled 0 Regex filters and -1 whitelisted domains in 0.5 msec (0 errors)
[2018-08-21 10:35:31.421] /etc/pihole/gravity.list: parsed 2756366 domains (took 80524.2 ms)

*** [ DIAGNOSING ]: Locale
LANG=en_GB.UTF-8

*** [ DIAGNOSING ]: Pi-hole log
-rw-r--r-- 1 pihole pihole 48198 Aug 21 10:40 /var/log/pihole.log
-----head of pihole.log------
Aug 21 10:29:11 dnsmasq[1353]: started, version pi-hole-2.79 cachesize 10000
Aug 21 10:29:11 dnsmasq[1353]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify
Aug 21 10:29:11 dnsmasq[1353]: warning: failed to change owner of /var/log/pihole.log: Operation not permitted
Aug 21 10:29:11 dnsmasq[1353]: using nameserver 1.0.0.1#53
Aug 21 10:29:11 dnsmasq[1353]: using nameserver 1.1.1.1#53
Aug 21 10:29:11 dnsmasq[1353]: read /etc/hosts - 5 addresses
Aug 21 10:29:11 dnsmasq[1353]: failed to load names from /etc/pihole/local.list: No such file or directory
Aug 21 10:29:11 dnsmasq[1353]: failed to load names from /etc/pihole/black.list: No such file or directory
Aug 21 10:29:11 dnsmasq[1353]: failed to load names from /etc/pihole/gravity.list: No such file or directory
Aug 21 10:29:16 dnsmasq[1353]: 1 127.0.0.1/35902 query[AAAA] raw.githubusercontent.com from 127.0.0.1
Aug 21 10:29:16 dnsmasq[1353]: 1 127.0.0.1/35902 forwarded raw.githubusercontent.com to 1.0.0.1
Aug 21 10:29:16 dnsmasq[1353]: 1 127.0.0.1/35902 forwarded raw.githubusercontent.com to 1.1.1.1
Aug 21 10:29:16 dnsmasq[1353]: 1 127.0.0.1/35902 reply raw.githubusercontent.com is
Aug 21 10:29:16 dnsmasq[1353]: 1 127.0.0.1/35902 reply github.map.fastly.net is NODATA-IPv6
Aug 21 10:29:16 dnsmasq[1353]: 2 127.0.0.1/58341 query[A] raw.githubusercontent.com from 127.0.0.1
Aug 21 10:29:16 dnsmasq[1353]: 2 127.0.0.1/58341 cached raw.githubusercontent.com is
Aug 21 10:29:16 dnsmasq[1353]: 2 127.0.0.1/58341 forwarded raw.githubusercontent.com to 1.1.1.1
Aug 21 10:29:16 dnsmasq[1353]: 2 127.0.0.1/58341 reply raw.githubusercontent.com is
Aug 21 10:29:16 dnsmasq[1353]: 2 127.0.0.1/58341 reply github.map.fastly.net is 151.101.84.133
Aug 21 10:29:16 dnsmasq[1353]: read /etc/hosts - 5 addresses

[✓] ** FINISHED DEBUGGING! **

DL6ER · August 21, 2018, 10:24am

Note that dnsmasq is not used with Pi-hole v4.0. The debugger needs to be adjusted to have this text you saw removed (pinging @jacob.salmela).
Further, dnsmasq is replaced by our own daemon, pihole-FTL, which includes the most recent version of dnsmasq.

Something suspicious in your debug log is:

Apparently, pihole-FTL took 80 seconds (!) to read in the list of blocked domains. This may be an issue of the almost 3 million domains. Please try reducing the number of imported lists first. On a Raspberry Pi 3B, I wouldn't expect even 2.7 mio. queries to take so long during import. For a Zero, it might very well be that you hit the limits of this device.

I'm using the standard lists (128,674 domains) and here the import takes about 0.5 seconds. Extrapolating this, the import of 3 mio. queries should not take longer than maybe 15 seconds - clearly not 80... but then the Pi 3B is much more powerful than a Zero W, so you may habe to upgrade if you need those almost 3mio. Blocking entries. It may also be an issue with your SD card (reading speed too low), but that would not explain the high RAM usage. How much RAM does the Zero W have?

Question 1: As said, please try first with reducing the sourced domains to something clearly less than 1 mio. so that we can troubleshoot if it is due to the large amount of lists. Does it improve the overall situation?

Question 2: What is exactly using the high amounts of CPU / RAM (which process?)

Nighty · August 21, 2018, 11:10am

That does indeed look like an issue.
I wonder if that's been the cause each time I've done a reinstall (think I went through 3, with the same results, SD card was flashed between each attempt).

I'll try throwing out some lists and get back to you.

Answer 1:
I'll get back to you about this.

Answer 2:
It's been moving a bit up and down over the last hour, saw it at 0.5 for a bit, but I believe it increased as I changed pages within the interface.

Edit: Mind you, I'm a novice when it comes to Linux based systems, pretty much just recently I've gotten into it, due to the Pi's and my NAS('es). Pretty avid computer user though.

Nighty · August 21, 2018, 11:21am

I've taken off quite a few lists, down to ~600k now, these are the results after about 10 minutes or so -
RAM usage remains high, CPU usage looks better however, currently at ~0.15-0.3 (though closer to 0.0-0.1 with pre-4.0).
Mind you, I had no issues with 3 mill blocks pre-4.0, no performance issues or what not. I suppose it could be the SD could be the issue, is there a linux benchmark I could try, for checking this?

Also, I've got to add that I've currently no devices running on it (but my one NAS, which has internet access blocked, and doesn't really do anything, sleeping atm), so it is basically just idling.

DL6ER · August 21, 2018, 12:03pm

Hmm, can you also post the output of

echo ">cacheinfo" | nc pi.hole 4711

?

It seems that the CPU usage is very low on your screenshot, pihole-FTL seems to use about 0.6%. The previously high load was caused by pihole-FTL running at 100% while loading the lists so not unexpected. I will nevertheless check back with the team if they are aware of such limitations.

One other thing that came to my mind: Do you have IPv6 connectivity (test it, e.g., here)? If not, then this would explain a doubling in memory usage that can be fixed by setting the blocking mode to NXDOMAIN. See Blocking mode - Pi-hole documentation

Nighty · August 21, 2018, 12:14pm

Yeah, my apologies, I keep getting the 'load' and CPU usage mixed up on this Linux thing.
The load is high, the CPU is not, as according to 'htop'.

I do not currently have IPv6 connectivity, and didn't pick it during install (though I did have it enabled pre-4.0). The website linked also states 'No IPv6 address detected'.

Copypasting in the echo above seems to do nothing, but as said, Linux novice, I might possibly be doing this wrong.
I did find a site talking about the NXDOMAIN thing while trying to troubleshoot before posting here, I didn't quite manage to get it done though, will try again and get back to you.
Thanks for the help thus far,

Edit:
If I'm doing the NXDOMAIN thing correct ('sudo nano /etc/pihole/pihole-FTL.conf'?), then there's nothing in this file. Should I just add 'BLOCKINGMODE=NXDOMAIN' to it, save, reboot the device?

DL6ER · August 21, 2018, 12:19pm

Yes, you can also run sudo pihole restartdns . A full reboot is not required.

No worries, the load is more a long-term thing (how much work had to be done recently) while the CPU% is a momentary number. They are both meaningful but it takes some time for the load to settle down after there was intense activity on a machine.

We just reproduced your installation (with that many domains) and can confirm a rather high memory utilization. 3mio. domains are pushing devices like Raspberry Pis to their limits. It should be improved with NXDOMAIN.

Nighty · August 21, 2018, 12:54pm

It felt like it needed a full restart anyhow

After changing the (empty) file, and rebooting, I'm now seeing load numbers that still bounces around a bit, while Memory Usage is down to 19.3% (with a few more lists added, now up on 631k domains.
What does the NXDOMAIN actually do, mainly in concerns of privacy?

Also, in case of more testing for you guys, then the Pre-4.0 ran the approx. 3mill domains just fine, with a fine memory and load usage (load was usually, as mentioned before, near 0).

In either case, seems this has fixed the issue for now, so I'll again thank you for your help and assistance.

I'll slowly try to add more domains again, and see how it turns out.
Alternatively I suppose I could just take a look at what actually goes on, and just individually blacklist what I do not like seeing - which might really be the smarter approach (not to mention faster), as compared to just blocking what looks to be about 1 million random porn sites (judging by the list called "top1m.list").

Edit:
I do however still have 'issues' with this not looking just right, is that just me, or is it something I'm missing?

DL6ER · August 21, 2018, 1:04pm

It doesn't change anything concerning privacy. With the default blocking mode (NULL), we store each domain twice in the DNS cache as it is per domains and per query type (one A and one AAAA record). For NXDOMAIN, we store each domain only once as we store a general "this domain does not exist". It is expected that this uses exactly half the memory it uses for the other blocking modes. Why you're seeing a fourfold reduction (from around 80% to around 20%) is something I cannot explain at the moment.

The load should settle down close to 0 at some point.

I'm personally perfectly fine with the standard set of lists (blocking about 120,000 domains) but I'm also a rather conservative Internet user not looking much off the main roads.

DL6ER · August 21, 2018, 1:06pm

What exactly is off? That this one table is not loading or something else?

Concerning the one.one.one.one two times, this is expected as they were strange enough to give both IPs the same hostname...

$ dig -x 1.1.1.1 +short
one.one.one.one.

$ dig -x 1.0.0.1 +short
one.one.one.one.

Nighty · August 21, 2018, 1:24pm

What exactly is off? That this one table is not loading or something else?

It seems like the "Top Clients" has fixed itself, and does indeed show now.
And, indeed. Was mainly the 1.1.1.1 thing, and probably just me not being used to the other changes just yet (such as 'Blocklist', 'cache', and all the ANY, SRV, SOA [...] things on the Query Types.

I’m personally perfectly fine with the standard set of lists (blocking about 120,000 domains) but I’m also a rather conservative Internet user not looking much off the main roads.

As for blocking "all the things".. I suppose I'm just a bit paranoid like that at times. Looking at what it blocks, and what my devices actually tries to access.. It rarely comes across any of those domains, if ever.
Having re-enabled all but 4 lists I'm now up to ~640k blocks. So the remaining lists might be quite.. hefty.

Why you’re seeing a fourfold reduction (from around 80% to around 20%) is something I cannot explain at the moment.

Well, there's something for you guys to have fun with figuring out!

Thanks again, I appreciate all the help, and the software.

DL6ER · August 21, 2018, 1:55pm

Oh yes, those are new with Pi-hole v4.0

They may still be handled fine by your Pi-hole but maybe you should consider upgrading to a device with more RAM for stable operation.

Nighty · August 21, 2018, 2:16pm

They may still be handled fine by your Pi-hole but maybe you should consider upgrading to a device with more RAM for stable operation.

Oh, I purchased a Pi3 B+ a couple of weeks ago, as I kind of liked this Pi Zero W. My idea was for it to run Retropie, Ubiquiti controller, and probably some VPN stuff at some point. Unsure if I'd want to throw Pi-Hole on top of it though, don't want to push that device to a halt.
Is why I got the Pi Zero just for this, and nothing else running on it -- though I was thinking of trying to add Ubiquiti's Controller software to it, but dunno if that'd "kill" it. Seeing the memory being at 80% earlier kind of made me not do it.

DL6ER · August 21, 2018, 2:21pm

With NXDOMAIN your memory should not go up again, so you should be safe.

I think I figured it out by now and it is expected: IPv6 addresses are 4x as long as IPv4 addresses and since we skip the IPv6 cache entries now, we're over-proportionally saving memory here.

Nighty · August 21, 2018, 2:27pm

Oh, a final question if I may,

As I manually installed dnsmasq, as a way to try and sort out things I had no clue about.. Do you suggest me uninstalling it again? I saw that it couldn't start due to port 53 being used (by what I'm guessing is the new FTL).

I think I figured it out by now and it is expected: IPv6 addresses are 4x as long as IPv4 addresses and since we skip the IPv6 cache entries now, we’re over-proportionally saving memory here.

That would indeed make sense, and with 3 million blocked domains.. stored twice was it(?), yeah.

DL6ER · August 21, 2018, 2:39pm

You can do this. FTL brings dnsmasq inbuilt, so anything you wanted to try with dnsmasq may also be achieved with FTL.

Nighty · August 22, 2018, 3:19pm

I'll just hijack my own thread here,

I decided to move everything to my Pi3B+, so the Zero W is currently my play testing machine, before moving things to the Pi3 - -
I was trying to add my NAS (for storage) to the Pi, through SMB. Eventually after looking through things I figured out that while I can ping my NAS, I can't actually connect to it - And I think the NXDOMAIN might be the issue(?)

pi@pihole:~ $ nslookup 192.168.xx.xx
Server: 127.0.0.1
Address: 127.0.0.1#53

** server can't find xx.xx.168.192.in-addr.arpa: NXDOMAIN

I notice that the IP is reversed, and doing nslookup xx.xx.168.192 kind of works (can't find by domain though).
This isn't quite solving my issues of connecting to the SMB though, but that I believe is a topic for somewhere else - So, I just wanted to ask if the NXDOMAIN business could be the cause of it?

Thanks,

DL6ER · August 22, 2018, 3:25pm

No, that's unlikely. The NXDOMAIN that is returned here is the correct: it does not know the answer. Do you use Pi-hole's DHCP server? If not (and the NAS uses a static IP), you will have to create an entry in your /etc/hosts for it

system · September 12, 2018, 3:25pm

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.