I recently noticed on my Raspberry 1 B+ device with the "top" command that the "pihole-FTL" process is using above 98% CPU, and when running "pihole -c" even above 100%.
The output of "pihole -c" even shows a line containing
Blocked: FTL offline
Firing the command "sudo systemctl status pihole-FTL" shows:
● pihole-FTL.service - LSB: pihole-FTL daemon
Loaded: loaded (/etc/init.d/pihole-FTL; generated; vendor preset: enabled)
Active: active (exited) since Fri 2018-06-08 21:18:16 CEST; 14min ago
Docs: man:systemd-sysv-generator(8)
Process: 2680 ExecStop=/etc/init.d/pihole-FTL stop (code=exited, status=1/FAILURE)
Process: 2730 ExecStart=/etc/init.d/pihole-FTL start (code=exited, status=0/SUCCESS)
Jun 08 21:18:15 dnsbox systemd[1]: Starting LSB: pihole-FTL daemon...
Jun 08 21:18:15 dnsbox pihole-FTL[2730]: Not running
Jun 08 21:18:15 dnsbox pihole-FTL[2730]: rm: cannot remove '/var/run/pihole/FTL.sock': No such file or directory
Jun 08 21:18:15 dnsbox su[2744]: Successful su for pihole by root
Jun 08 21:18:15 dnsbox su[2744]: + ??? root:pihole
Jun 08 21:18:15 dnsbox su[2744]: pam_unix(su:session): session opened for user pihole by (uid=0)
Jun 08 21:18:16 dnsbox pihole-FTL[2730]: FTL started!
Jun 08 21:18:16 dnsbox systemd[1]: Started LSB: pihole-FTL daemon.
Even a manual restart of the FTL process, does not resolve the situation
Using the web browser, no DNS statistics are being displayed, am getting a pop message with "An error occurred while loading the data: Connection refused. Is FTL running?".
According to your debug log, you already have close to one million queries in the log and your /var/log/pihole.log file is about 300 MB in file size.
Although Pi-hole can run on low-performance hardware, there are certain limitations to this. Maybe your FTL daemon is/was still busy with parsing the history information or the initial log file parsing: traversing up to a million DNS requests will take very long on a Raspberry Pi B+.
How many clients does this Raspberry Pi serve?
If you're expecting a daily traffic of several hundreds of thousands of queries, you should consider upgrading to a beefer hardware so that Pi-hole can keep up with computing all the statistics.
If everything looks fine, then you may need to investigate why the pihole.log file is so large, does the nightly flushing work as expected?
Following yesterday's situation, I manually shut down the "pihole-FTL" process, and restarted it again this evening. Here the corresponding entries of the "/var/log/pihole-FTL.log" file:
[2018-06-09 19:52:47.257] ########## FTL started! ##########
[2018-06-09 19:52:47.257] FTL branch: v3.0
[2018-06-09 19:52:47.257] FTL version: v3.0
[2018-06-09 19:52:47.257] FTL commit: a1b2fa03f8f3e3713b57a26a4e9a062fec0573d7
[2018-06-09 19:52:47.258] FTL date: 2018-02-14 12:45:47 -0800
[2018-06-09 19:52:47.258] FTL user: pihole
[2018-06-09 19:52:47.258] Notice: Found no readable FTL config file
[2018-06-09 19:52:47.258] Using default settings
[2018-06-09 19:52:47.258] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
[2018-06-09 19:52:47.259] SOCKET_LISTENING: only local
[2018-06-09 19:52:47.259] QUERY_DISPLAY: Show queries
[2018-06-09 19:52:47.259] AAAA_QUERY_ANALYSIS: Show AAAA queries
[2018-06-09 19:52:47.259] MAXDBDAYS: max age for stored queries is 365 days
[2018-06-09 19:52:47.259] RESOLVE_IPV6: Resolve IPv6 addresses
[2018-06-09 19:52:47.259] RESOLVE_IPV4: Resolve IPv4 addresses
[2018-06-09 19:52:47.260] DBINTERVAL: saving to DB file every minute
[2018-06-09 19:52:47.260] DBFILE: Using /etc/pihole/pihole-FTL.db
[2018-06-09 19:52:47.260] MAXLOGAGE: Importing up to 24.0 hours of log data
[2018-06-09 19:52:47.260] Finished config file parsing
[2018-06-09 19:52:47.279] Found no other running pihole-FTL process
[2018-06-09 19:52:47.300] PID of FTL process: 15121
[2018-06-09 19:52:57.924] Gravity list entries: 2831130
[2018-06-09 19:52:57.924] Blacklist entries: 1
[2018-06-09 19:52:57.924] No wildcard blocking list present
[2018-06-09 19:52:57.946] Database initialized
[2018-06-09 19:52:57.954] Imported 0 queries from the long-term database
[2018-06-09 19:52:57.954] Starting initial log file parsing
[2018-06-09 19:52:57.954] Reading from /var/log/pihole.log (rw-r--r--)
[2018-06-09 19:52:57.956] Notice: Increasing queries struct size from 0 to 10000 (440.12 KB)
[2018-06-09 19:52:57.956] Notice: Increasing overTime struct size from 0 to 100 (444.12 KB)
[2018-06-09 19:52:57.957] Notice: Increasing domains struct size from 0 to 1000 (464.12 KB)
[2018-06-09 19:52:57.957] Notice: Increasing clients struct size from 0 to 10 (464.30 KB)
[2018-06-09 19:52:57.959] New client: 127.0.0.1 localhost (0/10)
[2018-06-09 19:52:57.959] Notice: Increasing forwarded struct size from 0 to 4 (464.39 KB)
[2018-06-09 19:52:58.389] New forward server: 208.67.220.123 resolver2-fs.opendns.com (0/4)
[2018-06-09 19:52:58.806] New forward server: 208.67.222.123 resolver1-fs.opendns.com (1/4)
[2018-06-09 19:52:59.226] New client: 192.168.0.20 (1/10)
[2018-06-09 19:52:59.771] New client: 192.168.0.21 (2/10)
[2018-06-09 19:53:00.296] New client: 192.168.0.28 (3/10)
[2018-06-09 19:53:02.847] New client: 192.168.0.23 (4/10)
[2018-06-09 19:53:04.900] New client: 192.168.0.50 (5/10)
[2018-06-09 19:53:09.024] New client: 192.168.0.43 (6/10)
[2018-06-09 19:53:42.750] Notice: Increasing queries struct size from 10000 to 20000 (923.30 KB)
[2018-06-09 19:54:06.157] Finished initial log file parsing
[2018-06-09 19:54:06.158] -> Total DNS queries: 12684
[2018-06-09 19:54:06.160] -> Cached DNS queries: 6913
[2018-06-09 19:54:06.161] -> Forwarded DNS queries: 1956
[2018-06-09 19:54:06.162] -> Exactly blocked DNS queries: 3815
[2018-06-09 19:54:06.163] -> Wildcard blocked DNS queries: 0
[2018-06-09 19:54:06.164] -> Unknown DNS queries: 0
[2018-06-09 19:54:06.166] -> Unique domains: 787
[2018-06-09 19:54:06.167] -> Unique clients: 7
[2018-06-09 19:54:06.168] -> Known forward destinations: 2
[2018-06-09 19:54:06.169] Successfully accessed setupVars.conf
[2018-06-09 19:54:06.171] Listening on port 4711 for incoming IPv4 telnet connections
[2018-06-09 19:54:06.173] Listening on Unix socket
And from above log file entries, it seems to be running fine now. I can even connect to the web portal and read the statistics.
Coming back to your original question on how many clients the PiHole is serving with DNS queries:
During business hours of a working week, max. 2 clients.
During the evening, max. 4 clients.
During a weekend, max. 7 clients
From my perception, the usage is fair decent. Thus, an upgrade towards better hardware does not seem to be justified here. Something else might have happened the day(s) before...
With the "top" command I see that the overall CPU utilisation of all processes is always below 5%, even the "pihole-FTL" process does not exceed 3%.
Using the "pihole -c" command, I see that the CPU utilisation is always 8% as outlined with:
|¯¯¯(¯)_|¯|_ ___|¯|___ Core: v3.3.1
| ¯_/¯|_| ' \/ _ \ / -_) Web: v3.3
|_| |_| |_||_\___/_\___| FTL: v3.0
——————————————————————————————————————————————————————————
Hostname: dnsbox (Raspberry Pi 1, Model B+)
Uptime: 23:51:35
Task Load: 0.18 0.16 0.39 (Active: 2 of 39 tasks)
CPU usage: 8% (900 MHz @ 309k)
RAM usage: 51% (Used: 244 MB of 481 MB)
HDD usage: 8% (Used: 2 GB of 29 GB)
LAN addr: 192.168.0.11 (Gateway: 192.168.0.254)
Pi-hole: Active (Blocking: 2831139 sites)
Ads Today: 36% (Total: 6319 of 17421)
Local Qrys: 64% (2 DNS servers)
I will closely monitor the PiHole platform now, and check for the nightly flushing...
During the start-up process, I'm observing the following log entries in the file "/var/log/pihole-FTL.log*":
[2018-06-14 19:57:49.548] ########## FTL started! ##########
[2018-06-14 19:57:49.548] FTL branch: v3.0
[2018-06-14 19:57:49.548] FTL version: v3.0
[2018-06-14 19:57:49.548] FTL commit: a1b2fa03f8f3e3713b57a26a4e9a062fec0573d7
[2018-06-14 19:57:49.548] FTL date: 2018-02-14 12:45:47 -0800
[2018-06-14 19:57:49.549] FTL user: pihole
[2018-06-14 19:57:49.549] Notice: Found no readable FTL config file
[2018-06-14 19:57:49.549] Using default settings
[2018-06-14 19:57:49.549] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
[2018-06-14 19:57:49.550] SOCKET_LISTENING: only local
[2018-06-14 19:57:49.550] QUERY_DISPLAY: Show queries
[2018-06-14 19:57:49.550] AAAA_QUERY_ANALYSIS: Show AAAA queries
[2018-06-14 19:57:49.550] MAXDBDAYS: max age for stored queries is 365 days
[2018-06-14 19:57:49.550] RESOLVE_IPV6: Resolve IPv6 addresses
[2018-06-14 19:57:49.551] RESOLVE_IPV4: Resolve IPv4 addresses
[2018-06-14 19:57:49.551] DBINTERVAL: saving to DB file every minute
[2018-06-14 19:57:49.551] DBFILE: Using /etc/pihole/pihole-FTL.db
[2018-06-14 19:57:49.551] MAXLOGAGE: Importing up to 24.0 hours of log data
[2018-06-14 19:57:49.551] Finished config file parsing
[2018-06-14 19:57:49.569] Found no other running pihole-FTL process
[2018-06-14 19:57:49.590] PID of FTL process: 2829
[2018-06-14 19:58:01.009] Gravity list entries: 2838316
[2018-06-14 19:58:01.009] Blacklist entries: 1
[2018-06-14 19:58:01.010] No wildcard blocking list present
[2018-06-14 19:58:01.015] Database initialized
[2018-06-14 19:58:01.021] Notice: Increasing queries struct size from 0 to 10000 (440.12 KB)
[2018-06-14 19:58:01.021] Notice: Increasing domains struct size from 0 to 1000 (460.12 KB)
[2018-06-14 19:58:01.022] Notice: Increasing clients struct size from 0 to 10 (460.28 KB)
[2018-06-14 19:58:01.215] New client: 192.168.0.21 (0/10)
[2018-06-14 19:58:01.215] Notice: Increasing forwarded struct size from 0 to 4 (460.38 KB)
[2018-06-14 19:58:01.639] New forward server: 208.67.220.123 resolver2-fs.opendns.com (0/4)
[2018-06-14 19:58:01.640] Notice: Increasing overTime struct size from 0 to 100 (464.42 KB)
[2018-06-14 19:58:02.060] New forward server: 208.67.222.123 resolver1-fs.opendns.com (1/4)
[2018-06-14 19:58:02.244] New client: 192.168.0.20 (1/10)
[2018-06-14 19:58:02.427] New client: 192.168.0.23 (2/10)
[2018-06-14 19:58:02.643] New client: 192.168.0.50 (3/10)
[2018-06-14 19:58:02.824] New client: 192.168.0.28 (4/10)
[2018-06-14 19:58:02.915] New client: 127.0.0.1 localhost (5/10)
[2018-06-14 19:58:03.100] New client: 192.168.0.43 (6/10)
[2018-06-14 19:58:04.183] Notice: Increasing queries struct size from 10000 to 20000 (918.95 KB)
[2018-06-14 19:58:05.566] Notice: Increasing queries struct size from 20000 to 30000 (1.36 MB)
[2018-06-14 19:58:06.918] Notice: Increasing queries struct size from 30000 to 40000 (1.80 MB)
[2018-06-14 19:58:08.345] Notice: Increasing queries struct size from 40000 to 50000 (2.24 MB)
[2018-06-14 19:58:09.811] Notice: Increasing queries struct size from 50000 to 60000 (2.68 MB)
[2018-06-14 19:58:11.208] Notice: Increasing queries struct size from 60000 to 70000 (3.12 MB)
[2018-06-14 19:58:12.550] Notice: Increasing queries struct size from 70000 to 80000 (3.56 MB)
[2018-06-14 19:58:13.952] Notice: Increasing queries struct size from 80000 to 90000 (4.00 MB)
[2018-06-14 19:58:15.300] Notice: Increasing queries struct size from 90000 to 100000 (4.44 MB)
[2018-06-14 19:58:16.645] Notice: Increasing queries struct size from 100000 to 110000 (4.88 MB)
[2018-06-14 19:58:18.053] Notice: Increasing queries struct size from 110000 to 120000 (5.32 MB)
[2018-06-14 19:58:18.567] Notice: Increasing overTime struct size from 100 to 200 (5.33 MB)
[2018-06-14 19:58:19.382] Notice: Increasing queries struct size from 120000 to 130000 (5.77 MB)
[2018-06-14 19:58:20.769] Notice: Increasing queries struct size from 130000 to 140000 (6.21 MB)
[2018-06-14 19:58:21.642] Notice: Increasing domains struct size from 1000 to 2000 (6.23 MB)
[2018-06-14 19:58:22.213] Notice: Increasing queries struct size from 140000 to 150000 (6.67 MB)
[2018-06-14 19:58:23.648] Notice: Increasing queries struct size from 150000 to 160000 (7.11 MB)
[2018-06-14 19:58:25.042] Notice: Increasing queries struct size from 160000 to 170000 (7.55 MB)
[2018-06-14 19:58:26.429] Notice: Increasing queries struct size from 170000 to 180000 (7.99 MB)
[2018-06-14 19:58:27.868] Notice: Increasing queries struct size from 180000 to 190000 (8.43 MB)
[2018-06-14 19:58:29.314] Notice: Increasing queries struct size from 190000 to 200000 (8.87 MB)
[2018-06-14 19:58:30.724] Notice: Increasing queries struct size from 200000 to 210000 (9.31 MB)
[2018-06-14 19:58:32.187] Notice: Increasing queries struct size from 210000 to 220000 (9.75 MB)
[2018-06-14 19:58:33.585] Notice: Increasing queries struct size from 220000 to 230000 (10.19 MB)
[2018-06-14 19:58:35.033] Notice: Increasing queries struct size from 230000 to 240000 (10.63 MB)
[2018-06-14 19:58:36.438] Notice: Increasing queries struct size from 240000 to 250000 (11.07 MB)
[2018-06-14 19:58:37.898] Notice: Increasing queries struct size from 250000 to 260000 (11.51 MB)
[2018-06-14 19:58:39.294] Notice: Increasing queries struct size from 260000 to 270000 (11.95 MB)
[2018-06-14 19:58:40.741] Notice: Increasing queries struct size from 270000 to 280000 (12.39 MB)
[2018-06-14 19:58:42.148] Notice: Increasing queries struct size from 280000 to 290000 (12.83 MB)
[2018-06-14 19:58:43.609] Notice: Increasing queries struct size from 290000 to 300000 (13.28 MB)
[2018-06-14 19:58:45.037] Notice: Increasing queries struct size from 300000 to 310000 (13.72 MB)
[2018-06-14 19:58:46.718] Notice: Increasing queries struct size from 310000 to 320000 (14.16 MB)
[2018-06-14 19:58:48.151] Notice: Increasing queries struct size from 320000 to 330000 (14.60 MB)
[2018-06-14 19:58:48.874] New client: 192.168.0.22 (7/10)
[2018-06-14 19:58:49.838] Notice: Increasing queries struct size from 330000 to 340000 (15.04 MB)
[2018-06-14 19:58:51.020] Imported 338213 queries from the long-term database
[2018-06-14 19:58:51.029] Starting initial log file parsing
[2018-06-14 19:58:51.030] Reading from /var/log/pihole.log (rw-r--r--)
[2018-06-14 20:03:22.338] Notice: Increasing queries struct size from 340000 to 350000 (15.48 MB)
Why is the "Increasing queries struct size from..." always popping up?
Executing the command "pihole -c*" shows the following:
|¯¯¯(¯)_|¯|_ ___|¯|___ Core: v3.3.1
| ¯_/¯|_| ' \/ _ \ / -_) Web: v3.3
|_| |_| |_||_\___/_\___| FTL: v3.0
————————————————————————————
Hostname: dnsbox (Raspberry Pi 1, Model B+)
Uptime: 5 days, 23:27:22
Task Load: 1.26 1.12 1.06 (Active: 2 of 38 tasks)
CPU usage: 96% (900 MHz @ 313k)
RAM usage: 53% (Used: 257 MB of 481 MB)
HDD usage: 7% (Used: 2 GB of 29 GB)
LAN addr: 192.168.0.11 (Gateway: 192.168.0.254)
Pi-hole: Active (Blocking: 0 sites)
Ads Today: 0% (Total: 0 of 0)
Local Qrys: 0% (2 DNS servers)
Blocked: FTL offline
Top Advert:
Top Domain:
Top Client:
Executing the command "ls -al /var/log/pihole.log*" shows the following:
-rw-r--r-- 1 dnsmasq root 136408859 Jun 14 20:10 /var/log/pihole.log
-rw-r--r-- 1 dnsmasq root 3238378 Jun 14 00:00 /var/log/pihole.log.1
-rw-r--r-- 1 dnsmasq root 428163 Jun 13 00:00 /var/log/pihole.log.2.gz
-rw-r--r-- 1 dnsmasq root 278254 Jun 12 00:00 /var/log/pihole.log.3.gz
-rw-r--r-- 1 dnsmasq root 492274 Jun 11 00:00 /var/log/pihole.log.4.gz
-rw-r--r-- 1 dnsmasq root 651041 Jun 10 00:01 /var/log/pihole.log.5.gz
The file "/var/log/pihole.log" did grow a lot today, and am wondering whether there's any application limitation that may impair handling such big log files...
I share the same understanding, and connected this morning to the platform, I see that the "pihole-FTL" process is still running and am wondering about the entries in the file “/var/log/pihole-FTL.log*” file:
[2018-06-15 00:00:41.100] New client: 192.168.0.22 (7/10)
[2018-06-15 00:00:41.529] Notice: Increasing queries struct size from 330000 to 340000 (15.03 MB)
[2018-06-15 00:00:41.925] Notice: Increasing queries struct size from 340000 to 350000 (15.47 MB)
[2018-06-15 00:00:42.087] New client: 192.168.0.11 dnsbox.local (8/10)
[2018-06-15 00:00:42.335] Notice: Increasing queries struct size from 350000 to 360000 (15.91 MB)
[2018-06-15 00:00:42.728] Notice: Increasing queries struct size from 360000 to 370000 (16.35 MB)
[2018-06-15 00:00:43.083] New client: 192.168.0.30 (9/10)
[2018-06-15 00:00:43.084] Notice: Increasing clients struct size from 10 to 20 (16.35 MB)
[2018-06-15 00:00:43.115] Imported 362543 queries from the long-term database
[2018-06-15 00:00:43.117] Reading from /var/log/pihole.log (rw-r--r--)
[2018-06-15 05:10:01.740] Notice: Increasing overTime struct size from 100 to 200 (16.36 MB)
[2018-06-15 08:46:06.472] Notice: Increasing queries struct size from 370000 to 380000 (16.80 MB)
[2018-06-15 09:04:35.310] Notice: Increasing queries struct size from 380000 to 390000 (17.24 MB)
[2018-06-15 09:23:53.577] Notice: Increasing queries struct size from 390000 to 400000 (17.68 MB)
[2018-06-15 09:43:47.697] Notice: Increasing queries struct size from 400000 to 410000 (18.12 MB)
What do these statements “Increasing queries struct size from...” actually mean?
Is there any logical limitation on the size of the log files that FTL can manage in a timely manner?
They are meant to find scenarios like yours with tons of thousands of queries. They indicate that FTL is about to allocate memory (it's first logged, then allocated). So if FTL crashes because of its memory footprint getting too large, then we'd have at least a hint what to look at in the logs.
No, we had a test system running on fairly beefy hardware (32 GB RAM, 500 GB HDD for the log file, ...) and observed that FTL was still working perfectly fine even with hundreds of millions (!) of queries in the log. Also the web dashboard we still usable with almost no noticeable delays:
Thank you for the insight provided. I checked the DNS history, and it could handle over 7000 queries.
I could not check all the details the week before and end of last week about the nature of the DNS queries as I've shutdown and started up again the "pihole-FTL" as a measure not to overload the platform. Here an example screen dump of the peak DNS queries:
Though, have been able to find out the top domains, which result to be originating from the "Kaspersky Endpoint Security" application as depicted with:
