Pi-Hole partly inoperable (API dead)

Help Community Help
1

The issue I am facing:
After restarting the host, Pi-Hole seemed to work (pihole status and pihole restartdns showed/worked fine, DNS resolving was operational I think). Unfortunately,

  • a) no API responses were delivered
    (which is why I discovered this issue at all - the Home Assistant integration complained about connection issues / no API response)
  • b) once logging in to the admin web interface, it showed no data at all:

(please note: as I forgot to take a screenshot, I rebuilt that image from my memory roughly one hour after the incident)

grafik
grafik2222×1507 185 KB

Details about my system:

  • Pi 4, Raspberry Pi OS
  • Latest Pi-hole versions: Pi-hole v5.17.3 / FTL v5.24 / Web Interface v5.21

What I have changed since installing Pi-hole:
I only restarted the host.

I ran pihole -r (repair) which fixed the issue immediately (web page showing data again, API working).

Logs look fine to me (sudo cat /var/log/pihole/FTL.log | grep -c "database is locked" gave 0), no idea WHY Pi-Hole was not fully working (at least the API part seems to have been dead).

Before you guys complain: yes, the database still is big. Perhaps - and only perhaps - if it would be smaller, that incident would not have happened. To rule things out I'd prefer if someone could have a look at the FTL.log excerpt:

FTL.log excerpt 
[2024-01-12 17:09:00.102 11018/T11043] Notice: Database size is 3625.64 MB, deleted 29 rows
[2024-01-12 17:20:00.124 11018/T11043] Notice: Database size is 3625.73 MB, deleted 39 rows
[2024-01-12 17:30:00.082 11018/T11043] Notice: Database size is 3625.80 MB, deleted 30 rows
[2024-01-12 17:37:25.462 11018M] Shutting down...
[2024-01-12 17:37:25.798 11018M] Finished final database update (stored 56 queries)
[2024-01-12 17:37:25.798 11018M] Waiting for threads to join
[2024-01-12 17:37:25.798 11018M] Thread database (0) is idle, terminating it.
[2024-01-12 17:37:25.800 11018M] Thread housekeeper (1) is idle, terminating it.
[2024-01-12 17:37:25.800 11018M] Thread DNS client (2) is idle, terminating it.
[2024-01-12 17:37:25.800 11018M] All threads joined
[2024-01-12 17:37:25.800 11018M] Joining API worker thread 0
[2024-01-12 17:37:25.800 11018M] Joining API worker thread 1
[2024-01-12 17:37:25.800 11018M] Joining API worker thread 2
[2024-01-12 17:37:25.801 11018M] Joining API worker thread 3
[2024-01-12 17:37:25.801 11018M] Joining API worker thread 4
[2024-01-12 17:37:25.977 11018M] ########## FTL terminated after 5d 17h 30m 49s  (code 0)! ##########
[2024-01-12 17:37:44.329 858M] Using log file /var/log/pihole/FTL.log
[2024-01-12 17:37:44.383 858M] ########## FTL started on <Servername> ! ##########
[2024-01-12 17:37:44.387 858M] FTL branch: master
[2024-01-12 17:37:44.387 858M] FTL version: v5.24
[2024-01-12 17:37:44.387 858M] FTL commit: 01227311
[2024-01-12 17:37:44.387 858M] FTL date: 2024-01-06 16:47:56 +0000
[2024-01-12 17:37:44.387 858M] FTL user: pihole
[2024-01-12 17:37:44.387 858M] Compiled for armv7hf (compiled on CI) using arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0
[2024-01-12 17:37:44.388 858M] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
[2024-01-12 17:37:44.388 858M]    SOCKET_LISTENING: only local
[2024-01-12 17:37:44.388 858M]    AAAA_QUERY_ANALYSIS: Show AAAA queries
[2024-01-12 17:37:44.388 858M]    MAXDBDAYS: max age for stored queries is 1095 days
[2024-01-12 17:37:44.388 858M]    RESOLVE_IPV6: Resolve IPv6 addresses
[2024-01-12 17:37:44.388 858M]    RESOLVE_IPV4: Resolve IPv4 addresses
[2024-01-12 17:37:44.389 858M]    DBINTERVAL: saving to DB file every minute
[2024-01-12 17:37:44.389 858M]    DBFILE: Using /etc/pihole/pihole-FTL.db
[2024-01-12 17:37:44.389 858M]    MAXLOGAGE: Importing up to 24.0 hours of log data
[2024-01-12 17:37:44.389 858M]    PRIVACYLEVEL: Set to 0
[2024-01-12 17:37:44.421 858M]    IGNORE_LOCALHOST: Show queries from localhost
[2024-01-12 17:37:44.422 858M]    BLOCKINGMODE: Null IPs for blocked domains
[2024-01-12 17:37:44.422 858M]    ANALYZE_ONLY_A_AND_AAAA: Disabled. Analyzing all queries
[2024-01-12 17:37:44.422 858M]    DBIMPORT: Importing history from database
[2024-01-12 17:37:44.422 858M]    PIDFILE: Using /run/pihole-FTL.pid
[2024-01-12 17:37:44.423 858M]    SOCKETFILE: Using /run/pihole/FTL.sock
[2024-01-12 17:37:44.423 858M]    SETUPVARSFILE: Using /etc/pihole/setupVars.conf
[2024-01-12 17:37:44.423 858M]    MACVENDORDB: Using /etc/pihole/macvendor.db
[2024-01-12 17:37:44.423 858M]    GRAVITYDB: Using /etc/pihole/gravity.db
[2024-01-12 17:37:44.423 858M]    PARSE_ARP_CACHE: Active
[2024-01-12 17:37:44.423 858M]    CNAME_DEEP_INSPECT: Active
[2024-01-12 17:37:44.424 858M]    DELAY_STARTUP: No delay requested.
[2024-01-12 17:37:44.424 858M]    BLOCK_ESNI: Enabled, blocking _esni.{blocked domain}
[2024-01-12 17:37:44.424 858M]    NICE: Set process niceness to -10 (default)
[2024-01-12 17:37:44.424 858M]    MAXNETAGE: Removing IP addresses and host names from network table after 1095 days
[2024-01-12 17:37:44.425 858M]    NAMES_FROM_NETDB: Enabled, trying to get names from network database
[2024-01-12 17:37:44.425 858M]    EDNS0_ECS: Overwrite client from ECS information
[2024-01-12 17:37:44.425 858M]    REFRESH_HOSTNAMES: Periodically refreshing IPv4 names
[2024-01-12 17:37:44.433 858M]    RATE_LIMIT: Rate-limiting client making more than 5000 queries in 60 seconds
[2024-01-12 17:37:44.445 858M]    LOCAL_IPV4: Automatic interface-dependent detection of address
[2024-01-12 17:37:44.446 858M]    LOCAL_IPV6: Automatic interface-dependent detection of address
[2024-01-12 17:37:44.446 858M]    BLOCK_IPV4: Automatic interface-dependent detection of address
[2024-01-12 17:37:44.446 858M]    BLOCK_IPV6: Automatic interface-dependent detection of address
[2024-01-12 17:37:44.446 858M]    REPLY_ADDR4: Using IPv4 address xxx.xxx.xxx.xxx  instead of automatically determined IP address
[2024-01-12 17:37:44.446 858M]    SHOW_DNSSEC: Enabled, showing automatically generated DNSSEC queries
[2024-01-12 17:37:44.446 858M]    MOZILLA_CANARY: Enabled
[2024-01-12 17:37:44.447 858M]    PIHOLE_PTR: internal PTR generation enabled (pi.hole)
[2024-01-12 17:37:44.447 858M]    ADDR2LINE: Enabled
[2024-01-12 17:37:44.447 858M]    REPLY_WHEN_BUSY: Drop queries when the database is busy
[2024-01-12 17:37:44.447 858M]    BLOCK_TTL: 2 seconds
[2024-01-12 17:37:44.447 858M]    BLOCK_ICLOUD_PR: Enabled
[2024-01-12 17:37:44.447 858M]    CHECK_LOAD: Disabled
[2024-01-12 17:37:44.447 858M]    CHECK_SHMEM: Warning if shared-memory usage exceeds 90%
[2024-01-12 17:37:44.448 858M]    CHECK_DISK: Warning if certain disk usage exceeds 90%
[2024-01-12 17:37:44.458 858M] Finished config file parsing
[2024-01-12 17:37:44.485 858M] Creating mutex
[2024-01-12 17:37:44.485 858M] Creating mutex
[2024-01-12 17:37:44.509 858M] PID of FTL process: 858
[2024-01-12 17:37:44.554 858M] Database version is 12
[2024-01-12 17:37:44.554 858M] Resizing "FTL-strings" from 40960 to (81920 * 1) == 81920 (/dev/shm: 1.2MB used, 2.0GB total, FTL uses 1.2MB)
[2024-01-12 17:37:44.559 858M] Imported 0 alias-clients
[2024-01-12 17:37:44.559 858M] Database successfully initialized
[2024-01-12 17:37:44.679 858M] New upstream server: xxx.xxx.xxx.xxx :53 (0/1024)
[2024-01-12 17:37:44.705 858M] New upstream server: xxx.xxx.xxx.xxx :53 (1/1024)
[2024-01-12 17:37:44.960 858M] New upstream server: xxx.xxx.xxx.xxx :53 (2/1024)
[2024-01-12 17:37:45.945 858M] Resizing "FTL-queries" from 180224 to (8192 * 44) == 360448 (/dev/shm: 1.3MB used, 2.0GB total, FTL uses 1.3MB)
[2024-01-12 17:37:46.976 858M] Resizing "FTL-queries" from 360448 to (12288 * 44) == 540672 (/dev/shm: 1.4MB used, 2.0GB total, FTL uses 1.4MB)
[2024-01-12 17:37:48.166 858M] Resizing "FTL-queries" from 540672 to (16384 * 44) == 720896 (/dev/shm: 1.6MB used, 2.0GB total, FTL uses 1.6MB)
[2024-01-12 17:37:48.428 858M] Resizing "FTL-domains" from 20480 to (2048 * 20) == 40960 (/dev/shm: 1.8MB used, 2.0GB total, FTL uses 1.8MB)
[2024-01-12 17:37:48.580 858M] Resizing "FTL-queries" from 720896 to (20480 * 44) == 901120 (/dev/shm: 1.8MB used, 2.0GB total, FTL uses 1.8MB)
[2024-01-12 17:37:48.840 858M] Resizing "FTL-queries" from 901120 to (24576 * 44) == 1081344 (/dev/shm: 2.0MB used, 2.0GB total, FTL uses 2.0MB)
[2024-01-12 17:37:50.167 858M] Resizing "FTL-queries" from 1081344 to (28672 * 44) == 1261568 (/dev/shm: 2.2MB used, 2.0GB total, FTL uses 2.2MB)
[2024-01-12 17:37:50.528 858M] Resizing "FTL-queries" from 1261568 to (32768 * 44) == 1441792 (/dev/shm: 2.4MB used, 2.0GB total, FTL uses 2.4MB)
[2024-01-12 17:37:50.769 858M] Resizing "FTL-strings" from 81920 to (122880 * 1) == 122880 (/dev/shm: 2.5MB used, 2.0GB total, FTL uses 2.5MB)
[2024-01-12 17:37:50.802 858M] Resizing "FTL-queries" from 1441792 to (36864 * 44) == 1622016 (/dev/shm: 2.6MB used, 2.0GB total, FTL uses 2.6MB)
[2024-01-12 17:37:50.926 858M] Resizing "FTL-queries" from 1622016 to (40960 * 44) == 1802240 (/dev/shm: 2.8MB used, 2.0GB total, FTL uses 2.8MB)
[2024-01-12 17:37:51.063 858M] Resizing "FTL-queries" from 1802240 to (45056 * 44) == 1982464 (/dev/shm: 2.9MB used, 2.0GB total, FTL uses 2.9MB)
[2024-01-12 17:37:51.235 858M] Resizing "FTL-queries" from 1982464 to (49152 * 44) == 2162688 (/dev/shm: 3.1MB used, 2.0GB total, FTL uses 3.1MB)
[2024-01-12 17:37:51.361 858M] Resizing "FTL-queries" from 2162688 to (53248 * 44) == 2342912 (/dev/shm: 3.3MB used, 2.0GB total, FTL uses 3.3MB)
[2024-01-12 17:37:51.533 858M] Resizing "FTL-queries" from 2342912 to (57344 * 44) == 2523136 (/dev/shm: 3.5MB used, 2.0GB total, FTL uses 3.5MB)
[2024-01-12 17:37:51.670 858M] Resizing "FTL-queries" from 2523136 to (61440 * 44) == 2703360 (/dev/shm: 3.7MB used, 2.0GB total, FTL uses 3.7MB)
[2024-01-12 17:37:51.820 858M] Resizing "FTL-queries" from 2703360 to (65536 * 44) == 2883584 (/dev/shm: 3.9MB used, 2.0GB total, FTL uses 3.8MB)
[2024-01-12 17:37:51.972 858M] Resizing "FTL-queries" from 2883584 to (69632 * 44) == 3063808 (/dev/shm: 4.0MB used, 2.0GB total, FTL uses 4.0MB)
[2024-01-12 17:37:52.176 858M] Resizing "FTL-queries" from 3063808 to (73728 * 44) == 3244032 (/dev/shm: 4.2MB used, 2.0GB total, FTL uses 4.2MB)
[2024-01-12 17:37:52.403 858M] Resizing "FTL-queries" from 3244032 to (77824 * 44) == 3424256 (/dev/shm: 4.4MB used, 2.0GB total, FTL uses 4.4MB)
[2024-01-12 17:37:52.495 858M] Resizing "FTL-domains" from 40960 to (3072 * 20) == 61440 (/dev/shm: 4.6MB used, 2.0GB total, FTL uses 4.6MB)
[2024-01-12 17:37:52.617 858M] Resizing "FTL-queries" from 3424256 to (81920 * 44) == 3604480 (/dev/shm: 4.6MB used, 2.0GB total, FTL uses 4.6MB)
[2024-01-12 17:37:52.778 858M] Resizing "FTL-queries" from 3604480 to (86016 * 44) == 3784704 (/dev/shm: 4.8MB used, 2.0GB total, FTL uses 4.8MB)
[2024-01-12 17:37:52.919 858M] Resizing "FTL-queries" from 3784704 to (90112 * 44) == 3964928 (/dev/shm: 5.0MB used, 2.0GB total, FTL uses 4.9MB)
[2024-01-12 17:37:53.052 858M] Resizing "FTL-queries" from 3964928 to (94208 * 44) == 4145152 (/dev/shm: 5.1MB used, 2.0GB total, FTL uses 5.1MB)
[2024-01-12 17:37:53.098 858M] Imported 91752 queries from the long-term database
[2024-01-12 17:37:53.100 858M]  -> Total DNS queries: 91752
[2024-01-12 17:37:53.100 858M]  -> Cached DNS queries: 31446
[2024-01-12 17:37:53.101 858M]  -> Forwarded DNS queries: 50002
[2024-01-12 17:37:53.101 858M]  -> Blocked DNS queries: 9504
[2024-01-12 17:37:53.101 858M]  -> Unknown DNS queries: 22
[2024-01-12 17:37:53.101 858M]  -> Unique domains: 2196
[2024-01-12 17:37:53.101 858M]  -> Unique clients: XXX
[2024-01-12 17:37:53.101 858M]  -> Known forward destinations: 3
[2024-01-12 17:37:53.101 858M] Successfully accessed setupVars.conf
[2024-01-12 17:37:53.111 858M] listening on 0.0.0.0 port 53
[2024-01-12 17:37:53.111 858M] listening on :: port 53
[2024-01-12 17:37:53.114 858M] PID of FTL process: 858
[2024-01-12 17:37:53.114 858M] Listening on port 4711 for incoming IPv4 telnet connections
[2024-01-12 17:37:53.115 858M] Listening on port 4711 for incoming IPv6 telnet connections
[2024-01-12 17:37:53.116 858M] Listening on port 4711 for incoming socket telnet connections
[2024-01-12 17:37:53.116 858M] INFO: FTL is running as user pihole (UID 998)
[2024-01-12 17:37:53.119 858M] Reloading DNS cache
[2024-01-12 17:37:53.511 858/T1555] Compiled 1 whitelist and 3 blacklist regex filters for 29 clients in 287.9 msec
[2024-01-12 17:37:53.513 858/T1555] Blocking status is enabled
[2024-01-12 17:37:55.128 858M] Resizing "FTL-dns-cache" from 4096 to (512 * 16) == 8192 (/dev/shm: 5.3MB used, 2.0GB total, FTL uses 5.3MB)
[2024-01-12 17:40:00.064 858/T1555] Notice: Database size is 3625.87 MB, deleted 28 rows
[2024-01-12 17:44:35.324 858M] Resizing "FTL-dns-cache" from 8192 to (768 * 16) == 12288 (/dev/shm: 6.2MB used, 2.0GB total, FTL uses 5.3MB)
[2024-01-12 17:50:00.108 858/T1555] Notice: Database size is 3625.94 MB, deleted 48 rows
[2024-01-12 17:55:30.894 858M] Resizing "FTL-queries" from 4145152 to (98304 * 44) == 4325376 (/dev/shm: 6.2MB used, 2.0GB total, FTL uses 5.3MB)
[2024-01-12 17:56:05.124 858M] Resizing "FTL-dns-cache" from 12288 to (1024 * 16) == 16384 (/dev/shm: 6.4MB used, 2.0GB total, FTL uses 5.5MB)
[2024-01-12 18:00:00.109 858/T1555] Notice: Database size is 3626.01 MB, deleted 477 rows
[2024-01-12 18:06:16.625 858M] Resizing "FTL-dns-cache" from 16384 to (1280 * 16) == 20480 (/dev/shm: 6.4MB used, 2.0GB total, FTL uses 5.5MB)
[2024-01-12 18:10:00.098 858/T1555] Notice: Database size is 3626.06 MB, deleted 42 rows
[2024-01-12 18:12:26.932 858M] Resizing "FTL-dns-cache" from 20480 to (1536 * 16) == 24576 (/dev/shm: 6.4MB used, 2.0GB total, FTL uses 5.5MB)
[2024-01-12 18:15:02.534 858M] Resizing "FTL-dns-cache" from 24576 to (1792 * 16) == 28672 (/dev/shm: 6.4MB used, 2.0GB total, FTL uses 5.5MB)
[2024-01-12 18:20:00.141 858/T1555] Notice: Database size is 3626.19 MB, deleted 159 rows
[2024-01-12 18:25:43.433 858M] Resizing "FTL-dns-cache" from 28672 to (2048 * 16) == 32768 (/dev/shm: 6.4MB used, 2.0GB total, FTL uses 5.5MB)
[2024-01-12 18:28:38.685 858M] Resizing "FTL-queries" from 4325376 to (102400 * 44) == 4505600 (/dev/shm: 6.4MB used, 2.0GB total, FTL uses 5.5MB)
[2024-01-12 18:28:49.575 858M] Resizing "FTL-strings" from 122880 to (163840 * 1) == 163840 (/dev/shm: 6.6MB used, 2.0GB total, FTL uses 5.7MB)
[2024-01-12 18:28:50.800 858M] Resizing "FTL-domains" from 61440 to (4096 * 20) == 81920 (/dev/shm: 6.6MB used, 2.0GB total, FTL uses 5.7MB)
[2024-01-12 18:30:00.117 858/T1555] Notice: Database size is 3626.26 MB, deleted 43 rows
[2024-01-12 18:32:55.717 858M] Resizing "FTL-dns-cache" from 32768 to (2304 * 16) == 36864 (/dev/shm: 6.6MB used, 2.0GB total, FTL uses 5.8MB)
[2024-01-12 18:34:45.617 858M] Shutting down...
[2024-01-12 18:34:45.905 858M] Finished final database update (stored 148 queries)
[2024-01-12 18:34:45.905 858/T1555] Terminating database thread
[2024-01-12 18:34:45.905 858M] Waiting for threads to join
[2024-01-12 18:34:45.906 858M] Thread housekeeper (1) is idle, terminating it.
[2024-01-12 18:34:45.907 858M] Thread DNS client (2) is idle, terminating it.
[2024-01-12 18:34:45.907 858M] All threads joined
[2024-01-12 18:34:45.908 858M] Joining API worker thread 0
[2024-01-12 18:34:45.908 858M] Joining API worker thread 1
[2024-01-12 18:34:45.908 858M] Joining API worker thread 2
[2024-01-12 18:34:45.908 858M] Joining API worker thread 3
[2024-01-12 18:34:45.908 858M] Joining API worker thread 4
[2024-01-12 18:34:45.926 858M] ########## FTL terminated after 57m 1s  (code 0)! ##########
[2024-01-12 18:34:46.116 26054M] Using log file /var/log/pihole/FTL.log
[2024-01-12 18:34:46.116 26054M] ########## FTL started on <Servername> ! ##########
[2024-01-12 18:34:46.116 26054M] FTL branch: master
[2024-01-12 18:34:46.116 26054M] FTL version: v5.24
[2024-01-12 18:34:46.116 26054M] FTL commit: 01227311
[2024-01-12 18:34:46.117 26054M] FTL date: 2024-01-06 16:47:56 +0000
[2024-01-12 18:34:46.117 26054M] FTL user: pihole
[2024-01-12 18:34:46.117 26054M] Compiled for armv7hf (compiled on CI) using arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0
[2024-01-12 18:34:46.117 26054M] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
[2024-01-12 18:34:46.117 26054M]    SOCKET_LISTENING: only local
[2024-01-12 18:34:46.117 26054M]    AAAA_QUERY_ANALYSIS: Show AAAA queries
[2024-01-12 18:34:46.117 26054M]    MAXDBDAYS: max age for stored queries is 1095 days
[2024-01-12 18:34:46.117 26054M]    RESOLVE_IPV6: Resolve IPv6 addresses
[2024-01-12 18:34:46.117 26054M]    RESOLVE_IPV4: Resolve IPv4 addresses
[2024-01-12 18:34:46.117 26054M]    DBINTERVAL: saving to DB file every minute
[2024-01-12 18:34:46.117 26054M]    DBFILE: Using /etc/pihole/pihole-FTL.db
[2024-01-12 18:34:46.117 26054M]    MAXLOGAGE: Importing up to 24.0 hours of log data
[2024-01-12 18:34:46.117 26054M]    PRIVACYLEVEL: Set to 0
[2024-01-12 18:34:46.117 26054M]    IGNORE_LOCALHOST: Show queries from localhost
[2024-01-12 18:34:46.117 26054M]    BLOCKINGMODE: Null IPs for blocked domains
[2024-01-12 18:34:46.117 26054M]    ANALYZE_ONLY_A_AND_AAAA: Disabled. Analyzing all queries
[2024-01-12 18:34:46.117 26054M]    DBIMPORT: Importing history from database
[2024-01-12 18:34:46.117 26054M]    PIDFILE: Using /run/pihole-FTL.pid
[2024-01-12 18:34:46.117 26054M]    SOCKETFILE: Using /run/pihole/FTL.sock
[2024-01-12 18:34:46.118 26054M]    SETUPVARSFILE: Using /etc/pihole/setupVars.conf
[2024-01-12 18:34:46.118 26054M]    MACVENDORDB: Using /etc/pihole/macvendor.db
[2024-01-12 18:34:46.118 26054M]    GRAVITYDB: Using /etc/pihole/gravity.db
[2024-01-12 18:34:46.118 26054M]    PARSE_ARP_CACHE: Active
[2024-01-12 18:34:46.118 26054M]    CNAME_DEEP_INSPECT: Active
[2024-01-12 18:34:46.118 26054M]    DELAY_STARTUP: No delay requested.
[2024-01-12 18:34:46.118 26054M]    BLOCK_ESNI: Enabled, blocking _esni.{blocked domain}
[2024-01-12 18:34:46.118 26054M]    NICE: Set process niceness to -10 (default)
[2024-01-12 18:34:46.118 26054M]    MAXNETAGE: Removing IP addresses and host names from network table after 1095 days
[2024-01-12 18:34:46.118 26054M]    NAMES_FROM_NETDB: Enabled, trying to get names from network database
[2024-01-12 18:34:46.118 26054M]    EDNS0_ECS: Overwrite client from ECS information
[2024-01-12 18:34:46.118 26054M]    REFRESH_HOSTNAMES: Periodically refreshing IPv4 names
[2024-01-12 18:34:46.118 26054M]    RATE_LIMIT: Rate-limiting client making more than 5000 queries in 60 seconds
[2024-01-12 18:34:46.118 26054M]    LOCAL_IPV4: Automatic interface-dependent detection of address
[2024-01-12 18:34:46.118 26054M]    LOCAL_IPV6: Automatic interface-dependent detection of address
[2024-01-12 18:34:46.118 26054M]    BLOCK_IPV4: Automatic interface-dependent detection of address
[2024-01-12 18:34:46.118 26054M]    BLOCK_IPV6: Automatic interface-dependent detection of address
[2024-01-12 18:34:46.118 26054M]    REPLY_ADDR4: Using IPv4 address xxx.xxx.xxx.xxx  instead of automatically determined IP address
[2024-01-12 18:34:46.119 26054M]    SHOW_DNSSEC: Enabled, showing automatically generated DNSSEC queries
[2024-01-12 18:34:46.119 26054M]    MOZILLA_CANARY: Enabled
[2024-01-12 18:34:46.119 26054M]    PIHOLE_PTR: internal PTR generation enabled (pi.hole)
[2024-01-12 18:34:46.119 26054M]    ADDR2LINE: Enabled
[2024-01-12 18:34:46.119 26054M]    REPLY_WHEN_BUSY: Drop queries when the database is busy
[2024-01-12 18:34:46.119 26054M]    BLOCK_TTL: 2 seconds
[2024-01-12 18:34:46.119 26054M]    BLOCK_ICLOUD_PR: Enabled
[2024-01-12 18:34:46.119 26054M]    CHECK_LOAD: Disabled
[2024-01-12 18:34:46.119 26054M]    CHECK_SHMEM: Warning if shared-memory usage exceeds 90%
[2024-01-12 18:34:46.119 26054M]    CHECK_DISK: Warning if certain disk usage exceeds 90%
[2024-01-12 18:34:46.119 26054M] Finished config file parsing
[2024-01-12 18:34:46.119 26054M] Creating mutex
[2024-01-12 18:34:46.120 26054M] Creating mutex
[2024-01-12 18:34:46.121 26054M] PID of FTL process: 26054
[2024-01-12 18:34:46.123 26054M] Database version is 12
[2024-01-12 18:34:46.123 26054M] Resizing "FTL-strings" from 40960 to (81920 * 1) == 81920 (/dev/shm: 2.1MB used, 2.0GB total, FTL uses 1.2MB)
[2024-01-12 18:34:46.138 26054M] Imported 0 alias-clients
[2024-01-12 18:34:46.138 26054M] Database successfully initialized
[2024-01-12 18:34:46.151 26054M] New upstream server: xxx.xxx.xxx.xxx :53 (0/1024)
[2024-01-12 18:34:46.152 26054M] New upstream server: xxx.xxx.xxx.xxx :53 (1/1024)
[2024-01-12 18:34:46.160 26054M] New upstream server: xxx.xxx.xxx.xxx :53 (2/1024)
[2024-01-12 18:34:46.230 26054M] Resizing "FTL-queries" from 180224 to (8192 * 44) == 360448 (/dev/shm: 2.1MB used, 2.0GB total, FTL uses 1.3MB)
[2024-01-12 18:34:46.304 26054M] Resizing "FTL-queries" from 360448 to (12288 * 44) == 540672 (/dev/shm: 2.3MB used, 2.0GB total, FTL uses 1.4MB)
[2024-01-12 18:34:46.370 26054M] Resizing "FTL-domains" from 20480 to (2048 * 20) == 40960 (/dev/shm: 2.5MB used, 2.0GB total, FTL uses 1.6MB)
[2024-01-12 18:34:46.392 26054M] Resizing "FTL-queries" from 540672 to (16384 * 44) == 720896 (/dev/shm: 2.5MB used, 2.0GB total, FTL uses 1.6MB)
[2024-01-12 18:34:46.464 26054M] Resizing "FTL-queries" from 720896 to (20480 * 44) == 901120 (/dev/shm: 2.7MB used, 2.0GB total, FTL uses 1.8MB)
[2024-01-12 18:34:46.591 26054M] Resizing "FTL-queries" from 901120 to (24576 * 44) == 1081344 (/dev/shm: 2.9MB used, 2.0GB total, FTL uses 2.0MB)
[2024-01-12 18:34:46.678 26054M] Resizing "FTL-queries" from 1081344 to (28672 * 44) == 1261568 (/dev/shm: 3.1MB used, 2.0GB total, FTL uses 2.2MB)
[2024-01-12 18:34:46.720 26054M] Resizing "FTL-queries" from 1261568 to (32768 * 44) == 1441792 (/dev/shm: 3.2MB used, 2.0GB total, FTL uses 2.4MB)
[2024-01-12 18:34:46.762 26054M] Resizing "FTL-queries" from 1441792 to (36864 * 44) == 1622016 (/dev/shm: 3.4MB used, 2.0GB total, FTL uses 2.5MB)
[2024-01-12 18:34:46.806 26054M] Resizing "FTL-queries" from 1622016 to (40960 * 44) == 1802240 (/dev/shm: 3.6MB used, 2.0GB total, FTL uses 2.7MB)
[2024-01-12 18:34:46.860 26054M] Resizing "FTL-strings" from 81920 to (122880 * 1) == 122880 (/dev/shm: 3.8MB used, 2.0GB total, FTL uses 2.9MB)
[2024-01-12 18:34:46.863 26054M] Resizing "FTL-queries" from 1802240 to (45056 * 44) == 1982464 (/dev/shm: 3.8MB used, 2.0GB total, FTL uses 2.9MB)
[2024-01-12 18:34:46.911 26054M] Resizing "FTL-queries" from 1982464 to (49152 * 44) == 2162688 (/dev/shm: 4.0MB used, 2.0GB total, FTL uses 3.1MB)
[2024-01-12 18:34:46.948 26054M] Resizing "FTL-queries" from 2162688 to (53248 * 44) == 2342912 (/dev/shm: 4.2MB used, 2.0GB total, FTL uses 3.3MB)
[2024-01-12 18:34:46.982 26054M] Resizing "FTL-queries" from 2342912 to (57344 * 44) == 2523136 (/dev/shm: 4.4MB used, 2.0GB total, FTL uses 3.5MB)
[2024-01-12 18:34:47.017 26054M] Resizing "FTL-queries" from 2523136 to (61440 * 44) == 2703360 (/dev/shm: 4.5MB used, 2.0GB total, FTL uses 3.7MB)
[2024-01-12 18:34:47.052 26054M] Resizing "FTL-queries" from 2703360 to (65536 * 44) == 2883584 (/dev/shm: 4.7MB used, 2.0GB total, FTL uses 3.8MB)
[2024-01-12 18:34:47.094 26054M] Resizing "FTL-queries" from 2883584 to (69632 * 44) == 3063808 (/dev/shm: 4.9MB used, 2.0GB total, FTL uses 4.0MB)
[2024-01-12 18:34:47.160 26054M] Resizing "FTL-queries" from 3063808 to (73728 * 44) == 3244032 (/dev/shm: 5.1MB used, 2.0GB total, FTL uses 4.2MB)
[2024-01-12 18:34:47.211 26054M] Resizing "FTL-domains" from 40960 to (3072 * 20) == 61440 (/dev/shm: 5.3MB used, 2.0GB total, FTL uses 4.4MB)
[2024-01-12 18:34:47.220 26054M] Resizing "FTL-queries" from 3244032 to (77824 * 44) == 3424256 (/dev/shm: 5.3MB used, 2.0GB total, FTL uses 4.4MB)
[2024-01-12 18:34:47.257 26054M] Resizing "FTL-queries" from 3424256 to (81920 * 44) == 3604480 (/dev/shm: 5.5MB used, 2.0GB total, FTL uses 4.6MB)
[2024-01-12 18:34:47.298 26054M] Resizing "FTL-queries" from 3604480 to (86016 * 44) == 3784704 (/dev/shm: 5.6MB used, 2.0GB total, FTL uses 4.8MB)
[2024-01-12 18:34:47.333 26054M] Resizing "FTL-queries" from 3784704 to (90112 * 44) == 3964928 (/dev/shm: 5.8MB used, 2.0GB total, FTL uses 4.9MB)
[2024-01-12 18:34:47.390 26054M] Resizing "FTL-queries" from 3964928 to (94208 * 44) == 4145152 (/dev/shm: 6.0MB used, 2.0GB total, FTL uses 5.1MB)
[2024-01-12 18:34:47.455 26054M] Resizing "FTL-queries" from 4145152 to (98304 * 44) == 4325376 (/dev/shm: 6.2MB used, 2.0GB total, FTL uses 5.3MB)
[2024-01-12 18:34:47.508 26054M] Imported 97836 queries from the long-term database
[2024-01-12 18:34:47.510 26054M]  -> Total DNS queries: 97836
[2024-01-12 18:34:47.510 26054M]  -> Cached DNS queries: 35480
[2024-01-12 18:34:47.510 26054M]  -> Forwarded DNS queries: 51642
[2024-01-12 18:34:47.510 26054M]  -> Blocked DNS queries: 9902
[2024-01-12 18:34:47.510 26054M]  -> Unknown DNS queries: 24
[2024-01-12 18:34:47.510 26054M]  -> Unique domains: 2971
[2024-01-12 18:34:47.510 26054M]  -> Unique clients: XXX
[2024-01-12 18:34:47.510 26054M]  -> Known forward destinations: 3
[2024-01-12 18:34:47.510 26054M] Successfully accessed setupVars.conf
[2024-01-12 18:34:47.512 26054M] listening on 0.0.0.0 port 53
[2024-01-12 18:34:47.512 26054M] listening on :: port 53
[2024-01-12 18:34:47.513 26054M] PID of FTL process: 26054
[2024-01-12 18:34:47.513 26054M] Listening on port 4711 for incoming IPv4 telnet connections
[2024-01-12 18:34:47.514 26054M] Listening on port 4711 for incoming IPv6 telnet connections
[2024-01-12 18:34:47.514 26054M] Listening on port 4711 for incoming socket telnet connections
[2024-01-12 18:34:47.515 26054M] INFO: FTL is running as user pihole (UID 998)
[2024-01-12 18:34:47.515 26054M] Reloading DNS cache
[2024-01-12 18:34:47.681 26054/T26073] Compiled 1 whitelist and 3 blacklist regex filters for 29 clients in 63.3 msec
[2024-01-12 18:34:47.681 26054/T26073] Blocking status is enabled
[2024-01-12 18:34:49.590 26054M] Resizing "FTL-dns-cache" from 4096 to (512 * 16) == 8192 (/dev/shm: 6.4MB used, 2.0GB total, FTL uses 5.5MB)
[2024-01-12 18:36:58.702 26054M] Resizing "FTL-queries" from 4325376 to (102400 * 44) == 4505600 (/dev/shm: 6.4MB used, 2.0GB total, FTL uses 5.5MB)
[2024-01-12 18:38:42.205 26054M] Resizing "FTL-domains" from 61440 to (4096 * 20) == 81920 (/dev/shm: 6.5MB used, 2.0GB total, FTL uses 5.7MB)
[2024-01-12 18:40:00.038 26054/T26073] Notice: Database size is 3626.36 MB, deleted 28 rows
[2024-01-12 18:40:01.930 26054M] Resizing "FTL-dns-cache" from 8192 to (768 * 16) == 12288 (/dev/shm: 6.6MB used, 2.0GB total, FTL uses 5.7MB)
[2024-01-12 18:40:34.422 26054M] Resizing "FTL-strings" from 122880 to (163840 * 1) == 163840 (/dev/shm: 6.6MB used, 2.0GB total, FTL uses 5.7MB)
[2024-01-12 18:45:25.611 26054M] Resizing "FTL-dns-cache" from 12288 to (1024 * 16) == 16384 (/dev/shm: 6.6MB used, 2.0GB total, FTL uses 5.7MB)

please note:

  • masked only few things ("XXX")
  • time codes for reference:
    • restarted host @ 2024-01-12 17:37:XX
    • ran pihole -r @ 2024-01-12 18:34:XX

Only potentially suspicious thing looking at the full FTL.log is: before the host restart "Resizing ..." entries only happened rarely, only every few hours. Since the incident and repair, it is logged much more frequently:

[2024-01-12 19:23:51.086 9189M] Resizing "FTL-queries" from 4505600 to (106496 * 44) == 4685824 (/dev/shm: 6.6MB used, 2.0GB total, FTL uses 5.7MB)
[2024-01-12 19:24:56.444 9189M] Resizing "FTL-dns-cache" from 8192 to (768 * 16) == 12288 (/dev/shm: 6.8MB used, 2.0GB total, FTL uses 5.9MB)
[2024-01-12 19:29:55.941 9189M] Resizing "FTL-domains" from 81920 to (5120 * 20) == 102400 (/dev/shm: 6.8MB used, 2.0GB total, FTL uses 5.9MB)
[2024-01-12 19:30:00.049 9189/T9251] Notice: Database size is 3626.81 MB, deleted 374 rows
[2024-01-12 19:33:25.421 9189M] Resizing "FTL-dns-cache" from 12288 to (1024 * 16) == 16384 (/dev/shm: 6.8MB used, 2.0GB total, FTL uses 5.9MB)
[2024-01-12 19:40:00.116 9189/T9251] Notice: Database size is 3626.87 MB, deleted 646 rows
[2024-01-12 19:42:24.498 9189M] Resizing "FTL-dns-cache" from 16384 to (1280 * 16) == 20480 (/dev/shm: 6.8MB used, 2.0GB total, FTL uses 5.9MB)
[2024-01-12 19:47:57.809 9189M] Resizing "FTL-dns-cache" from 20480 to (1536 * 16) == 24576 (/dev/shm: 6.8MB used, 2.0GB total, FTL uses 5.9MB)
[2024-01-12 19:48:20.364 9189M] Resizing "FTL-strings" from 163840 to (204800 * 1) == 204800 (/dev/shm: 6.8MB used, 2.0GB total, FTL uses 5.9MB)
[2024-01-12 19:50:00.050 9189/T9251] Notice: Database size is 3626.92 MB, deleted 298 rows
[2024-01-12 19:55:10.359 9189M] Resizing "FTL-dns-cache" from 24576 to (1792 * 16) == 28672 (/dev/shm: 6.9MB used, 2.0GB total, FTL uses 6.0MB)
[2024-01-12 20:00:00.137 9189/T9251] Notice: Database size is 3626.98 MB, deleted 934 rows
2

Please upload a debug log and post just the token URL that is generated after the log is uploaded by running the following command from the Pi-hole host terminal:

pihole -d

or do it through the Web interface:

Tools > Generate Debug Log

3

Can we shorten this (don't want to have the same discussion as last year in Database locked (again) - #5 by bcutter ff. as the debug log still contains too much sensitive information): what specifically are you looking at?

I won't provide the debug log as is and filtering it before custom uploading it takes quite some time.

4

I won't be able to help you with this issue. The debug log is our primary diagnostic tool and avoids multiple questions and replies.

1 Like
7

I can provide one, with a bit of custom masking from me. Before I'll do so:

Is it helpful at all as the issue has been fixed already (running a pihole -r repair)?
Good chances anything helpful might still be in the logs?

8

Sounds like your problem is solved. Look through your pihole.log and FTL.log and see if there is anything unusual in the logs (or previous days logs).

9

The debug log contains only a small section from the head and tail of the Pi-hole logs in order to capture anything notable at startup or at the time of the debug log. Since your issue has gone away they probably won't reveal anything useful. In the earlier thread I mentioned the utility atop. Did you install that in the end? It might be useful if this happens again, in order to see if Home Assistant or something else is having an impact.

10

I mentioned right at the beginning it is fixed...

...and I could not spot anything interesting in neither pihole.log nor FTL.log. That's the main reason why I started this thread. If things are broken for an unknown reason, that's important to clarify, isn't it?

Yes I installed it - but did not really use it so far. Also not right in the middle of a "Pi-hole is not working" situation. First thing that comes to my mind is to check for a locked database using sudo grep -c "database is locked" /var/log/pihole/FTL.log and quickly fixing it by running pihole -r. As there were "only" 3 outages/incidents within 12 months, I simply forget to have a look at atop once things don't operate as they should. And even if I would start it and have a look: it's not really user-friendly - like Glances, listing issues in a separate section or highlighting potential issues in red. What to look at with atop?

11

Fair enough, it's only been seven months since it was suggested.

Up to you. It's a very powerful tool. You've got some reading to do.

As I said in July, It's a recording version of top with extra detail for networking and i/o. It will allow you to go back in time to any point and see in detail how your Pi's resources were being utilised. You can use this information to help determine if something installed is impacting applictions such as Pi-hole.

12

Yes. So where's the news in your post :slight_smile:

Running atop really does not give me any remarks on things to worry. Everything looking good.

My browser history reminded me I already took a look at A guide to atop command in Linux | DigitalOcean in July last year. Had a few times a look specifically at (sudo) atop -d, also nothing exciting in there.

14

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.