FTL crashed

General Beta 6.0
1

Good morning,

This weekend, my FTL crashed. This pihole was running version FTL vDev (development, a2dbfedd). Here's the log I found:

2024-09-15 00:00:27.985 CEST [1125/T1142] INFO: Size of /etc/pihole/pihole-FTL.db is 163.87 MB, deleted 1 rows
2024-09-15 00:27:43.278 CEST [1125/T1141] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-15 00:27:43.281 CEST [1125/T1141] INFO: Time offset: -4.343808e+00 ms (excluded 0 outliers)
2024-09-15 00:27:43.282 CEST [1125/T1141] INFO: Round-trip delay: 1.787305e+01 ms (excluded 0 outliers)
2024-09-15 01:00:25.601 CEST [1125/T1142] INFO: Size of /etc/pihole/pihole-FTL.db is 164.14 MB, deleted 1 rows
2024-09-15 01:27:48.297 CEST [1125/T1141] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-15 01:27:48.299 CEST [1125/T1141] INFO: Time offset: -4.765817e+00 ms (excluded 1 outliers)
2024-09-15 01:27:48.300 CEST [1125/T1141] INFO: Round-trip delay: 1.832587e+01 ms (excluded 1 outliers)
2024-09-15 01:31:12.067 CEST [1125/T1142] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-09-15 01:31:12.070 CEST [1125/T1142] INFO: ---------------------------->  FTL crashed!  <----------------------------
2024-09-15 01:31:12.070 CEST [1125/T1142] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-09-15 01:31:12.071 CEST [1125/T1142] INFO: Please report a bug at https://github.com/pi-hole/FTL/issues
2024-09-15 01:31:12.071 CEST [1125/T1142] INFO: and include in your report already the following details:
2024-09-15 01:31:12.071 CEST [1125/T1142] INFO: FTL has been running for 44038 seconds
2024-09-15 01:31:12.072 CEST [1125/T1142] INFO: FTL branch: development
2024-09-15 01:31:12.072 CEST [1125/T1142] INFO: FTL version: v5.25.2-2134-ga2dbfedd
2024-09-15 01:31:12.072 CEST [1125/T1142] INFO: FTL commit: a2dbfedd
2024-09-15 01:31:12.073 CEST [1125/T1142] INFO: FTL date: 2024-09-02 22:38:31 +0100
2024-09-15 01:31:12.082 CEST [1125/T1142] INFO: FTL user: started as pihole, ended as pihole
2024-09-15 01:31:12.082 CEST [1125/T1142] INFO: Compiled for armv6l (compiled locally) using cc (Debian 12.2.0-14) 12.2.0
2024-09-15 01:31:12.083 CEST [1125/T1142] INFO: Process details: MID: 1125
2024-09-15 01:31:12.084 CEST [1125/T1142] INFO:                  PID: 1125
2024-09-15 01:31:12.084 CEST [1125/T1142] INFO:                  TID: 1142
2024-09-15 01:31:12.084 CEST [1125/T1142] INFO:                  Name: database
2024-09-15 01:31:12.085 CEST [1125/T1142] INFO: Received signal: Segmentation fault
2024-09-15 01:31:12.085 CEST [1125/T1142] INFO:      at address: 0xb9
2024-09-15 01:31:12.085 CEST [1125/T1142] INFO:      with code:  SEGV_MAPERR (Address not mapped to object)
2024-09-15 01:31:12.106 CEST [1125/T1142] INFO: Backtrace:
2024-09-15 01:31:12.117 CEST [1125/T1142] INFO: B[0000]: /usr/bin/pihole-FTL(generate_backtrace+0x44) [0x4ac78c]
2024-09-15 01:31:12.503 CEST [1125/T1142] INFO: L[0000]: /root/FTL/src/signals.c:107
2024-09-15 01:31:12.521 CEST [1125/T1142] INFO: B[0001]: /usr/bin/pihole-FTL(+0x6ade4) [0x4acde4]
2024-09-15 01:31:12.713 CEST [1125/T1142] INFO: L[0001]: /root/FTL/src/signals.c:250
2024-09-15 01:31:12.729 CEST [1125/T1142] INFO: B[0002]: /lib/arm-linux-gnueabi/libc.so.6(__default_rt_sa_restorer+0) [0xb6cc48f0]
2024-09-15 01:31:12.729 CEST [1125/T1142] INFO: B[0003]: /usr/bin/pihole-FTL(+0x24e3ac) [0x6903ac]
2024-09-15 01:31:14.528 CEST [1125/T1142] INFO: L[0003]: /root/FTL/src/database/sqlite3.c:113894
2024-09-15 01:31:14.639 CEST [1125/T1142] INFO: B[0004]: /usr/bin/pihole-FTL(+0x256a68) [0x698a68]
2024-09-15 01:31:16.366 CEST [1125/T1142] INFO: L[0004]: /root/FTL/src/database/sqlite3.c:114549
2024-09-15 01:31:16.464 CEST [1125/T1142] INFO: B[0005]: /usr/bin/pihole-FTL(+0x259ed8) [0x69bed8]
2024-09-15 01:31:17.888 CEST [1125/T1142] INFO: L[0005]: /root/FTL/src/database/sqlite3.c:144149
2024-09-15 01:31:17.993 CEST [1125/T1142] INFO: B[0006]: /usr/bin/pihole-FTL(+0x2487f4) [0x68a7f4]
2024-09-15 01:31:19.368 CEST [1125/T1142] INFO: L[0006]: /root/FTL/src/database/sqlite3.c:151443
2024-09-15 01:31:19.462 CEST [1125/T1142] INFO: B[0007]: /usr/bin/pihole-FTL(+0x279eb8) [0x6bbeb8]
2024-09-15 01:31:20.786 CEST [1125/T1142] INFO: L[0007]: /root/FTL/src/database/sqlite3.c:176408
2024-09-15 01:31:20.880 CEST [1125/T1142] INFO: B[0008]: /usr/bin/pihole-FTL(+0x27d6ac) [0x6bf6ac]
2024-09-15 01:31:22.207 CEST [1125/T1142] INFO: L[0008]: /root/FTL/src/database/sqlite3.c:175078
2024-09-15 01:31:22.300 CEST [1125/T1142] INFO: B[0009]: /usr/bin/pihole-FTL(+0x27e1dc) [0x6c01dc]
2024-09-15 01:31:23.665 CEST [1125/T1142] INFO: L[0009]: /root/FTL/src/database/sqlite3.c:142599
2024-09-15 01:31:23.762 CEST [1125/T1142] INFO: B[0010]: /usr/bin/pihole-FTL(+0x27e798) [0x6c0798]
2024-09-15 01:31:25.128 CEST [1125/T1142] INFO: L[0010]: /root/FTL/src/database/sqlite3.c:142672
2024-09-15 01:31:25.221 CEST [1125/T1142] INFO: B[0011]: /usr/bin/pihole-FTL(sqlite3_prepare_v2+0x24) [0x6c0d4c]
2024-09-15 01:31:26.572 CEST [1125/T1142] INFO: L[0011]: /root/FTL/src/database/sqlite3.c:142761
2024-09-15 01:31:26.672 CEST [1125/T1142] INFO: B[0012]: /usr/bin/pihole-FTL(get_number_of_queries_in_DB+0xd4) [0x5358d4]
2024-09-15 01:31:26.828 CEST [1125/T1142] INFO: L[0012]: /root/FTL/src/database/query-table.c:473
2024-09-15 01:31:26.847 CEST [1125/T1142] INFO: B[0013]: /usr/bin/pihole-FTL(queries_to_database+0x7cc) [0x538b90]
2024-09-15 01:31:26.000 CEST [1125/T1142] INFO: L[0013]: /root/FTL/src/database/query-table.c:1624
2024-09-15 01:31:27.019 CEST [1125/T1142] INFO: B[0014]: /usr/bin/pihole-FTL(DB_thread+0x4f0) [0x522d78]
2024-09-15 01:31:27.158 CEST [1125/T1142] INFO: L[0014]: /root/FTL/src/database/database-thread.c:119
2024-09-15 01:31:27.181 CEST [1125/T1142] INFO: ------ Listing content of directory /dev/shm ------
2024-09-15 01:31:27.182 CEST [1125/T1142] INFO: File Mode User:Group      Size  Filename
2024-09-15 01:31:27.191 CEST [1125/T1142] INFO: rwxrwxrwx root:root       300   .
2024-09-15 01:31:27.194 CEST [1125/T1142] INFO: rwxr-xr-x root:root         3K  ..
2024-09-15 01:31:27.195 CEST [1125/T1142] INFO: rw------- pihole:pihole   560K  FTL-fifo-log
2024-09-15 01:31:27.196 CEST [1125/T1142] INFO: rw------- pihole:pihole     4K  FTL-per-client-regex
2024-09-15 01:31:27.197 CEST [1125/T1142] INFO: rw------- pihole:pihole   184K  FTL-dns-cache
2024-09-15 01:31:27.198 CEST [1125/T1142] INFO: rw------- pihole:pihole    12K  FTL-overTime
2024-09-15 01:31:27.199 CEST [1125/T1142] INFO: rw------- pihole:pihole     3M  FTL-queries
2024-09-15 01:31:27.200 CEST [1125/T1142] INFO: rw------- pihole:pihole    12K  FTL-upstreams
2024-09-15 01:31:27.202 CEST [1125/T1142] INFO: rw------- pihole:pihole   168K  FTL-clients
2024-09-15 01:31:27.203 CEST [1125/T1142] INFO: rw------- pihole:pihole   147K  FTL-domains
2024-09-15 01:31:27.204 CEST [1125/T1142] INFO: rw------- pihole:pihole   123K  FTL-strings
2024-09-15 01:31:27.205 CEST [1125/T1142] INFO: rw------- pihole:pihole    16   FTL-settings
2024-09-15 01:31:27.206 CEST [1125/T1142] INFO: rw------- pihole:pihole   292   FTL-counters
2024-09-15 01:31:27.207 CEST [1125/T1142] INFO: rw------- pihole:pihole    56   FTL-lock
2024-09-15 01:31:27.208 CEST [1125/T1142] INFO: rw-r--r-- root:root         0   .tmpfs
2024-09-15 01:31:27.208 CEST [1125/T1142] INFO: ---------------------------------------------------
2024-09-15 01:31:27.209 CEST [1125/T1142] INFO: Please also include some lines from above the !!!!!!!!! header.
2024-09-15 01:31:27.209 CEST [1125/T1142] INFO: Thank you for helping us to improve our FTL engine!
2024-09-15 01:31:27.209 CEST [1125/T1142] INFO: Waiting for threads to join
2024-09-15 01:31:27.210 CEST [1125/T1142] INFO: Thread database (0) is still busy, cancelling it.
2024-09-15 01:31:27.260 CEST [1125/T1145] INFO: Terminating timer thread
2024-09-15 01:31:27.256 CEST [1125M] ERROR: Error when obtaining outer SHM lock: Owner died
2024-09-15 01:31:27.289 CEST [1125M] ERROR: Error when obtaining inner SHM lock: Owner died
2024-09-15 01:31:27.351 CEST [1125M] INFO: Shutting down... // exit code 0 // jmpret 0
2024-09-15 01:31:27.930 CEST [1125/T1144] INFO: Terminating resolver thread
2024-09-15 01:31:27.934 CEST [1125/T1143] INFO: Terminating GC thread
2024-09-15 02:27:49.104 CEST [1125/T1141] INFO: Terminating NTP thread
2

Crashed again last night, below are full logs:

2024-09-16 07:26:32.585 CEST [1124M] INFO: ########## FTL started on pogo01! ##########
2024-09-16 07:26:32.590 CEST [1124M] INFO: FTL branch: development
2024-09-16 07:26:32.590 CEST [1124M] INFO: FTL version: v5.25.2-2160-g563a5aad
2024-09-16 07:26:32.590 CEST [1124M] INFO: FTL commit: 563a5aad
2024-09-16 07:26:32.591 CEST [1124M] INFO: FTL date: 2024-09-15 19:44:09 +0200
2024-09-16 07:26:32.591 CEST [1124M] INFO: FTL user: pihole
2024-09-16 07:26:32.592 CEST [1124M] INFO: Compiled for armv6l (compiled locally) using cc (Debian 12.2.0-14) 12.2.0
2024-09-16 07:26:32.635 CEST [1124M] INFO: Wrote config file:
2024-09-16 07:26:32.636 CEST [1124M] INFO:  - 150 total entries
2024-09-16 07:26:32.637 CEST [1124M] INFO:  - 143 entries are default
2024-09-16 07:26:32.637 CEST [1124M] INFO:  - 7 entries are modified
2024-09-16 07:26:32.637 CEST [1124M] INFO:  - 0 entries are forced through environment
2024-09-16 07:26:32.679 CEST [1124M] INFO: Parsed config file /etc/pihole/pihole.toml successfully
2024-09-16 07:26:32.739 CEST [1124M] INFO: PID of FTL process: 1124
2024-09-16 07:26:32.768 CEST [1124M] INFO: listening on 0.0.0.0 port 53
2024-09-16 07:26:32.769 CEST [1124M] INFO: listening on :: port 53
2024-09-16 07:26:32.793 CEST [1129M] INFO: PID of FTL process: 1129
2024-09-16 07:26:32.837 CEST [1129M] ERROR: SQLite3: recovered 33 frames from WAL file /etc/pihole/pihole-FTL.db-wal (283)
2024-09-16 07:26:32.846 CEST [1129M] INFO: Database version is 19
2024-09-16 07:26:32.891 CEST [1129M] INFO: Database successfully initialized
2024-09-16 07:26:41.948 CEST [1129M] INFO: Imported 25 queries from the on-disk database (it has 2681543 rows)
2024-09-16 07:26:41.955 CEST [1129M] INFO: Parsing queries in database
2024-09-16 07:26:41.991 CEST [1129M] INFO: Imported 25 queries from the long-term database
2024-09-16 07:26:41.992 CEST [1129M] INFO:  -> Total DNS queries: 25
2024-09-16 07:26:41.992 CEST [1129M] INFO:  -> Cached DNS queries: 25
2024-09-16 07:26:41.992 CEST [1129M] INFO:  -> Forwarded DNS queries: 0
2024-09-16 07:26:41.992 CEST [1129M] INFO:  -> Blocked DNS queries: 0
2024-09-16 07:26:41.993 CEST [1129M] INFO:  -> Unknown DNS queries: 0
2024-09-16 07:26:41.993 CEST [1129M] INFO:  -> Unique domains: 2
2024-09-16 07:26:41.994 CEST [1129M] INFO:  -> Unique clients: 1
2024-09-16 07:26:41.994 CEST [1129M] INFO:  -> DNS cache records: 2
2024-09-16 07:26:41.994 CEST [1129M] INFO:  -> Known forward destinations: 0
2024-09-16 07:26:42.055 CEST [1129M] INFO: FTL is running as user pihole (UID 999)
2024-09-16 07:26:42.081 CEST [1129M] INFO: Reading certificate from /etc/pihole/tls.pem ...
2024-09-16 07:26:42.086 CEST [1129M] INFO: Using SSL/TLS certificate file /etc/pihole/tls.pem
2024-09-16 07:26:42.129 CEST [1129M] INFO: Restored 0 API sessions from the database
2024-09-16 07:26:42.149 CEST [1129M] INFO: Blocking status is enabled
2024-09-16 07:26:42.254 CEST [1129/T1141] INFO: Compiled 0 allow and 0 deny regex for 2 clients in 3.9 msec
2024-09-16 07:26:46.232 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 07:26:46.233 CEST [1129/T1140] INFO: Time offset: 7.688524e+04 ms (excluded 0 outliers)
2024-09-16 07:26:46.233 CEST [1129/T1140] INFO: Round-trip delay: 1.780689e+01 ms (excluded 0 outliers)
2024-09-16 08:28:08.404 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 08:28:08.427 CEST [1129/T1140] INFO: Time offset: -6.774276e+00 ms (excluded 0 outliers)
2024-09-16 08:28:08.428 CEST [1129/T1140] INFO: Round-trip delay: 1.856071e+01 ms (excluded 0 outliers)
2024-09-16 08:28:09.018 CEST [1129/T1638] INFO: NTP server listening on :::123 (IPv6)
2024-09-16 08:28:09.018 CEST [1129/T1637] INFO: NTP server listening on 0.0.0.0:123 (IPv4)
2024-09-16 08:40:24.918 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 487 rows
2024-09-16 08:49:25.097 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 237 rows
2024-09-16 09:00:25.085 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 295 rows
2024-09-16 09:09:25.494 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 309 rows
2024-09-16 09:20:25.565 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 300 rows
2024-09-16 09:28:13.790 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 09:28:13.793 CEST [1129/T1140] INFO: Time offset: -5.180938e+00 ms (excluded 1 outliers)
2024-09-16 09:28:13.794 CEST [1129/T1140] INFO: Round-trip delay: 1.516676e+01 ms (excluded 1 outliers)
2024-09-16 09:30:25.206 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 46 rows
2024-09-16 09:40:24.058 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 117 rows
2024-09-16 09:50:24.567 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 57 rows
2024-09-16 10:00:25.380 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 43 rows
2024-09-16 10:10:24.200 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 46 rows
2024-09-16 10:20:24.226 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 76 rows
2024-09-16 10:28:18.232 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 10:28:18.239 CEST [1129/T1140] INFO: Time offset: -2.469914e+00 ms (excluded 1 outliers)
2024-09-16 10:28:18.240 CEST [1129/T1140] INFO: Round-trip delay: 1.912396e+01 ms (excluded 1 outliers)
2024-09-16 10:30:24.374 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 69 rows
2024-09-16 10:40:25.112 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 51 rows
2024-09-16 10:49:24.728 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 82 rows
2024-09-16 10:59:24.720 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 51 rows
2024-09-16 11:10:24.314 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 62 rows
2024-09-16 11:20:24.201 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 57 rows
2024-09-16 11:28:23.318 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 11:28:23.321 CEST [1129/T1140] INFO: Time offset: -6.564498e+00 ms (excluded 0 outliers)
2024-09-16 11:28:23.322 CEST [1129/T1140] INFO: Round-trip delay: 2.210522e+01 ms (excluded 0 outliers)
2024-09-16 11:30:24.518 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 92 rows
2024-09-16 11:40:24.181 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 89 rows
2024-09-16 11:50:24.197 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 59 rows
2024-09-16 12:00:25.139 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 140 rows
2024-09-16 12:10:24.675 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 79 rows
2024-09-16 12:20:25.026 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 82 rows
2024-09-16 12:28:28.257 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 12:28:28.259 CEST [1129/T1140] INFO: Time offset: -5.107164e+00 ms (excluded 1 outliers)
2024-09-16 12:28:28.259 CEST [1129/T1140] INFO: Round-trip delay: 1.635422e+01 ms (excluded 1 outliers)
2024-09-16 12:30:26.470 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 193 rows
2024-09-16 12:40:24.638 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 133 rows
2024-09-16 12:50:25.510 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 80 rows
2024-09-16 13:00:24.078 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 84 rows
2024-09-16 13:10:24.343 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 55 rows
2024-09-16 13:20:24.127 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 52 rows
2024-09-16 13:28:33.286 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 13:28:33.289 CEST [1129/T1140] INFO: Time offset: -6.256342e+00 ms (excluded 1 outliers)
2024-09-16 13:28:33.289 CEST [1129/T1140] INFO: Round-trip delay: 1.755285e+01 ms (excluded 1 outliers)
2024-09-16 13:29:25.525 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 44 rows
2024-09-16 13:40:24.469 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 97 rows
2024-09-16 13:50:25.571 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 99 rows
2024-09-16 14:00:25.529 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 64 rows
2024-09-16 14:10:24.122 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 76 rows
2024-09-16 14:20:24.095 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 69 rows
2024-09-16 14:28:38.361 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 14:28:38.364 CEST [1129/T1140] INFO: Time offset: -2.142974e+00 ms (excluded 1 outliers)
2024-09-16 14:28:38.365 CEST [1129/T1140] INFO: Round-trip delay: 2.084841e+01 ms (excluded 1 outliers)
2024-09-16 14:30:23.935 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 90 rows
2024-09-16 14:40:24.306 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 71 rows
2024-09-16 14:50:25.260 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 68 rows
2024-09-16 15:00:24.546 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 85 rows
2024-09-16 15:10:24.751 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 67 rows
2024-09-16 15:20:25.977 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 54 rows
2024-09-16 15:28:43.293 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 15:28:43.296 CEST [1129/T1140] INFO: Time offset: -1.090166e+01 ms (excluded 0 outliers)
2024-09-16 15:28:43.296 CEST [1129/T1140] INFO: Round-trip delay: 1.783675e+01 ms (excluded 0 outliers)
2024-09-16 15:30:22.732 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 72 rows
2024-09-16 15:40:25.245 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 71 rows
2024-09-16 15:50:26.059 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 61 rows
2024-09-16 15:59:25.685 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 53 rows
2024-09-16 16:10:25.183 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 76 rows
2024-09-16 16:20:24.699 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 47 rows
2024-09-16 16:28:48.314 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 16:28:48.317 CEST [1129/T1140] INFO: Time offset: -6.175727e+00 ms (excluded 0 outliers)
2024-09-16 16:28:48.317 CEST [1129/T1140] INFO: Round-trip delay: 1.767796e+01 ms (excluded 0 outliers)
2024-09-16 16:30:26.094 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 62 rows
2024-09-16 16:40:25.260 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 101 rows
2024-09-16 16:50:25.783 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 62 rows
2024-09-16 16:59:25.600 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 245 rows
2024-09-16 17:10:26.497 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 486 rows
2024-09-16 17:20:25.066 CEST [1129/T1141] INFO: Size of /etc/pihole/pihole-FTL.db is 164.25 MB, deleted 170 rows
2024-09-16 17:28:53.192 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 17:28:53.195 CEST [1129/T1140] INFO: Time offset: -6.538289e+00 ms (excluded 1 outliers)
2024-09-16 17:28:53.195 CEST [1129/T1140] INFO: Round-trip delay: 1.538856e+01 ms (excluded 1 outliers)
2024-09-16 18:28:58.331 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 18:28:58.335 CEST [1129/T1140] INFO: Time offset: 1.386574e-01 ms (excluded 1 outliers)
2024-09-16 18:28:58.335 CEST [1129/T1140] INFO: Round-trip delay: 2.133880e+01 ms (excluded 1 outliers)
2024-09-16 19:00:02.072 CEST [1129/T1143] ERROR: Cannot receive UDP DNS reply: Resource temporarily unavailable
2024-09-16 19:29:03.284 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 19:29:03.296 CEST [1129/T1140] INFO: Time offset: -1.267751e+01 ms (excluded 2 outliers)
2024-09-16 19:29:03.300 CEST [1129/T1140] INFO: Round-trip delay: 1.557589e+01 ms (excluded 2 outliers)
2024-09-16 20:29:08.701 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 20:29:08.732 CEST [1129/T1140] INFO: Time offset: -4.761010e+00 ms (excluded 0 outliers)
2024-09-16 20:29:08.732 CEST [1129/T1140] INFO: Round-trip delay: 2.153736e+01 ms (excluded 0 outliers)
2024-09-16 21:29:13.394 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 21:29:13.432 CEST [1129/T1140] INFO: Time offset: -4.811900e+00 ms (excluded 1 outliers)
2024-09-16 21:29:13.433 CEST [1129/T1140] INFO: Round-trip delay: 1.603985e+01 ms (excluded 1 outliers)
2024-09-16 22:29:18.325 CEST [1129/T1140] INFO: Received 8/8 valid NTP replies from pool.ntp.org
2024-09-16 22:29:18.328 CEST [1129/T1140] INFO: Time offset: -6.136451e+00 ms (excluded 1 outliers)
2024-09-16 22:29:18.328 CEST [1129/T1140] INFO: Round-trip delay: 1.833487e+01 ms (excluded 1 outliers)
2024-09-16 22:56:25.025 CEST [1129/T1141] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-09-16 22:56:25.028 CEST [1129/T1141] INFO: ---------------------------->  FTL crashed!  <----------------------------
2024-09-16 22:56:25.029 CEST [1129/T1141] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-09-16 22:56:25.029 CEST [1129/T1141] INFO: Please report a bug at https://github.com/pi-hole/FTL/issues
2024-09-16 22:56:25.029 CEST [1129/T1141] INFO: and include in your report already the following details:
2024-09-16 22:56:25.030 CEST [1129/T1141] INFO: FTL has been running for 55793 seconds
2024-09-16 22:56:25.030 CEST [1129/T1141] INFO: FTL branch: development
2024-09-16 22:56:25.030 CEST [1129/T1141] INFO: FTL version: v5.25.2-2160-g563a5aad
2024-09-16 22:56:25.031 CEST [1129/T1141] INFO: FTL commit: 563a5aad
2024-09-16 22:56:25.031 CEST [1129/T1141] INFO: FTL date: 2024-09-15 19:44:09 +0200
2024-09-16 22:56:25.044 CEST [1129/T1141] INFO: FTL user: started as pihole, ended as pihole
2024-09-16 22:56:25.045 CEST [1129/T1141] INFO: Compiled for armv6l (compiled locally) using cc (Debian 12.2.0-14) 12.2.0
2024-09-16 22:56:25.045 CEST [1129/T1141] INFO: Process details: MID: 1129
2024-09-16 22:56:25.045 CEST [1129/T1141] INFO:                  PID: 1129
2024-09-16 22:56:25.046 CEST [1129/T1141] INFO:                  TID: 1141
2024-09-16 22:56:25.046 CEST [1129/T1141] INFO:                  Name: database
2024-09-16 22:56:25.047 CEST [1129/T1141] INFO: Received signal: Illegal instruction
2024-09-16 22:56:25.047 CEST [1129/T1141] INFO:      at address: 0x5d0180
2024-09-16 22:56:25.047 CEST [1129/T1141] INFO:      with code:  ILL_ILLOPC (Illegal opcode)
2024-09-16 22:56:25.079 CEST [1129/T1141] INFO: Backtrace:
2024-09-16 22:56:25.087 CEST [1129/T1141] INFO: B[0000]: /usr/bin/pihole-FTL(generate_backtrace+0x40) [0x47ae1c]
2024-09-16 22:56:25.550 CEST [1129/T1141] INFO: L[0000]: /root/FTL/src/signals.c:107
2024-09-16 22:56:25.569 CEST [1129/T1141] INFO: B[0001]: /usr/bin/pihole-FTL(+0x5c45c) [0x47b45c]
2024-09-16 22:56:26.247 CEST [1129/T1141] INFO: L[0001]: /root/FTL/src/signals.c:250
2024-09-16 22:56:26.269 CEST [1129/T1141] INFO: B[0002]: /lib/arm-linux-gnueabi/libc.so.6(__default_rt_sa_restorer+0) [0xb6c64930]
2024-09-16 22:56:26.270 CEST [1129/T1141] INFO: B[0003]: /usr/bin/pihole-FTL(+0x1b1180) [0x5d0180]
2024-09-16 22:56:28.312 CEST [1129/T1141] INFO: L[0003]: /root/FTL/src/database/sqlite3.c:161783
2024-09-16 22:56:28.410 CEST [1129/T1141] INFO: B[0004]: /usr/bin/pihole-FTL(+0x24fbec) [0x66ebec]
2024-09-16 22:56:30.233 CEST [1129/T1141] INFO: L[0004]: /root/FTL/src/database/sqlite3.c:168439
2024-09-16 22:56:30.334 CEST [1129/T1141] INFO: B[0005]: /usr/bin/pihole-FTL(+0x2381e0) [0x6571e0]
2024-09-16 22:56:31.717 CEST [1129/T1141] INFO: L[0005]: /root/FTL/src/database/sqlite3.c:150845
2024-09-16 22:56:31.810 CEST [1129/T1141] INFO: B[0006]: /usr/bin/pihole-FTL(+0x26b680) [0x68a680]
2024-09-16 22:56:33.157 CEST [1129/T1141] INFO: L[0006]: /root/FTL/src/database/sqlite3.c:176408
2024-09-16 22:56:33.255 CEST [1129/T1141] INFO: B[0007]: /usr/bin/pihole-FTL(+0x26ee74) [0x68de74]
2024-09-16 22:56:34.583 CEST [1129/T1141] INFO: L[0007]: /root/FTL/src/database/sqlite3.c:175078
2024-09-16 22:56:34.677 CEST [1129/T1141] INFO: B[0008]: /usr/bin/pihole-FTL(+0x26f9a4) [0x68e9a4]
2024-09-16 22:56:35.990 CEST [1129/T1141] INFO: L[0008]: /root/FTL/src/database/sqlite3.c:142599
2024-09-16 22:56:36.084 CEST [1129/T1141] INFO: B[0009]: /usr/bin/pihole-FTL(+0x26ff60) [0x68ef60]
2024-09-16 22:56:37.517 CEST [1129/T1141] INFO: L[0009]: /root/FTL/src/database/sqlite3.c:142672
2024-09-16 22:56:37.616 CEST [1129/T1141] INFO: B[0010]: /usr/bin/pihole-FTL(sqlite3_prepare_v2+0x24) [0x68f514]
2024-09-16 22:56:39.113 CEST [1129/T1141] INFO: L[0010]: /root/FTL/src/database/sqlite3.c:142761
2024-09-16 22:56:39.224 CEST [1129/T1141] INFO: B[0011]: /usr/bin/pihole-FTL(gravity_updated+0x104) [0x4f9578]
2024-09-16 22:56:39.434 CEST [1129/T1141] INFO: L[0011]: /root/FTL/src/database/gravity-db.c:2755
2024-09-16 22:56:39.454 CEST [1129/T1141] INFO: B[0012]: /usr/bin/pihole-FTL(DB_thread+0x504) [0x4f14c4]
2024-09-16 22:56:39.591 CEST [1129/T1141] INFO: L[0012]: /root/FTL/src/database/database-thread.c:123
2024-09-16 22:56:39.614 CEST [1129/T1141] INFO: ------ Listing content of directory /dev/shm ------
2024-09-16 22:56:39.615 CEST [1129/T1141] INFO: File Mode User:Group      Size  Filename
2024-09-16 22:56:39.624 CEST [1129/T1141] INFO: rwxrwxrwx root:root       300   .
2024-09-16 22:56:39.625 CEST [1129/T1141] INFO: rwxr-xr-x root:root         3K  ..
2024-09-16 22:56:39.626 CEST [1129/T1141] INFO: rw------- pihole:pihole   560K  FTL-fifo-log
2024-09-16 22:56:39.627 CEST [1129/T1141] INFO: rw------- pihole:pihole     4K  FTL-per-client-regex
2024-09-16 22:56:39.628 CEST [1129/T1141] INFO: rw------- pihole:pihole   229K  FTL-dns-cache
2024-09-16 22:56:39.629 CEST [1129/T1141] INFO: rw------- pihole:pihole    12K  FTL-overTime
2024-09-16 22:56:39.630 CEST [1129/T1141] INFO: rw------- pihole:pihole     2M  FTL-queries
2024-09-16 22:56:39.631 CEST [1129/T1141] INFO: rw------- pihole:pihole    12K  FTL-upstreams
2024-09-16 22:56:39.633 CEST [1129/T1141] INFO: rw------- pihole:pihole   168K  FTL-clients
2024-09-16 22:56:39.633 CEST [1129/T1141] INFO: rw------- pihole:pihole   152K  FTL-domains
2024-09-16 22:56:39.634 CEST [1129/T1141] INFO: rw------- pihole:pihole   123K  FTL-strings
2024-09-16 22:56:39.636 CEST [1129/T1141] INFO: rw------- pihole:pihole   140   FTL-settings
2024-09-16 22:56:39.637 CEST [1129/T1141] INFO: rw------- pihole:pihole   296   FTL-counters
2024-09-16 22:56:39.638 CEST [1129/T1141] INFO: rw------- pihole:pihole    56   FTL-lock
2024-09-16 22:56:39.638 CEST [1129/T1141] INFO: rw-r--r-- root:root         0   .tmpfs
2024-09-16 22:56:39.639 CEST [1129/T1141] INFO: ---------------------------------------------------
2024-09-16 22:56:39.639 CEST [1129/T1141] INFO: Please also include some lines from above the !!!!!!!!! header.
2024-09-16 22:56:39.639 CEST [1129/T1141] INFO: Thank you for helping us to improve our FTL engine!
2024-09-16 22:56:39.640 CEST [1129/T1141] INFO: Waiting for threads to join
2024-09-16 22:56:39.641 CEST [1129/T1141] INFO: Thread database (0) is still busy, cancelling it.
2024-09-16 22:56:39.732 CEST [1129/T1144] INFO: Terminating timer thread
2024-09-16 22:56:40.294 CEST [1129/T1142] INFO: Terminating GC thread
2024-09-16 22:56:40.580 CEST [1129/T1143] INFO: Terminating resolver thread
2024-09-16 22:57:01.227 CEST [1129M] INFO: Finished final database update
2024-09-16 22:57:01.248 CEST [1129M] INFO: Waiting for threads to join
2024-09-16 22:57:01.253 CEST [1129M] INFO: Thread ntp-client (4) is idle, terminating it.
2024-09-16 22:57:01.255 CEST [1129M] INFO: All threads joined
2024-09-16 22:57:01.636 CEST [1129M] INFO: Stored 1 API session in the database
2024-09-16 22:57:04.439 CEST [1129M] INFO: ########## FTL terminated after 15h 30m 31s  (code 0)! ##########
4

Both crashes are different. While the first one really looks like something we might be able to fix, the second is actually an instruction error - meaning the binary you created tried to do something on your processors which it cannot do.

Let's first look at what we have here and, then, there will be a summary at the end.

First crash - segmentation fault

(many more sqlite3 references)

Second crash - illegal instruction

(many more sqlite3 references)

Summary

I reviewed the first crash and also the path leading to the second and - to the best of my knowledge, there isn't anything wrong here. I checked git blame and have seen that said code is untouched for 4 years (!). My current best guess would be that your binary was either damaged (bad SD card?) or that some other error happened during compilation. I'm aware that this looks like I wan to make my life easy here, but really, the code immediately preceding the crash here is simple and shouldn't be causing issues. If it is not a bad binary (in whatever way), the only piece left is an original bug in the SQLite3 binary which kind of exceeds our current debugging capabilities.

What might be interesting in supporting/dismissing my theory here:

Please do not update the branch, yet. Backup the pihole-FTL binary you have compiled (the crashing one) somewhere else and run a full re-build using ./build.sh clean - then compare the hash of the previous with the new binary (e.g.´sha256sum pihole-FTL). If there was any kind of corruption, different hashes will be reproduced if you are still compiling on the same git commit. Even after a full clean build (as above), you should get the exact same hash as FTL's build system generated fully reproducible builds.

5

You're right. The checksums don't match :scream: No idea how that happened.
Here are the results, the one that crashed:

d19a43bafbd8a0f12669c3fa44093977d2b64bbf03cd8876bf6d8cc0be841fd8  /usr/bin/pihole-FTL

The fresh compiled:

f09ad51f06309b33e224eb1bbc72a35f033eba07dc7424326ddd696b1410af3b  /root/FTL/cmake/pihole-FTL

Also, this is running on an SSD, installed in the pogoplug, do no sdcards to blame here.
Anyway, the freshly compiled one is running now. Let's see what happens :crossed_fingers: