PiHole crashed - Segmentation fault - Resizing queries in /dev/shm

Please follow the below template, it will help us to help you!

If you are Experiencing issues with a Pi-hole install that has non-standard elements (e.g you are using nginx instead of lighttpd, or there is some other aspect of your install that is customised) - please use the Community Help category.

Expected Behaviour:

not crashing

  • Ubuntu 20.04.3 LTS as a virtual machine on
  • vmware ESXi 6.5.0 Update 1 (Build 7388607)
  • Pi-hole v5.3.1. Web Interface 5.5.1. FTL v5.8.1.
  • VM has two cpu cores and 4GB RAM
  • There are about 50k incoming queries in a 10min timeframe.

Actual Behaviour:

  • PiHole was unresponsive. Neither DNS requests nor ssh access nor ping requests were answered. Basically the whole machine was unresponsive.
  • Not really time for analysis. So a quick reboot (via hypervisor). Worked after that for a couple of minutes. After that WebGUI was still working but showed "LOST" in the dashboard in the green query display. So another reboot.
  • It is working now for about two hours I would say.
  • After I that I tried to figure out what was wrong.

The moment right before the first reboot should be this.

[2022-01-13 06:38:06.283 1060M] Resizing "FTL-dns-cache" from 42680320 to (2667776 * 16) == 42684416 (/dev/shm: 318.7MB used, 2.1GB total, FTL uses 318.7MB)
[2022-01-13 06:41:28.175 1060M] Resizing "FTL-dns-cache" from 42684416 to (2668032 * 16) == 42688512 (/dev/shm: 318.7MB used, 2.1GB total, FTL uses 318.7MB)
[2022-01-13 06:43:42.302 1060M] Resizing "FTL-dns-cache" from 42688512 to (2668288 * 16) == 42692608 (/dev/shm: 318.7MB used, 2.1GB total, FTL uses 318.7MB)
[2022-01-13 06:46:07.663 1060M] Resizing "FTL-dns-cache" from 42692608 to (2668544 * 16) == 42696704 (/dev/shm: 318.7MB used, 2.1GB total, FTL uses 318.7MB)
[2022-01-13 06:47:23.569 1060/T1064] Resizing "FTL-dns-cache" from 42696704 to (2668800 * 16) == 42700800 (/dev/shm: 318.8MB used, 2.1GB total, FTL uses 318. 7MB)
[2022-01-13 06:49:03.670 1060/T1064] Resizing "FTL-dns-cache" from 42700800 to (2669056 * 16) == 42704896 (/dev/shm: 318.8MB used, 2.1GB total, FTL uses 318. 7MB)
[2022-01-13 06:51:39.757 1060M] Resizing "FTL-dns-cache" from 42704896 to (2669312 * 16) == 42708992 (/dev/shm: 318.8MB used, 2.1GB total, FTL uses 318.7MB)
[2022-01-13 06:52:36.384 1060M] Resizing "FTL-strings" from 25026560 to (25067520 * 1) == 25067520 (/dev/shm: 318.8MB used, 2.1GB total, FTL uses 318.8MB)
[2022-01-13 07:56:56.192 1060M] Shutting down...

a bit earlier I see

[2022-01-13 05:35:45.758 1060M] Resizing "FTL-dns-cache" from 42565632 to (2660608 * 16) == 42569728 (/dev/shm: 318.6MB used, 2.1GB total, FTL uses 318.5MB)
[2022-01-13 05:38:45.090 1060M] Resizing "FTL-dns-cache" from 42569728 to (2660864 * 16) == 42573824 (/dev/shm: 318.6MB used, 2.1GB total, FTL uses 318.5MB)
[2022-01-13 05:39:32.339 1060M] FATAL: Trying to access query ID -1, but maximum is 3686400
[2022-01-13 05:39:32.339 1060M]        found in FTL_query_in_progress() (/root/project/src/dnsmasq_interface.c:2181)
[2022-01-13 05:39:32.339 1060M] Failed to unlock SHM lock: Operation not permitted
[2022-01-13 05:40:29.620 1060M] FATAL: Trying to access query ID -1, but maximum is 3686400
[2022-01-13 05:40:29.621 1060M]        found in FTL_query_in_progress() (/root/project/src/dnsmasq_interface.c:2181)
[2022-01-13 05:40:29.621 1060M] Failed to unlock SHM lock: Operation not permitted
[2022-01-13 05:41:55.071 1060M] Resizing "FTL-dns-cache" from 42573824 to (2661120 * 16) == 42577920 (/dev/shm: 318.6MB used, 2.1GB total, FTL uses 318.6MB)
[2022-01-13 05:44:04.339 1060/T1064] Resizing "FTL-dns-cache" from 42577920 to (2661376 * 16) == 42582016 (/dev/shm: 318.6MB used, 2.1GB total, FTL uses 318. 6MB)
[2022-01-13 05:45:35.447 1060M] Resizing "FTL-dns-cache" from 42582016 to (2661632 * 16) == 42586112 (/dev/shm: 318.6MB used, 2.1GB total, FTL uses 318.6MB)
[2022-01-13 05:47:26.773 1060M] Resizing "FTL-dns-cache" from 42586112 to (2661888 * 16) == 42590208 (/dev/shm: 318.6MB used, 2.1GB total, FTL uses 318.6MB)

A couple of minutes later there is this.

[2022-01-13 08:02:28.163 1085M] Resizing "FTL-dns-cache" from 315392 to (19968 * 16) == 319488 (/dev/shm: 3.1MB used, 2.1GB total, FTL uses 3.0MB)
[2022-01-13 08:02:32.132 1085M] Resizing "FTL-dns-cache" from 319488 to (20224 * 16) == 323584 (/dev/shm: 3.1MB used, 2.1GB total, FTL uses 3.0MB)
[2022-01-13 08:02:36.446 1085M] Resizing "FTL-dns-cache" from 323584 to (20480 * 16) == 327680 (/dev/shm: 3.1MB used, 2.1GB total, FTL uses 3.0MB)
[2022-01-13 08:02:38.886 1141/F1085] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2022-01-13 08:02:38.886 1141/F1085] ---------------------------->  FTL crashed!  <----------------------------
[2022-01-13 08:02:38.886 1141/F1085] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2022-01-13 08:02:38.886 1141/F1085] Please report a bug at https://github.com/pi-hole/FTL/issues
[2022-01-13 08:02:38.886 1141/F1085] and include in your report already the following details:
[2022-01-13 08:02:38.886 1141/F1085] FTL has been running for 295 seconds
[2022-01-13 08:02:38.886 1141/F1085] FTL branch: master
:e[K
e[K[2022-01-13 08:02:38.886 1141/F1085] FTL version: v5.8.1
[2022-01-13 08:02:38.886 1141/F1085] FTL commit: b90ab8b1
[2022-01-13 08:02:38.886 1141/F1085] FTL date: 2021-04-21 20:03:47 +0100
[2022-01-13 08:02:38.886 1141/F1085] FTL user: started as pihole, ended as pihole
[2022-01-13 08:02:38.886 1141/F1085] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2022-01-13 08:02:38.886 1141/F1085] Process details: MID: 1085
[2022-01-13 08:02:38.886 1141/F1085]                  PID: 1141
[2022-01-13 08:02:38.886 1141/F1085]                  TID: 1141
[2022-01-13 08:02:38.886 1141/F1085]                  Name: pihole-FTL
[2022-01-13 08:02:38.887 1141/F1085] Received signal: Segmentation fault
[2022-01-13 08:02:38.887 1141/F1085]      at address: 0x7f72c923d000
[2022-01-13 08:02:38.887 1141/F1085]      with code:  SEGV_MAPERR (Address not mapped to object)
[2022-01-13 08:02:38.892 1141/F1085] Backtrace:
[2022-01-13 08:02:38.892 1141/F1085] B[0000]: /usr/bin/pihole-FTL(+0x5a80b) [0x55bc636a680b]
[2022-01-13 08:02:38.894 1141/F1085] L[0000]: N/A (0x5a80b)
[2022-01-13 08:02:38.894 1141/F1085] B[0001]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f72c94643c0]
[2022-01-13 08:02:38.894 1141/F1085] B[0002]: /usr/bin/pihole-FTL(_getClient+0x43) [0x55bc636a6353]
[2022-01-13 08:02:38.895 1141/F1085] L[0002]: N/A (0x5a353)
[2022-01-13 08:02:38.896 1141/F1085] B[0003]: /usr/bin/pihole-FTL(+0x624fd) [0x55bc636ae4fd]
[2022-01-13 08:02:38.897 1141/F1085] L[0003]: N/A (0x624fd)
[2022-01-13 08:02:38.897 1141/F1085] B[0004]: /usr/bin/pihole-FTL(+0x7ed61) [0x55bc636cad61]
[2022-01-13 08:02:38.899 1141/F1085] L[0004]: N/A (0x7ed61)
[2022-01-13 08:02:38.899 1141/F1085] B[0005]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f72c94643c0]
[2022-01-13 08:02:38.899 1141/F1085] B[0006]: /lib/x86_64-linux-gnu/libpthread.so.0(read+0x4c) [0x7f72c946336c]
[2022-01-13 08:02:38.899 1141/F1085] B[0007]: /usr/bin/pihole-FTL(read_write+0x6e) [0x55bc6370ba4e]
[2022-01-13 08:02:38.900 1141/F1085] L[0007]: N/A (0xbfa4e)
[2022-01-13 08:02:38.900 1141/F1085] B[0008]: /usr/bin/pihole-FTL(tcp_request+0x17b) [0x55bc636d499b]
[2022-01-13 08:02:38.902 1141/F1085] L[0008]: N/A (0x8899b)
[2022-01-13 08:02:38.902 1141/F1085] B[0009]: /usr/bin/pihole-FTL(+0x7e811) [0x55bc636ca811]
[2022-01-13 08:02:38.903 1141/F1085] L[0009]: N/A (0x7e811)
[2022-01-13 08:02:38.903 1141/F1085] B[0010]: /usr/bin/pihole-FTL(main_dnsmasq+0x12d9) [0x55bc636cc7e9]
[2022-01-13 08:02:38.905 1141/F1085] L[0010]: N/A (0x807e9)
[2022-01-13 08:02:38.905 1141/F1085] B[0011]: /usr/bin/pihole-FTL(main+0x11f) [0x55bc636912af]
[2022-01-13 08:02:38.905 1142/F1085] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2022-01-13 08:02:38.905 1142/F1085] ---------------------------->  FTL crashed!  <----------------------------
[2022-01-13 08:02:38.905 1142/F1085] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2022-01-13 08:02:38.905 1142/F1085] Please report a bug at https://github.com/pi-hole/FTL/issues
[2022-01-13 08:02:38.905 1142/F1085] and include in your report already the following details:
[2022-01-13 08:02:38.906 1142/F1085] FTL has been running for 295 seconds
[2022-01-13 08:02:38.906 1142/F1085] FTL branch: master
[2022-01-13 08:02:38.906 1142/F1085] FTL version: v5.8.1
[2022-01-13 08:02:38.906 1142/F1085] FTL commit: b90ab8b1
[2022-01-13 08:02:38.906 1142/F1085] FTL date: 2021-04-21 20:03:47 +0100
[2022-01-13 08:02:38.906 1142/F1085] FTL user: started as pihole, ended as pihole
[2022-01-13 08:02:38.906 1142/F1085] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2022-01-13 08:02:38.906 1142/F1085] Process details: MID: 1085
[2022-01-13 08:02:38.906 1142/F1085]                  PID: 1142
[2022-01-13 08:02:38.906 1142/F1085]                  TID: 1142
[2022-01-13 08:02:38.906 1142/F1085]                  Name: pihole-FTL
[2022-01-13 08:02:38.906 1142/F1085] Received signal: Segmentation fault
[2022-01-13 08:02:38.906 1142/F1085]      at address: 0x7f72c923d000
[2022-01-13 08:02:38.906 1142/F1085]      with code:  SEGV_MAPERR (Address not mapped to object)
[2022-01-13 08:02:38.907 1142/F1085] Backtrace:
[2022-01-13 08:02:38.907 1142/F1085] B[0000]: /usr/bin/pihole-FTL(+0x5a80b) [0x55bc636a680b]
[2022-01-13 08:02:38.909 1142/F1085] L[0000]: N/A (0x5a80b)
[2022-01-13 08:02:38.909 1142/F1085] B[0001]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f72c94643c0]
[2022-01-13 08:02:38.909 1142/F1085] B[0002]: /usr/bin/pihole-FTL(_getClient+0x43) [0x55bc636a6353]
[2022-01-13 08:02:38.909 1141/F1085] L[0011]: N/A (0x452af)
[2022-01-13 08:02:38.909 1141/F1085] B[0012]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f72c92840b3]
[2022-01-13 08:02:38.909 1141/F1085] B[0013]: /usr/bin/pihole-FTL(_start+0x2a) [0x55bc6369138a]
[2022-01-13 08:02:38.910 1142/F1085] L[0002]: N/A (0x5a353)
[2022-01-13 08:02:38.910 1142/F1085] B[0003]: /usr/bin/pihole-FTL(+0x624fd) [0x55bc636ae4fd]
[2022-01-13 08:02:38.911 1141/F1085] L[0013]: N/A (0x4538a)
[2022-01-13 08:02:38.911 1141/F1085] ------ Listing content of directory /dev/shm ------
[2022-01-13 08:02:38.911 1141/F1085] File Mode User:Group      Size  Filename
[2022-01-13 08:02:38.911 1141/F1085] rwxrwxrwx root:root       260   .
[2022-01-13 08:02:38.911 1141/F1085] rwxr-xr-x root:root         4K  ..
[2022-01-13 08:02:38.911 1141/F1085] rw------- pihole:pihole     4K  FTL-per-client-regex
[2022-01-13 08:02:38.911 1141/F1085] rw------- pihole:pihole   328K  FTL-dns-cache
[2022-01-13 08:02:38.911 1141/F1085] rw------- pihole:pihole     8K  FTL-overTime
[2022-01-13 08:02:38.911 1141/F1085] rw------- pihole:pihole     2M  FTL-queries
[2022-01-13 08:02:38.911 1141/F1085] rw------- pihole:pihole    20K  FTL-upstreams
[2022-01-13 08:02:38.912 1141/F1085] rw------- pihole:pihole   713K  FTL-clients
[2022-01-13 08:02:38.912 1141/F1085] rw------- pihole:pihole   160K  FTL-domains
[2022-01-13 08:02:38.912 1141/F1085] rw------- pihole:pihole   246K  FTL-strings
[2022-01-13 08:02:38.912 1141/F1085] rw------- pihole:pihole    12   FTL-settings
[2022-01-13 08:02:38.912 1141/F1085] rw------- pihole:pihole   224   FTL-counters
[2022-01-13 08:02:38.912 1141/F1085] rw------- pihole:pihole    48   FTL-lock
[2022-01-13 08:02:38.912 1141/F1085] ---------------------------------------------------
[2022-01-13 08:02:38.912 1141/F1085] Please also include some lines from above the !!!!!!!!! header.
[2022-01-13 08:02:38.912 1141/F1085] Thank you for helping us to improve our FTL engine!
[2022-01-13 08:02:38.912 1141/F1085] Asking parent pihole-FTL (PID 1085) to shut down
[2022-01-13 08:02:38.912 1141/F1085] FTL fork terminated!
[2022-01-13 08:02:38.912 1085M] Received: Real-time signal 2 (36 -> 2)
:e[K
e[K[2022-01-13 08:02:38.913 1143/F1085] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2022-01-13 08:02:38.913 1143/F1085] ---------------------------->  FTL crashed!  <----------------------------
[2022-01-13 08:02:38.913 1142/F1085] L[0003]: N/A (0x624fd)
[2022-01-13 08:02:38.913 1143/F1085] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2022-01-13 08:02:38.913 1142/F1085] B[0004]: /usr/bin/pihole-FTL(+0x7ed61) [0x55bc636cad61]
[2022-01-13 08:02:38.913 1143/F1085] Please report a bug at https://github.com/pi-hole/FTL/issues
[2022-01-13 08:02:38.913 1143/F1085] and include in your report already the following details:
[2022-01-13 08:02:38.913 1143/F1085] FTL has been running for 295 seconds
[2022-01-13 08:02:38.914 1145/F1085] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2022-01-13 08:02:38.914 1145/F1085] ---------------------------->  FTL crashed!  <----------------------------
[2022-01-13 08:02:38.914 1145/F1085] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2022-01-13 08:02:38.914 1145/F1085] Please report a bug at https://github.com/pi-hole/FTL/issues
[2022-01-13 08:02:38.914 1145/F1085] and include in your report already the following details:
[2022-01-13 08:02:38.914 1145/F1085] FTL has been running for 295 seconds
[2022-01-13 08:02:38.914 1145/F1085] FTL branch: master
[2022-01-13 08:02:38.914 1145/F1085] FTL version: v5.8.1
[2022-01-13 08:02:38.914 1145/F1085] FTL commit: b90ab8b1
[2022-01-13 08:02:38.914 1145/F1085] FTL date: 2021-04-21 20:03:47 +0100
[2022-01-13 08:02:38.914 1144/F1085] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2022-01-13 08:02:38.914 1144/F1085] ---------------------------->  FTL crashed!  <----------------------------
[2022-01-13 08:02:38.914 1145/F1085] FTL user: started as pihole, ended as pihole
[2022-01-13 08:02:38.914 1145/F1085] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2022-01-13 08:02:38.914 1145/F1085] Process details: MID: 1085
[2022-01-13 08:02:38.914 1144/F1085] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2022-01-13 08:02:38.914 1145/F1085]                  PID: 1145
[2022-01-13 08:02:38.914 1145/F1085]                  TID: 1145
[2022-01-13 08:02:38.914 1145/F1085]                  Name: pihole-FTL
[2022-01-13 08:02:38.914 1144/F1085] Please report a bug at https://github.com/pi-hole/FTL/issues
[2022-01-13 08:02:38.914 1145/F1085] Received signal: Segmentation fault
[2022-01-13 08:02:38.914 1145/F1085]      at address: 0x7f72c923d000
[2022-01-13 08:02:38.914 1145/F1085]      with code:  SEGV_MAPERR (Address not mapped to object)
[2022-01-13 08:02:38.914 1144/F1085] and include in your report already the following details:
[2022-01-13 08:02:38.914 1143/F1085] FTL branch: master
[2022-01-13 08:02:38.914 1144/F1085] FTL has been running for 295 seconds
[2022-01-13 08:02:38.915 1143/F1085] FTL version: v5.8.1
[2022-01-13 08:02:38.915 1144/F1085] FTL branch: master
[2022-01-13 08:02:38.915 1143/F1085] FTL commit: b90ab8b1
[2022-01-13 08:02:38.915 1145/F1085] Backtrace:
[2022-01-13 08:02:38.915 1143/F1085] FTL date: 2021-04-21 20:03:47 +0100
[2022-01-13 08:02:38.915 1144/F1085] FTL version: v5.8.1
[2022-01-13 08:02:38.915 1144/F1085] FTL commit: b90ab8b1
[2022-01-13 08:02:38.915 1143/F1085] FTL user: started as pihole, ended as pihole
[2022-01-13 08:02:38.915 1144/F1085] FTL date: 2021-04-21 20:03:47 +0100
[2022-01-13 08:02:38.915 1144/F1085] FTL user: started as pihole, ended as pihole
[2022-01-13 08:02:38.915 1145/F1085] B[0000]: /usr/bin/pihole-FTL(+0x5a80b) [0x55bc636a680b]
[2022-01-13 08:02:38.915 1143/F1085] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2022-01-13 08:02:38.915 1144/F1085] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2022-01-13 08:02:38.916 1085M] Shutting down...

I wanted to know what this resizing was about. I found this thread.

Although I still don't know exactly what it is for I can tell you it is getting bigger.
Starting with

[2022-01-13 08:05:24.952 1055M] Resizing "FTL-dns-cache" from 4096 to (512 * 16) == 8192 (/dev/shm: 761.9KB used, 2.1GB total, FTL uses 749.9KB)
[2022-01-13 08:05:26.006 1055M] Resizing "FTL-dns-cache" from 8192 to (768 * 16) == 12288 (/dev/shm: 766.0KB used, 2.1GB total, FTL uses 753.9KB)
[2022-01-13 08:05:27.593 1055M] Resizing "FTL-domains" from 12288 to (1024 * 24) == 24576 (/dev/shm: 770.0KB used, 2.1GB total, FTL uses 758.0KB)
[2022-01-13 08:05:27.609 1055M] Resizing "FTL-dns-cache" from 12288 to (1024 * 16) == 16384 (/dev/shm: 782.3KB used, 2.1GB total, FTL uses 770.3KB)
[2022-01-13 08:05:29.801 1055M] Resizing "FTL-dns-cache" from 16384 to (1280 * 16) == 20480 (/dev/shm: 786.4KB used, 2.1GB total, FTL uses 774.4KB)
[2022-01-13 08:05:34.094 1055M] Resizing "FTL-dns-cache" from 20480 to (1536 * 16) == 24576 (/dev/shm: 790.5KB used, 2.1GB total, FTL uses 778.5KB)
[2022-01-13 08:05:37.467 1055M] Resizing "FTL-dns-cache" from 24576 to (1792 * 16) == 28672 (/dev/shm: 794.6KB used, 2.1GB total, FTL uses 782.6KB)
[2022-01-13 08:05:40.084 1055M] Resizing "FTL-domains" from 24576 to (1536 * 24) == 36864 (/dev/shm: 798.7KB used, 2.1GB total, FTL uses 786.7KB)
[2022-01-13 08:05:40.315 1055M] Resizing "FTL-dns-cache" from 28672 to (2048 * 16) == 32768 (/dev/shm: 811.0KB used, 2.1GB total, FTL uses 799.0KB)
[2022-01-13 08:05:43.375 1055M] Resizing "FTL-dns-cache" from 32768 to (2304 * 16) == 36864 (/dev/shm: 815.1KB used, 2.1GB total, FTL uses 803.1KB)
[2022-01-13 08:05:45.164 1055M] Resizing "FTL-strings" from 81920 to (122880 * 1) == 122880 (/dev/shm: 819.2KB used, 2.1GB total, FTL uses 807.2KB)

we are now at

[2022-01-13 10:31:54.280 1055M] Resizing "FTL-dns-cache" from 2494464 to (156160 * 16) == 2498560 (/dev/shm: 44.7MB used, 2.1GB total, FTL uses 44.6MB)
[2022-01-13 10:32:14.073 1055M] Resizing "FTL-queries" from 39583744 to (622592 * 64) == 39845888 (/dev/shm: 44.7MB used, 2.1GB total, FTL uses 44.6MB)
[2022-01-13 10:32:29.926 1055M] Resizing "FTL-dns-cache" from 2498560 to (156416 * 16) == 2502656 (/dev/shm: 44.9MB used, 2.1GB total, FTL uses 44.9MB)
[2022-01-13 10:33:05.827 1055M] Resizing "FTL-dns-cache" from 2502656 to (156672 * 16) == 2506752 (/dev/shm: 44.9MB used, 2.1GB total, FTL uses 44.9MB)
[2022-01-13 10:33:34.277 1055M] Resizing "FTL-queries" from 39845888 to (626688 * 64) == 40108032 (/dev/shm: 44.9MB used, 2.1GB total, FTL uses 44.9MB)
[2022-01-13 10:33:38.808 1055M] Resizing "FTL-dns-cache" from 2506752 to (156928 * 16) == 2510848 (/dev/shm: 45.2MB used, 2.1GB total, FTL uses 45.2MB)
[2022-01-13 10:34:19.219 1055M] Resizing "FTL-dns-cache" from 2510848 to (157184 * 16) == 2514944 (/dev/shm: 45.2MB used, 2.1GB total, FTL uses 45.2MB)
[2022-01-13 10:34:44.600 1055M] Resizing "FTL-dns-cache" from 2514944 to (157440 * 16) == 2519040 (/dev/shm: 45.2MB used, 2.1GB total, FTL uses 45.2MB)
[2022-01-13 10:34:52.968 1055M] Resizing "FTL-queries" from 40108032 to (630784 * 64) == 40370176 (/dev/shm: 45.2MB used, 2.1GB total, FTL uses 45.2MB)
[2022-01-13 10:35:00.364 1055M] Resizing "FTL-strings" from 860160 to (901120 * 1) == 901120 (/dev/shm: 45.5MB used, 2.1GB total, FTL uses 45.5MB)
[2022-01-13 10:35:00.574 1055M] Resizing "FTL-dns-cache" from 2519040 to (157696 * 16) == 2523136 (/dev/shm: 45.5MB used, 2.1GB total, FTL uses 45.5MB)
[2022-01-13 10:35:30.055 1055M] Resizing "FTL-dns-cache" from 2523136 to (157952 * 16) == 2527232 (/dev/shm: 45.5MB used, 2.1GB total, FTL uses 45.5MB)
[2022-01-13 10:35:58.186 1055M] Resizing "FTL-dns-cache" from 2527232 to (158208 * 16) == 2531328 (/dev/shm: 45.5MB used, 2.1GB total, FTL uses 45.5MB)
[2022-01-13 10:36:11.459 1055M] Resizing "FTL-queries" from 40370176 to (634880 * 64) == 40632320 (/dev/shm: 45.5MB used, 2.1GB total, FTL uses 45.5MB)
[2022-01-13 10:36:24.605 1055M] Resizing "FTL-dns-cache" from 2531328 to (158464 * 16) == 2535424 (/dev/shm: 45.8MB used, 2.1GB total, FTL uses 45.8MB)
[2022-01-13 10:36:41.126 1055M] Resizing "FTL-domains" from 602112 to (25600 * 24) == 614400 (/dev/shm: 45.8MB used, 2.1GB total, FTL uses 45.8MB)

in about 2-3 hours.

I would like to understand what happened here and how to prevent that in the future while at the same time have a good documentation in this thread for future search engine users who might face the same issue.
Any help is appreciated.

Thanks!

Debug Token:

https://tricorder.pi-hole.net/1i4QeATx/

FTL stores all information it has about the queries within the last 24 hours in shared memory. We have to do this as TCP queries are handled non-blocking on continuous connections by dedicated workers (process forks) as mandated by the corresponding Internet standards.

This is expected as the memory demand grows contiguously within the first 24 hours as more and more queries are added. When queries are imported from the history database, this will be done in one batch right during startup (but I see in your debug log you disabled the database). Once FTL settled to a certain memory level (after said 24 hours), you should see these messages only rarely.
Your numbers don't look out of the ordinary, 600 KB for domains, 2,5 MB for the DNS cache, 40 MB for all queries, 860 KB for strings.

This is more than my entire household sees in 24 hours but shouldn't be an issue for the hardware you describe. Have you checked that the RAM memory is sufficient after roughly 24 hours?

This is interesting and points to an issue with the shared memory content (or query analysis). The crash is later seen in such a dedicated TCP worker (note the 1141/F1085 saying: PID 1141 fork of process 1085). Such a crash was fixed in FTL v5.9 via Improve locking during heavy TCP forking by DL6ER · Pull Request #1134 · pi-hole/FTL · GitHub

The version you are using

is before this fix and almost one year old. Please consider updating and your problem will likely already be solved.

Your post is five hours old as I read it, what is the current status?

Hi,
thanks for taking the time. I will quote specific points from your post and will answer as best as I can.

It was running fine for about 26 days. So I guess, yes?
But I can have a look tomorrow (roughly 12 hours from now).

Alright. Will do that as soon as possible.

We are at 1.7 million queries in total. Went gradually down to about 15k - 20k queries per 10min.

At the moment we're still going up.

[2022-01-13 17:16:55.565 1055M] Resizing "FTL-dns-cache" from 4472832 to (279808 * 16) == 4476928 (/dev/shm: 117.3MB used, 2.1GB total, FTL uses 117.3MB)
[2022-01-13 17:18:12.458 1055M] Resizing "FTL-queries" from 109314048 to (1712128 * 64) == 109576192 (/dev/shm: 117.3MB used, 2.1GB total, FTL uses 117.3MB)
[2022-01-13 17:18:49.594 1055M] Resizing "FTL-dns-cache" from 4476928 to (280064 * 16) == 4481024 (/dev/shm: 117.6MB used, 2.1GB total, FTL uses 117.6MB)
[2022-01-13 17:20:07.964 1055M] Resizing "FTL-dns-cache" from 4481024 to (280320 * 16) == 4485120 (/dev/shm: 117.6MB used, 2.1GB total, FTL uses 117.6MB)
[2022-01-13 17:20:48.696 1055M] Resizing "FTL-dns-cache" from 4485120 to (280576 * 16) == 4489216 (/dev/shm: 117.6MB used, 2.1GB total, FTL uses 117.6MB)
[2022-01-13 17:20:51.867 1055M] Resizing "FTL-queries" from 109576192 to (1716224 * 64) == 109838336 (/dev/shm: 117.6MB used, 2.1GB total, FTL uses 117.6MB)

But from your explanation I take it the fact that more memory is used is per design and nothing to worry about. And the crash has most likely nothing to do with this. And probably more so with the issue you mentioned in regards to TCP forking.
Correct?

Yes. I don't think there will be any issues after you upgraded your Pi-hole.

Right now we are 3.5 million queries in total and again at about 50k incoming queries per 10min.

root@dns01:/home/user01# free -t -m
              total        used        free      shared  buff/cache   available
Mem:           3931         247        2668         238        1015        3213
Swap:          2047           0        2047
Total:         5979         247        4716
root@dns01:/home/user01# df -h /dev/shm/
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           2.0G  228M  1.7G  12% /dev/shm

Now we are at 240MB in total after uptime just under 24 hours.

[2022-01-14 07:19:24.928 1055M] Resizing "FTL-dns-cache" from 8851456 to (553472 * 16) == 8855552 (/dev/shm: 236.6MB used, 2.1GB total, FTL uses 236.6MB)
[2022-01-14 07:19:55.231 1055M] Resizing "FTL-queries" from 221773824 to (3469312 * 64) == 222035968 (/dev/shm: 236.6MB used, 2.1GB total, FTL uses 236.6MB
[2022-01-14 07:20:02.304 1055M] Resizing "FTL-strings" from 2744320 to (2785280 * 1) == 2785280 (/dev/shm: 236.9MB used, 2.1GB total, FTL uses 236.9MB)
[2022-01-14 07:20:16.617 1055M] Resizing "FTL-dns-cache" from 8855552 to (553728 * 16) == 8859648 (/dev/shm: 236.9MB used, 2.1GB total, FTL uses 236.9MB)
[2022-01-14 07:21:02.316 1055M] Resizing "FTL-queries" from 222035968 to (3473408 * 64) == 222298112 (/dev/shm: 236.9MB used, 2.1GB total, FTL uses 236.9MB)
[2022-01-14 07:21:03.228 1055M] Resizing "FTL-dns-cache" from 8859648 to (553984 * 16) == 8863744 (/dev/shm: 237.2MB used, 2.1GB total, FTL uses 237.2MB)
[2022-01-14 07:21:57.559 1055M] Resizing "FTL-dns-cache" from 8863744 to (554240 * 16) == 8867840 (/dev/shm: 237.2MB used, 2.1GB total, FTL uses 237.2MB)
[2022-01-14 07:22:08.937 1055M] Resizing "FTL-queries" from 222298112 to (3477504 * 64) == 222560256 (/dev/shm: 237.2MB used, 2.1GB total, FTL uses 237.2MB)
[2022-01-14 07:22:52.918 1055M] Resizing "FTL-dns-cache" from 8867840 to (554496 * 16) == 8871936 (/dev/shm: 237.5MB used, 2.1GB total, FTL uses 237.4MB)
[2022-01-14 07:23:09.577 1055M] Resizing "FTL-queries" from 222560256 to (3481600 * 64) == 222822400 (/dev/shm: 237.5MB used, 2.1GB total, FTL uses 237.4MB)

As I understand this is expected behaviour.
I will however have an eye on it for the next few days.

Thank you for the quick and thourough replys!

These are quite extreme numbers - over 1600 clients. In which environment are you running your Pi-hole if I'm allowed to ask?

1500 clients? Are you running an open resolver - is port 53 open in the router?

1 Like

Run below one to display your ISP provided public IP:

curl ifconfig.me

Enter your public IP below to check if you run an open resolver that most likely can be abused to perform DNS reflection/amplification DDoS attacks against others:

https://openresolver.com/

Hey,

A big environment :wink:

Thank you to both of you for making sure my setup is safe.
I don't want to go into details but you don't have to worry.
The numbers you see are from the intended usecase and nothing out of the ordinary.