Trouble compiling FTL development

Expected Behaviour:

FTL compiles and runs with no errors

  • Ubuntu 26.04.1
  • Raspberry Pi 5
  • Bare Metal

Actual Behaviour:

FTL compiles but webserver errors on startup and is inaccessible.

2026-10-28 06:08:41.957 PDT [2689191/T2689206] DEBUG_WEBSERVER: mg_cry called from mg_start2:22242: Error creating timers (src/webserver/civetweb/civetweb.c:3494)
2026-10-28 06:08:41.957 PDT [2689191/T2689206] ERROR: Error creating timers

Debug Token: https://tricorder.pi-hole.net/hUpJ433c/

Additional details:

I have been compiling FTL on the development branch without issue for some time and within the last 3 days or so (unsure of the commit) the webserver refuses to start.

I can download the precompiled binary and it runs fine. I followed the install from source using the proposed guide here: https://deploy-preview-1400--pihole-docs.netlify.app/.

Here are the results of pihole-FTL -vv with both the precompiled binary and locally compiled one.

Locally compiled:

pi@pi5:~/FTL$ pihole-FTL -vv
****************************** FTL **********************************
Version:         v6.8-412-g86f915f7
Branch:          development
Commit:          86f915f7 (2026-10-28 18:17:22 +0100)
Architecture:    aarch64 (compiled locally)
Compiler:        cc (Ubuntu 15.2.0-16ubuntu1) 15.2.0
GLIBC version:   2.43

****************************** dnsmasq ******************************
Version:         pi-hole-v2.93+35
Features:        IPv6 GNU-getopt no-DBus no-UBus no-i18n IDN2 DHCP DHCPv6 Lua TFTP no-conntrack ipset no-nftset auth DNSSEC loop-detect inotify dumpfile

****************************** SQLite3 ******************************
Version:         3.54.1
Features:        ATOMIC_INTRINSICS=1 COMPILER=gcc-15.2.0 DEFAULT_AUTOVACUUM DEFAULT_CACHE_SIZE=4096 DEFAULT_FILE_FORMAT=4 DEFAULT_FOREIGN_KEYS DEFAULT_JOURNAL_SIZE_LIMIT=33554432 DEFAULT_MEMSTATUS=0 DEFAULT_MMAP_SIZE=0 DEFAULT_PAGE_SIZE=4096 DEFAULT_PCACHE_INITSZ=20 DEFAULT_RECURSIVE_TRIGGERS DEFAULT_SECTOR_SIZE=4096 DEFAULT_SYNCHRONOUS=1 DEFAULT_WAL_AUTOCHECKPOINT=1000 DEFAULT_WAL_SYNCHRONOUS=1 DEFAULT_WORKER_THREADS=0 DIRECT_OVERFLOW_READ DQS=0 ENABLE_CARRAY ENABLE_DBPAGE_VTAB ENABLE_DBSTAT_VTAB ENABLE_EXPLAIN_COMMENTS ENABLE_PERCENTILE LIKE_DOESNT_MATCH_BLOBS MALLOC_SOFT_LIMIT=1024 MAX_ATTACHED=10 MAX_COLUMN=2000 MAX_COMPOUND_SELECT=500 MAX_DEFAULT_PAGE_SIZE=8192 MAX_EXPR_DEPTH=1000 MAX_FUNCTION_ARG=1000 MAX_LENGTH=1000000000 MAX_LIKE_PATTERN_LENGTH=50000 MAX_MMAP_SIZE=0x7fff0000 MAX_PAGE_COUNT=0xfffffffe MAX_PAGE_SIZE=65536 MAX_SCHEMA=10000000 MAX_SQL_LENGTH=1000000000 MAX_TRIGGER_DEPTH=1000 MAX_VARIABLE_NUMBER=32766 MAX_VDBE_OP=250000000 MAX_WORKER_THREADS=8 MUTEX_PTHREADS OMIT_DEPRECATED OMIT_LOAD_EXTENSION OMIT_PROGRESS_CALLBACK OMIT_SHARED_CACHE SYSTEM_MALLOC TEMP_STORE=1 THREADSAFE=1

******************************** LUA ********************************
Version:         Lua 5.5.0
Libraries:       inspect.lua (8.27 kB)

***************************** LIBNETTLE *****************************
Version:         3.10
GMP:             Full

****************************** CivetWeb *****************************
Version:         1.17 (modified by Pi-hole)
Features:        Files: Yes, TLS: Yes (via TLS backend), CGI: No, IPv6: Yes,
                 WebSockets: No, Server-side JavaScript: No
                 Lua: Yes, Cache: Yes, Stats: No, Compression: No
                 HTTP2: Yes (via TLS backend), Unix domain sockets: No

************************ Advanced TLS backend ***********************
HTTP/1.0:        Yes (plaintext)
HTTP/1.1:        Yes
HTTP/2:          Yes
HTTP/3:          Yes

****************************** OpenSSL ******************************
Version:         4.0.0
Built on:        Wed Oct 28 04:29:53 2026 UTC
Platform:        linux-aarch64
CPU info:        OPENSSL_armcap=0xbd
Providers:       default: Yes, legacy: No, fips: No
QUIC:            Yes

****************************** cJSON ********************************
Version:         1.7.18

Precompiled:

pi@pi5:~/FTL$ pihole-FTL -vv
****************************** FTL **********************************
Version:         vDev-86f915f
Branch:          development
Commit:          86f915f7 (2026-10-28 18:17:22 +0100)
Architecture:    linux/arm64/v8 (compiled on CI)
Compiler:        cc (Alpine 15.2.0) 15.2.0
GLIBC version:   -

****************************** dnsmasq ******************************
Version:         pi-hole-v2.93+35
Features:        IPv6 GNU-getopt no-DBus no-UBus no-i18n IDN2 DHCP DHCPv6 Lua TFTP no-conntrack ipset no-nftset auth DNSSEC loop-detect inotify dumpfile

****************************** SQLite3 ******************************
Version:         3.54.1
Features:        ATOMIC_INTRINSICS=1 COMPILER=gcc-15.2.0 DEFAULT_AUTOVACUUM DEFAULT_CACHE_SIZE=4096 DEFAULT_FILE_FORMAT=4 DEFAULT_FOREIGN_KEYS DEFAULT_JOURNAL_SIZE_LIMIT=33554432 DEFAULT_MEMSTATUS=0 DEFAULT_MMAP_SIZE=0 DEFAULT_PAGE_SIZE=4096 DEFAULT_PCACHE_INITSZ=20 DEFAULT_RECURSIVE_TRIGGERS DEFAULT_SECTOR_SIZE=4096 DEFAULT_SYNCHRONOUS=1 DEFAULT_WAL_AUTOCHECKPOINT=1000 DEFAULT_WAL_SYNCHRONOUS=1 DEFAULT_WORKER_THREADS=0 DIRECT_OVERFLOW_READ DQS=0 ENABLE_CARRAY ENABLE_DBPAGE_VTAB ENABLE_DBSTAT_VTAB ENABLE_EXPLAIN_COMMENTS ENABLE_PERCENTILE LIKE_DOESNT_MATCH_BLOBS MALLOC_SOFT_LIMIT=1024 MAX_ATTACHED=10 MAX_COLUMN=2000 MAX_COMPOUND_SELECT=500 MAX_DEFAULT_PAGE_SIZE=8192 MAX_EXPR_DEPTH=1000 MAX_FUNCTION_ARG=1000 MAX_LENGTH=1000000000 MAX_LIKE_PATTERN_LENGTH=50000 MAX_MMAP_SIZE=0x7fff0000 MAX_PAGE_COUNT=0xfffffffe MAX_PAGE_SIZE=65536 MAX_SCHEMA=10000000 MAX_SQL_LENGTH=1000000000 MAX_TRIGGER_DEPTH=1000 MAX_VARIABLE_NUMBER=32766 MAX_VDBE_OP=250000000 MAX_WORKER_THREADS=8 MUTEX_PTHREADS OMIT_DEPRECATED OMIT_LOAD_EXTENSION OMIT_PROGRESS_CALLBACK OMIT_SHARED_CACHE SYSTEM_MALLOC TEMP_STORE=1 THREADSAFE=1

******************************** LUA ********************************
Version:         Lua 5.5.0
Libraries:       inspect.lua (8.27 kB)

***************************** LIBNETTLE *****************************
Version:         3.10
GMP:             Full

****************************** CivetWeb *****************************
Version:         1.17 (modified by Pi-hole)
Features:        Files: Yes, TLS: Yes (via TLS backend), CGI: No, IPv6: Yes,
                 WebSockets: No, Server-side JavaScript: No
                 Lua: Yes, Cache: Yes, Stats: No, Compression: No
                 HTTP2: Yes (via TLS backend), Unix domain sockets: No

************************ Advanced TLS backend ***********************
HTTP/1.0:        Yes (plaintext)
HTTP/1.1:        Yes
HTTP/2:          Yes
HTTP/3:          Yes

****************************** OpenSSL ******************************
Version:         4.0.0
Built on:        Tue Aug  4 18:57:03 2026 UTC
Platform:        linux-aarch64
CPU info:        OPENSSL_armcap=0x3d
Providers:       default: Yes, legacy: No, fips: No
QUIC:            Yes

****************************** cJSON ********************************
Version:         1.7.18

I'm not sure what the difference is here but it the problem definitely started after a recent (~3 days) commit.

I'm not sure if this is causing the difference, but the architectures are different:

I thought they were the same but I'm definitely not an expert on that. :man_shrugging:

Regardless, it was working before and not now. I also tried compiling against the master branch and it worked fine. So there's something in development... I think.

They usually refer to the same architecture, but I'm not sure why the locally compiled version shows a different description. I'm definitely not an expert on this matter.

Maybe @darkexplosiveqwx or @DL6ER can find the issue.

Thanks. I’ll wait and see if they have time look check into it.

TL;DR

Please try setting USE_STACK_SIZE in src/webserver/civetweb/CMakeLists.txt to 2097152.

Detailed Analysis

It did not reproduce on my AMD laptop (Fedora 44, OpenSSL 3.5.9, glibc 2.43) nor my Pi 5 (Debian 14, OpenSSL 3.6.5, glibc 2.43). It was however possible to reproduce it on my Dragon Q8B (Fedora 45, OpenSSL 4.0.3, glibc 2.44).

While stepping through it with gdb, I hit the following code branch in glibc:

497           if (__builtin_expect (size < ((guardsize + tls_static_size_for_stack
498                                          + MINIMAL_REST_STACK + pagesize_m1)
499                                         & ~pagesize_m1),
500                                 0))
501             /* The stack is too small (or the guard too large).  */
502             return EINVAL;

with the values of

(gdb) print size
$11 = 1114112
(gdb) print guardsize
$12 = 65536
(gdb) print pagesize_m1
$13 = 4095
(gdb) print tls_static_size_for_stack
$14 = 1076416
(gdb) print attr->stacksize
$15 = 1048576
(gdb) print attr->guardsize
$16 = 4096

at:

(gdb) bt
#0  allocate_stack (attr=0xffffe6869250, pdp=<synthetic pointer>, stack=<synthetic pointer>, stacksize=<synthetic pointer>) at /usr/src/debug/glibc-2.44-2.fc45.aarch64/nptl/allocatestack.c:497
#1  __pthread_create_2_1 (newthread=newthread@entry=0xffffe6869160, attr=attr@entry=0xffffe6869250, start_routine=<optimized out>, arg=arg@entry=0xffffd804f550) at pthread_create.c:680
#2  0x0000aaaaaab81cec [PAC] in mg_start_thread_with_id (func=<optimized out>, param=0xffffd804f550, threadidptr=0xffffd8045750)
    at /usr/src/debug/pihole-v7-7.0-ftl7.0.web7.0.rev5.fc45.aarch64/pi-hole-FTL-rpm-v7/src/webserver/civetweb/civetweb.c:5938
#3  timers_init (ctx=0xffffd804f550) at /usr/src/debug/pihole-v7-7.0-ftl7.0.web7.0.rev5.fc45.aarch64/pi-hole-FTL-rpm-v7/src/webserver/civetweb/timer.inl:273
#4  mg_start2 (init=<optimized out>, error=0xffffe6869528) at /usr/src/debug/pihole-v7-7.0-ftl7.0.web7.0.rev5.fc45.aarch64/pi-hole-FTL-rpm-v7/src/webserver/civetweb/civetweb.c:22240
#5  0x0000aaaaaab5c6ec [PAC] in http_init () at /usr/src/debug/pihole-v7-7.0-ftl7.0.web7.0.rev5.fc45.aarch64/pi-hole-FTL-rpm-v7/src/webserver/webserver.c:1934
#6  0x0000aaaaaab5eb8c [PAC] in webserver_thread (val=<optimized out>) at /usr/src/debug/pihole-v7-7.0-ftl7.0.web7.0.rev5.fc45.aarch64/pi-hole-FTL-rpm-v7/src/webserver/webserver.c:2231
#7  0x0000fffff72bede8 [PAC] in start_thread (arg=0xffffe686ac20) at pthread_create.c:454
#8  0x0000fffff732a5dc [PAC] in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone3.S:75

This EINVAL by pthread_create() gets turned into -1 by mg_start_thread_with_id() and finally logged as "Error creating timers".

The stack size was only too small by around 36KiB. On glibc, TLS is also typically included in the stack size. FTL PR #3017 (merged this monday) added 127KiB of new TLS which could've very likely been the cause that pushed the stack size over the edge. CI builds use static-pie musl, which sets up TLS separately from the regular stack and therefore does the stack size not apply and no issue is observed.

Reading the required TLS from the ELF object shows it being barely larger than 1MiB:

$ readelf -SW /usr/bin/pihole-FTL | grep -E '\.tdata|\.tbss'
  [22] .tbss             NOBITS          00000000005d5560 5d5560 10640f 00 WAT  0   0 16

Ob my laptop, TLS is comfortably below 1MiB, because Fedora 44 ships OpenSSL 3 and therefore excludes TLS variables used for QUIC functionality (since it requires OpenSSL 4+) at compile time:

$ readelf -SW /usr/bin/pihole-FTL | grep -E '.tdata|.tbss'
  [19] .tbss             NOBITS          0000000000806ce0 405ce0 0d62e1 00 WAT  0   0 16

While increasing the stack size seems like the obvious fix, I do have an idea to use lazy per-thread allocations for certain webserver and encrypted DNS variables, especially since not all are used by every thread.

Thank you for the detailed analysis. I will give it a shot and report back.

Success! It worked like a charm!

I appreciate the support!

Thanks @darkexplosiveqwx for the analysis. Keep large per-thread buffers out of static TLS - Pull Request #3318 - pi-hole/FTL - GitHub does the lazy per-thread allocation you suggested: the large DNS buffers of the webserver and the encrypted DNS code are now allocated on first use, which takes static TLS from about 1 MiB down to 89 KiB. A new test keeps it below 128 KiB, so it cannot creep back unnoticed. Once that is merged, the USE_STACK_SIZE change is no longer needed.

I tried the new branch with the default USE_STACK_SIZE and everything compiled and runs normally. Thanks for the fix!